glossary/scale/pipeline-bubble

Pipeline bubble

also bubble, pipeline stall

The time pipeline stages sit idle because the first micro-batch has not reached them yet or the last has already left. With p stages and m micro-batches in flight the bubble is about (p-1)/(m+p-1) of a batch, GPipe's formula, which is why a pipeline needs many more micro-batches than stages. Continuous batching hides it during decode by never letting the queue run dry, and chunked prefill hides it by feeding stages long chunks; what a pipeline cannot hide is that each stage reads its weights once per micro-batch, so its per-token latency stays that of the whole model.

See it happen

• explainers/
• intermediateCut every matrix, or cut the stack

  The same model, the same eight GPUs, the same requests in flight, cut three ways. Rows are GPUs on one clock: who is busy, who is waiting, and how often a request gets a token.

Related

• glossary/
• Pipeline parallelism

  Putting different layers on different GPUs and passing activations along the chain.

• Tensor parallelism

  Splitting each weight matrix across GPUs so that every layer's matmul runs on all of them at once, with an all-reduce to combine the partial results.

• Continuous batching

  Scheduling at the granularity of a single decode step instead of a whole batch.

• Chunked prefill

  Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.

• sources/
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (Huang et al., 2018)
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (Narayanan et al., 2021)
• topics/
• #scale