glossary/serving/generation-stall
Generation stall
also prefill-decode interference, decode stall
A pause in every decoding stream in a batch while the scheduler runs another request's prefill as its own iteration. The stall lasts the length of that prefill, seconds for a long prompt, and lands on the batch as one inter-token gap. It is the usual cause of a bad tail in inter-token latency when the median is fine, and it is what chunked prefill exists to remove.
1.84 s
For a 16K-token prompt prefilled alone on Llama 3 70B across 4 A100s, against a 25 ms decode step for the streams that wait through it.
See it happen
Related
- glossary/
- Chunked prefill
Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.
- Inter-token latency
The gap between one output token and the next, once streaming has started.
- Prefill
The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token.
- Request scheduling
Deciding, each step, which waiting requests join the batch and which running ones stay.
- sources/
- topics/