glossary/serving/chunked-prefill

Chunked prefill

also split prefill, Sarathi

Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work. Without it, one 100K prompt stalls every stream in the batch for the length of its prefill. With it, the prompt takes more steps to finish but nobody else's tokens stop. It trades first-token latency for the long request against inter-token latency for everyone else.

See it happen

• explainers/
• intermediateTwo phases, two bottlenecks

  Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.

Related

• glossary/
• Prefill

  The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token.

• Continuous batching

  Scheduling at the granularity of a single decode step instead of a whole batch.

• Inter-token latency

  The gap between one output token and the next, once streaming has started.

• Request scheduling

  Deciding, each step, which waiting requests join the batch and which running ones stay.

• sources/
• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills (Agrawal et al., 2023)
• topics/
• #prefill#scheduling#latency