feed/tags/prefill
prefill
One explainer you can run, 5 glossary terms, and 3 readings ordered as a path, each with one line on why it earns your time.
- Time to first token
The delay between a request arriving and the first token of its answer leaving.
- Chunked prefill
Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.
- Disaggregated serving
Running prefill and decode on separate pools of GPUs and shipping the KV cache between them.
- Prefill
The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token.
- Compute-bound
Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.
Current work, at the depth practitioners actually argue about.
- articleOptimizing on-device inference for Apple silicon
The roofline on one laptop. Decode hits 90% of the chip's sustained weight-read rate and removing the arithmetic changes nothing, so speculative decoding comes out 18% slower at batch 1.
Perplexity Engineering#decode#prefill#quantization#kernels#hardware - articleThe case for disaggregated LLM serving
Disaggregation is usually argued for on principle. This argues it on conditions (scale, network, and traffic balance) and is honest that below them it does not pay.
Fergus Finn#disaggregation#prefill#decode#scale - paperSplitwise: Efficient Generative LLM Inference Using Phase Splitting
Prefill and decode want different hardware. The clearest statement of why disaggregation keeps winning.
#disaggregation#prefill#decode