feed/tags/prefill
prefill
4 explainers you can run, 6 glossary terms, and 7 readings ordered as a path, each with one line on why it earns your time.
- intermediateTwo phases, two bottlenecks
Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.
- intermediateA long prompt should not stall everyone else
Sixteen streams mid-reply when a 16K prompt lands. Run its prefill alone and they freeze for two seconds; chunk it into decode steps and the longest gap is 30 milliseconds. The right chunk size is the ridge point.
- advancedSplit the phases, and the cache has to travel
Prefill on one pool, decode on another, and the whole KV cache in between. Send it after the prefill and it costs 15%; send it layer by layer and the same bytes over the same wire cost 0.19%.
- advancedCross-Model KV Cache Transfer in LLM Families
Escalate a request from a 14B model to its 32B sibling and price the handoff both ways: re-prefill the context, or map the small model's cache across with a per-head linear fit.
- Time to first token
The delay between a request arriving and the first token of its answer leaving.
- Chunked prefill
Splitting a long prompt's prefill into pieces of a few thousand tokens and running each piece in the same step as the batch's decode work.
- Generation stall
A pause in every decoding stream in a batch while the scheduler runs another request's prefill as its own iteration.
- Disaggregated serving
Running prefill and decode on separate pools of GPUs and shipping the KV cache between them.
- Prefill
The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token.
- Compute-bound
Limited by arithmetic throughput; the memory system keeps up and the tensor cores are the bottleneck.
Current work, at the depth practitioners actually argue about.
- paperContext Language Models
An agent's context becomes a file it can edit, not just a history its harness periodically summarizes. Existing models learn to keep notes, remove irrelevant tool output and update state in place. On BrowseComp-Plus with Qwen3.6-27B and a 32K context budget, that beats Codex-style summarization by 11.4% relative accuracy with 21.5% fewer prefix-reuse FLOPs. The serving consequence is the part to read here: editing the middle breaks prefix reuse and forces the suffix through prefill again. Suffix Cache Reuse keeps surviving tokens' old states instead, cutting compute by another 35% in the paper's SGLang comparison at matched task accuracy. Those states still encode the previous prefix, so this is approximate reuse, not an exact cache hit, and FLOPs are not a latency or billing measurement.
Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin and others#agents#long-context#prefix-cache#kv-cache#prefill#sglang - paperRecurrent Looped Transformer
A decoder that carries its final hidden state across every token, prompt and response alike, so depth grows with sequence length instead of being fixed by layer count. Worth reading here for what it costs rather than what it saves: recurrence prevents the decoder from processing all positions in parallel, so the prompt is consumed token by token and prefill as we know it stops existing, with only the causal encoder keeping sequence-level parallelism and a sliding window bounding the decoder cache. The author is explicit that this is a specification, reporting no throughput, latency or memory, and not claiming to reduce arithmetic or latency against a standard transformer. Read it as a design to argue with, not a result.
Yifan Zhang#prefill#decode#attention#long-context - articleOptimizing on-device inference for Apple silicon
The roofline on one laptop. Decode hits 90% of the chip's sustained weight-read rate and removing the arithmetic changes nothing, so speculative decoding comes out 18% slower at batch 1.
Perplexity Engineering#decode#prefill#quantization#kernels#hardware - articleThe case for disaggregated LLM serving
Disaggregation is usually argued for on principle. This argues it on conditions (scale, network, and traffic balance) and is honest that below them it does not pay.
Fergus Finn#disaggregation#prefill#decode#scale - paperCross-model KV cache transfer in LLM families: a closed-form linear mapping for prefill reuse
Every model swap in a cascade or a router throws the prefill away, because the bigger sibling cannot read the smaller one's cache. This paper fits a per-head ridge regression from the small model's keys and values to the large one's, with RoPE stripped so the map is position-free, on 500 calibration sequences. Four of six pairs keep 73 to 98% of standalone accuracy and the map runs 2.7 to 25x faster than re-prefill; two pairs fall apart, which is the honest part. Prefix caching across models, not just across requests.
Taekyung Heo, Rasoul Shafipour, Ritchie Zhao and others#kv-cache#prefix-cache#prefill#routing#cost - paperTaming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Names the generation stall and removes it: prefill in chunks that ride inside decode steps under a per-step token budget, so a long prompt never freezes the batch. 2.6x the serving capacity of vLLM for Mistral 7B on one A100 under tail-latency SLOs, 3.7x for Yi-34B on two. OSDI 2024; every engine has it now.
Amey Agrawal, Nitin Kedia, Ashish Panwar and others#chunked-prefill#prefill#scheduling#latency - paperSplitwise: Efficient Generative LLM Inference Using Phase Splitting
Prefill and decode want different hardware. The clearest statement of why disaggregation keeps winning.
#disaggregation#prefill#decode