feed/tags/decode
decode
6 explainers you can run, 10 glossary terms, and 8 readings ordered as a path, each with one line on why it earns your time.
- intermediateTwo phases, two bottlenecks
Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.
- beginnerSmaller weights run faster, a smaller cache does not
Two different things are called quantization, and on your machine one is worth 3x and the other is worth nothing. Move the weight precision, the cache precision and the context, and watch which one moves the clock.
- intermediateVerifying five tokens costs the same as making one
A small model guesses ahead, the large one checks every guess in one pass. Turn the draft, the lookahead, the acceptance rate and the batch, and watch the speedup come from idle compute and vanish once the batch uses it.
- advancedSplit the phases, and the cache has to travel
Prefill on one pool, decode on another, and the whole KV cache in between. Send it after the prefill and it costs 15%; send it layer by layer and the same bytes over the same wire cost 0.19%.
- intermediateEvery layer stops to agree, twice
Split a model across 1 to 8 GPUs and watch the decode step: each shard reads less, and 160 all-reduces a token cost the same whether the batch is 1 or 256.
- intermediateCut every matrix, or cut the stack
The same model, the same eight GPUs, the same requests in flight, cut three ways. Rows are GPUs on one clock: who is busy, who is waiting, and how often a request gets a token.
- Inter-token latency
The gap between one output token and the next, once streaming has started.
- Time per output token
The average time each output token took, computed as total generation time divided by tokens generated, excluding the first.
- Speculative decoding
Having a small draft model guess several tokens ahead, then checking all of them with the large model in one forward pass.
- Acceptance rate
The share of a draft model's guessed tokens that the target model accepts during speculative decoding.
- Disaggregated serving
Running prefill and decode on separate pools of GPUs and shipping the KV cache between them.
- Memory bandwidth
Bytes per second the GPU can move between its HBM and its compute units.
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- CUDA graphs
Recording a sequence of kernel launches once and replaying the whole sequence with a single call.
- Megakernel
A single GPU kernel that runs an entire decode step, with every operation expressed as tasks that the streaming multiprocessors pull from a queue and fine-grained counters standing in for kernel boundaries.
The mental models the rest of the field takes for granted.
Current work, at the depth practitioners actually argue about.
- paperRecurrent Looped Transformer
A decoder that carries its final hidden state across every token, prompt and response alike, so depth grows with sequence length instead of being fixed by layer count. Worth reading here for what it costs rather than what it saves: recurrence prevents the decoder from processing all positions in parallel, so the prompt is consumed token by token and prefill as we know it stops existing, with only the causal encoder keeping sequence-level parallelism and a sliding window bounding the decoder cache. The author is explicit that this is a specification, reporting no throughput, latency or memory, and not claiming to reduce arithmetic or latency against a standard transformer. Read it as a design to argue with, not a result.
Yifan Zhang#prefill#decode#attention#long-context - articleInside the megakernel serving engine for North Mini Code
A decode step on one H100 moves 6.6 GB of weights and 0.5 GB of cache, so its speed of light is about 470 tokens a second; vLLM gets 185 of them and the gaps between kernels are where the rest go. One persistent kernel with counter barriers, weight prefetch across tasks and idle SMs backfilled from the next layer gets 292, 1.58x at batch 1 and 1.32x at batch 8, with the code published. The clearest account yet of what a memory-bound step loses to launch boundaries.
Xiaochun Tong, Conway Zhu and Donglu Wang#kernels#decode#gpu#hbm#moe - articleOptimizing on-device inference for Apple silicon
The roofline on one laptop. Decode hits 90% of the chip's sustained weight-read rate and removing the arithmetic changes nothing, so speculative decoding comes out 18% slower at batch 1.
Perplexity Engineering#decode#prefill#quantization#kernels#hardware - articleThe case for disaggregated LLM serving
Disaggregation is usually argued for on principle. This argues it on conditions (scale, network, and traffic balance) and is honest that below them it does not pay.
Fergus Finn#disaggregation#prefill#decode#scale - paperSplitwise: Efficient Generative LLM Inference Using Phase Splitting
Prefill and decode want different hardware. The clearest statement of why disaggregation keeps winning.
#disaggregation#prefill#decode - paperAccelerating Large Language Model Decoding with Speculative Sampling
The same idea arrived at independently at DeepMind, on Chinchilla 70B with a 4B draft across a pod: two to two and a half times faster in a distributed setting where the draft has to be worth its own communication. The clearest write-up of the modified rejection sampling.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving and others#decode#latency#speculative-decoding - paperFast Inference from Transformers via Speculative Decoding
The paper that made a small model guessing ahead respectable: the target verifies the guesses in one pass and a rejection-sampling step keeps its output distribution exactly. Two to three times faster decode on T5-XXL with no retraining, and the expected tokens-per-step formula every later variant is measured against.
Yaniv Leviathan, Matan Kalman and Yossi Matias#decode#latency#speculative-decoding