Explainer
Verifying five tokens costs the same as making one
A decode step reads every weight in the model to produce one token, and the arithmetic units spend most of that time waiting. Speculative decoding puts a small model in front to guess a few tokens ahead, then has the large one check all the guesses in a single pass. Below, the same 48 tokens are produced both ways on the same four cards.
The check is exact. The target scores each guessed token against what it would have produced, keeps the run of guesses it agrees with, and replaces the first one it rejects with its own choice. Sampled with the right correction, the output has the same distribution as if the target had decoded alone. Nothing about quality changes; only how many tokens come out of each read of the weights.
Each step reads the target’s 140 GB once. Without speculation that buys one token per request; with it, the same read verifies 4 guesses and yields about 3.4 tokens.
The gain is the idle compute, and nothing else. At batch one, decoding Llama 3 70B reads 140 GB per token and uses well under one percent of the tensor cores. Verifying five positions instead of one reads the same bytes and does five times the arithmetic, which is still nothing. The step barely gets longer, so every accepted guess is a token for free, and the only real cost is the draft steps in front of it. The speedup you see is that idle bandwidth-to-compute gap being spent.
The acceptance rate is the whole result. With per-token acceptance a and lookahead k, a step yields (1 minus a to the k plus 1) over (1 minus a) tokens on average: at 80% and a lookahead of 4, about 3.4. A longer lookahead only helps while the draft is still right; past the first rejection every further guess is drafting time thrown away, which is why the curve for k of 8 sits below k of 4 at 50% acceptance. Acceptance is not a property of the technique. It is a property of the draft and of the text: code and boilerplate accept high, open-ended prose accepts low.
It stops paying when the batch fills the compute. Continuous batching is the other way to use idle compute, and the two compete for the same slack. As batch size grows the verify pass does k plus 1 tokens of arithmetic for every request, and somewhere in the hundreds it crosses the ridge and becomes compute-bound. From there each speculative step is slower than a plain one and the draft steps are pure overhead. Speculation is a latency tool for lightly loaded serving; a throughput-bound deployment already has nowhere to put the guesses.
What this model leaves out
Acceptance is a dial here. In a deployment it is measured, it varies token by token, and it is the number that decides whether speculation is worth running at all. The figures reported in the papers, two to three times at batch one, are for particular draft and target pairs on particular text.
Small models do not reach bandwidth peak. A 1B draft step is priced at the time to read 2.5 GB, well under a millisecond, but a model that small is dominated by kernel launch overhead and rarely gets close to that. Real draft steps cost more than this page charges them, which shortens the gains on the left of the chart.
Both models live in the same memory. The draft’s weights and its own KV cache come out of the budget the KV cache page fills, so speculating costs slots as well as time. Attention FLOPs are left out as on the roofline page. And the draft here is a separate model guessing one token at a time; drafts that grow from the target’s own hidden state, such as EAGLE and Medusa, or that propose a tree of candidates rather than a chain, change the costs but not the argument.
Read next
- explainers/
- Two phases, two bottlenecks
The roofline these step times come from, and why decode sits so far below the ridge.
- A batch is only as fast as its slowest member
The other way to spend idle compute, and the one speculation competes with.
- feed/
- Fast Inference from Transformers via Speculative Decoding
Leviathan, Kalman and Matias. The draft-and-verify scheme, with the proof that the output distribution is unchanged.
- Accelerating Large Language Model Decoding with Speculative Sampling
Chen and others at DeepMind. The same idea on a 70B model across a pod, with the rejection-sampling correction spelled out.
- glossary/