Prefill vs decode: two phases, two bottlenecks / inference.academy

First the whole prompt goes through the model in a single pass. That is prefill: it builds the KV cache and produces the first token, and it is everything the client waits for before anything appears. Then the model generates one token at a time, each step reading the entire cache and every weight again. That is decode, and it repeats until the answer ends.

Prompt: 1K tokens

Output: 128 tokens

Requests in the batch: 1

One request, start to finish

prefill, the whole prompt in one pass, worth 4.9 decode stepsdecode, one token per step

Time to first token

114.9 ms

prefill, compute bound

Inter-token latency

23.4 ms

one decode step, memory bound

Request latency

3.11 s

128 tokens end to end

Throughput

43 tok/s

across 1 request

Where each phase sits on the roofline

drag a point, or click a hollow one

Prefill runs the tensor cores at 100% of peak. Decode, on the same hardware with the same weights, gets 0.5%. Nothing is broken; a lone token cannot do enough arithmetic per byte to keep up.

Llama 3 70B in BF16 on 4 A100 40GB cards: 1.25 PFLOP/s dense peak against 6 TB/s. Two FLOPs per parameter per token; attention arithmetic is left out. Decode is costed at the midpoint of the growing context. Hollow points are the presets above. A roofline gives the better of two floors, never a measurement.

Prefill is bound by arithmetic. A thousand prompt tokens are a thousand rows of the same matrix multiply, and the weights are read from memory once for all of them. Arithmetic per byte is high, the tensor cores are the limit, and the first token costs about what a thousand decode steps would. Below roughly two hundred tokens the prompt is too short to earn back the read and even prefill falls onto the memory slope.

Decode is bound by bandwidth. One token per request per step, and the 140 GB of weights are read in full to produce it. At batch 1 that is one FLOP per byte, two hundred times short of the ridge, and the tensor cores sit at a fraction of a percent. Batching climbs the slope because the read is shared, exactly as the KV cache page argued. But every request brings its cache along. With the 1K prompt above, decode does not cross the ridge until 512 requests, and holding 512 of them takes 182 GB of cache, two more than four 80 GB cards have left once the weights are loaded. Give each request 4K instead and it never crosses at all. The ridge is reachable in principle and, on the budget from the previous page, almost never in practice.

This is why disaggregation exists. One engine serving both phases interleaves them, and a long prefill arriving mid-stream stalls every decode step in flight: the reader sees tokens pause. Split the phases onto separate pools and each can be tuned for its own wall, prefill for compute and decode for memory, with the cache shipped between them. Whether the shipping costs less than the interference is the question the whole disaggregation debate turns on.

The four numbers an SLO is written in

Each metric belongs to a phase, which is why tuning one rarely moves the other.

latency/

time to first token
The prefill, plus any time spent queued. Owned by compute and by prompt length. Prefix caching attacks this one.
inter-token latency
One decode step, the gap between consecutive tokens. Owned by bandwidth, by batch size, and by how much cache the batch drags. Quantisation and speculative decoding attack this one.
request latency
First token plus every step after it. For a long answer the prefill barely registers; for a short one it dominates.

throughput/

output tokens per second
Across every request in flight. Rises with batch until the cache no longer fits, which is where the KV cache page ended. The lever that raises it is the same one that lengthens inter-token latency, and that trade is the subject of the next explainer.

What this model leaves out

Two FLOPs per parameter per token counts the linear layers and ignores attention itself, which grows with context. At long contexts attention takes a real share of prefill, and its own arithmetic intensity is low, so prefill is somewhat less compute-bound than drawn here. FlashAttention is the response to exactly that.

A roofline is two floors and a maximum. Real kernels lose time to launch overhead, to the interconnect between four cards, to padding and to scheduling, and none of it appears here. Measured numbers sit below the roof, never on it.

Only the A100 40GB is offered, for the same reason as the previous page: its dense BF16 peak and bandwidth are published and settled. Newer parts have a higher ridge, not a different shape, and the two phases land on the same sides of it.

Two phases, two bottlenecks

The four numbers an SLO is written in

What this model leaves out

Read next