Skip to content
inference.academy

Explainer

Two phases, two bottlenecks

A request is answered in two phases that share every weight and every line of code, and still run into different walls. Move the prompt, the output and the batch below, and watch which limit each phase is waiting on.

First the whole prompt goes through the model in a single pass. That is prefill: it builds the KV cache and produces the first token, and it is everything the client waits for before anything appears. Then the model generates one token at a time, each step reading the entire cache and every weight again. That is decode, and it repeats until the answer ends.

Prompt: 1K tokens
Output: 128 tokens
Requests in the batch: 1
One request, start to finish
prefill, the whole prompt in one pass, worth 4.9 decode stepsdecode, one token per step
Time to first token
114.9 ms
prefill, compute bound
Inter-token latency
23.4 ms
one decode step, memory bound
Request latency
3.11 s
128 tokens end to end
Throughput
43 tok/s
across 1 request
Where each phase sits on the roofline
drag a point, or click a hollow one
1101001k10k10 TF100 TF1 PFFLOP per byte read from HBMridge 208memory-boundcompute-bound1.25 PFLOP/s peakprefill, 1K tokens1022 FLOP/B, 100% of peakdecode, batch 11.0 FLOP/B, 0.5% of peak

Prefill runs the tensor cores at 100% of peak. Decode, on the same hardware with the same weights, gets 0.5%. Nothing is broken; a lone token cannot do enough arithmetic per byte to keep up.

Llama 3 70B in BF16 on 4 A100 40GB cards: 1.25 PFLOP/s dense peak against 6 TB/s. Two FLOPs per parameter per token; attention arithmetic is left out. Decode is costed at the midpoint of the growing context. Hollow points are the presets above. A roofline gives the better of two floors, never a measurement.

Prefill is bound by arithmetic. A thousand prompt tokens are a thousand rows of the same matrix multiply, and the weights are read from memory once for all of them. Arithmetic per byte is high, the tensor cores are the limit, and the first token costs about what a thousand decode steps would. Below roughly two hundred tokens the prompt is too short to earn back the read and even prefill falls onto the memory slope.

Decode is bound by bandwidth. One token per request per step, and the 140 GB of weights are read in full to produce it. At batch 1 that is one FLOP per byte, two hundred times short of the ridge, and the tensor cores sit at a fraction of a percent. Batching climbs the slope because the read is shared, exactly as the KV cache page argued. But every request brings its cache along. With the 1K prompt above, decode does not cross the ridge until 512 requests, and holding 512 of them takes 182 GB of cache, two more than four 80 GB cards have left once the weights are loaded. Give each request 4K instead and it never crosses at all. The ridge is reachable in principle and, on the budget from the previous page, almost never in practice.

This is why disaggregation exists. One engine serving both phases interleaves them, and a long prefill arriving mid-stream stalls every decode step in flight: the reader sees tokens pause. Split the phases onto separate pools and each can be tuned for its own wall, prefill for compute and decode for memory, with the cache shipped between them. Whether the shipping costs less than the interference is the question the whole disaggregation debate turns on.

The four numbers an SLO is written in

Each metric belongs to a phase, which is why tuning one rarely moves the other.

latency/
  • time to first token

    The prefill, plus any time spent queued. Owned by compute and by prompt length. Prefix caching attacks this one.

  • inter-token latency

    One decode step, the gap between consecutive tokens. Owned by bandwidth, by batch size, and by how much cache the batch drags. Quantisation and speculative decoding attack this one.

  • request latency

    First token plus every step after it. For a long answer the prefill barely registers; for a short one it dominates.

throughput/
  • output tokens per second

    Across every request in flight. Rises with batch until the cache no longer fits, which is where the KV cache page ended. The lever that raises it is the same one that lengthens inter-token latency, and that trade is the subject of the next explainer.

What this model leaves out

Two FLOPs per parameter per token counts the linear layers and ignores attention itself, which grows with context. At long contexts attention takes a real share of prefill, and its own arithmetic intensity is low, so prefill is somewhat less compute-bound than drawn here. FlashAttention is the response to exactly that.

A roofline is two floors and a maximum. Real kernels lose time to launch overhead, to the interconnect between four cards, to padding and to scheduling, and none of it appears here. Measured numbers sit below the roof, never on it.

Only the A100 40GB is offered, for the same reason as the previous page: its dense BF16 peak and bandwidth are published and settled. Newer parts have a higher ridge, not a different shape, and the two phases land on the same sides of it.

Read next