Explainer
Split the phases, and the cache has to travel
Prefill wants compute and decode wants bandwidth, so one pool tuned for either is wrong for the other. Running them on separate machines lets each be sized on its own terms, and creates a problem the single pool never had: everything the prompt built has to be on the other machine before the first token can come out. Whether that costs anything is not a question about the cache. It is a question about whether the transfer was allowed to start early.
Below, one prompt through a disaggregated deployment. The top row is the prefill pool computing; the row underneath is the KV cache crossing the link. Both drawings are the same bytes over the same wire, and they differ only in when the sending is permitted to begin.
The handoff costs 0.19% here. All 10.7 GB of it still crosses the link, and almost none of it is visible, because every layer leaves while the layers after it are still being computed. Sent the other way it would cost 15%.
The transfer is not the cost. Failing to overlap it is. A 70B model prefilling 32K tokens on eight cards builds 10.7 GB of cache and takes about 1.45 seconds. Sent over 400 Gb/s InfiniBand after the prefill finishes, that is 215 milliseconds of dead time, 15% on top of the wait for the first token. Sent layer by layer as prefill produces it, each layer hides behind the layers still to be computed and the same 10.7 GB costs 0.19%. Nothing about the link, the model or the prompt changed between those two numbers. Only the scheduling did.
Which makes this a failure mode that looks like a hardware problem. Z.ai’s account of bringing up GLM-5.3-Flash includes exactly this bug, and it is worth reading because of how it was found. Their engineers set an acceptance criterion: prefill with the handoff should come within 5% of prefill alone. It was measuring over 20%. The cause was neither the fabric nor the cache size. Two components shared a process, and the dispatch call held Python’s global interpreter lock across its time in C++, so the thread responsible for submitting transfers could not be scheduled to submit them. The transfer was asynchronous in every sense except the one that mattered. Releasing the lock took the gap below 1%. An engineer watching only end-to-end throughput would have gone shopping for a faster network.
Overlap hides a fast transfer. It cannot hide a slow one. There is a floor, and the lab turns red when you hit it. Put the same 70B deployment on 25 GbE and the 10.7 GB takes 3.4 seconds against 1.45 seconds of prefill. The wire is now slower than the compute, so there is not enough prefill left to hide behind and streaming still costs more than the prefill itself. Disaggregation is an interconnect bet before it is anything else, which is why the published work on it assumes a fabric and why the first question to ask about a disaggregated design is what is between the two halves.
Big models disaggregate more easily than small ones, which is backwards from the intuition. Prefill compute scales with parameters while the cache scales with layers, heads and context, so the ratio that decides this improves as the model grows. At 32K over InfiniBand the handoff costs 0.19% on a 70B and 17.7% on a 3B: the small model finishes its prefill too quickly to hide its own cache behind. Attention design moves it further. A hybrid that keeps a growing cache on a quarter of its layers, or a sliding window that caps most of them, has a fraction of the bytes to move, and the lab lets you watch the transfer bar shrink as you change model rather than link.
What this model leaves out
The reason to do it at all. This page prices only what splitting costs. What it buys is on the other side of the ledger and it is usually larger: prefill stops stalling every decode stream it shares a GPU with, each pool can be batched on its own terms, and each can be sized and even bought differently. DistServe reports up to 7.4x more goodput under the same latency targets. A page that shows you a 15% cost and not a multiple of benefit is showing you one column.
Everything after the first token. The lab stops when decode could start. It does not model the decode pool, its batch, or the queueing in front of either pool, and in a busy cluster the wait for a free decode slot can dwarf the transfer this page is about.
A cache that never moves. The streaming here is the simple version. Real systems avoid the transfer rather than hiding it: routing a request to the pool that already holds its prefix, keeping the cache in a pool both halves can read, or layering a third stage for the multimodal encoder, which is what Z.ai’s encode-prefill-decode split does. Each changes the shape of this problem rather than solving it.
Honest link rates. Bandwidths are advertised line rates converted to bytes. Real fabrics deliver less, RDMA and its absence differ by more than the headline number suggests, and a shared cluster network is not yours alone. The prefill roofline is a ceiling in the same way. Read the percentages as the shape of the problem, not as your numbers.
Read next
- explainers/
- Two phases, two bottlenecks
Why the two halves want different machines in the first place, which is the argument this page is the invoice for.
- The load balancer decides the hit rate
The other way to avoid moving a cache: send the request to the machine that already has it.
- Every layer stops to agree, twice
The other traffic on the same fabric, and why an interconnect is a serving decision.
- feed/
- Toward Recursive Self-Improvement: How GLM Built Its Own Inference Infrastructure
Z.ai, 2026. The 5% criterion, the 20% measurement and the lock that caused it, written up in enough detail to follow.
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving
Zhong and others, 2024. The paper that made the case, and the source of the 7.4x goodput figure.
- Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Patel and others, 2023. The same split argued from the other direction, with the two pools on different hardware.
- glossary/