glossary/serving/disaggregated-serving
Disaggregated serving
also prefill-decode disaggregation, P/D disaggregation
Running prefill and decode on separate pools of GPUs and shipping the KV cache between them. Prefill is compute-bound and decode is memory-bound, so a single pool tuned for one is wrong for the other, and a long prefill stalls every decode stream sharing its GPU. Separate pools let each be batched and sized on its own terms. The cost is moving the cache, which is why the interconnect matters.
29.5 GB
The KV cache of one 88K-token agent context for Llama 3 70B, which has to cross the interconnect on every handoff.
See it happen
Related
- glossary/
- Prefill
The forward pass over the whole prompt at once, producing the KV cache for every token and the first output token.
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- Goodput
Throughput counted only for requests that met their latency targets.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- sources/
- DistServe: Disaggregating Prefill and Decoding (Zhong et al., 2024)
- Splitwise: Efficient Generative LLM Inference Using Phase Splitting (Patel et al., 2023)
- topics/