feed/tags/disaggregation
disaggregation
2 explainers you can run, one glossary term, and 4 readings ordered as a path, each with one line on why it earns your time.
- intermediateTwo phases, two bottlenecks
Move sequence length and batch size across a roofline and watch the limit flip from compute to memory bandwidth, the reason disaggregation exists.
- advancedSplit the phases, and the cache has to travel
Prefill on one pool, decode on another, and the whole KV cache in between. Send it after the prefill and it costs 15%; send it layer by layer and the same bytes over the same wire cost 0.19%.
Current work, at the depth practitioners actually argue about.
- articleToward Recursive Self-Improvement: How GLM Built Its Own Inference Infrastructure
A production stack for a 1M-context multimodal model brought up from scratch on 100,000 Chinese-made accelerators in under two weeks, tripling end-to-end throughput, then serving 62 trillion tokens in six days. The framing is that an agent did much of the work; the substance is the three debugging cases, and they are worth the read on their own. A TF32 default inside tl.dot quietly accumulating error through long-context state merges. An acceptance criterion of 5% on the prefill-to-decode handoff that measured over 20%, because a held interpreter lock kept the transfer thread from being scheduled, until releasing it took the gap under 1%.
#disaggregation#kernels#kv-cache#agents#hardware - toolNIXL: NVIDIA Inference Xfer Library
Disaggregation only pays if the KV cache moves between pools fast enough. This is the transfer layer underneath Dynamo doing that job.
#disaggregation#kv-cache#networking - articleThe case for disaggregated LLM serving
Disaggregation is usually argued for on principle. This argues it on conditions (scale, network, and traffic balance) and is honest that below them it does not pay.
Fergus Finn#disaggregation#prefill#decode#scale - paperSplitwise: Efficient Generative LLM Inference Using Phase Splitting
Prefill and decode want different hardware. The clearest statement of why disaggregation keeps winning.
#disaggregation#prefill#decode