feed/tags/kv-cache
KV cache
8 explainers you can run, 11 glossary terms, and 10 readings ordered as a path, each with one line on why it earns your time.
- beginnerThe KV cache is the thing you are actually renting
Fill a GPU with weights and per-request cache until it will not take another user. 320 KiB per token adds up faster than anyone expects.
- beginnerReserve for the worst case, or page
Forty requests of unknown length against one memory, under two allocators at once. The PagedAttention paper's 20 to 40% utilisation figure, reproduced rather than quoted.
- beginnerSmaller weights run faster, a smaller cache does not
Two different things are called quantization, and on your machine one is worth 3x and the other is worth nothing. Move the weight precision, the cache precision and the context, and watch which one moves the clock.
- beginnerCompute the system prompt once
Eight users, one system prompt, several turns each. Step through the requests and watch what each one actually computes once the cache is addressed by prefix.
- intermediateThe hit rate is the workload
Agent sessions taking turns against one card's memory. Add sessions and watch each turn's cost jump from nothing to ten seconds as its history is evicted.
- advancedSplit the phases, and the cache has to travel
Prefill on one pool, decode on another, and the whole KV cache in between. Send it after the prefill and it costs 15%; send it layer by layer and the same bytes over the same wire cost 0.19%.
- advancedCross-Model KV Cache Transfer in LLM Families
Escalate a request from a 14B model to its 32B sibling and price the handoff both ways: re-prefill the context, or map the small model's cache across with a per-head linear fit.
- intermediateThe load balancer decides the hit rate
The same sessions against the same fleet under round-robin, sticky sessions and a cache-aware router. One of them throws the prefix cache away.
- Preemption
Evicting a running request from the batch when the KV cache runs out, so the others can keep growing.
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
- Block table
The per-request list that says which physical block of KV cache holds each run of its tokens.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- Cross-layer KV sharing
Letting several transformer layers attend over one set of keys and values instead of each keeping its own, so the cache shrinks along the layer dimension the way grouped-query attention shrinks it along the head dimension.
- Cross-model KV cache transfer
Reusing the KV cache a small model built for a prompt as the starting cache of a larger model in the same family, so a cascade or router can escalate a request without the large model prefilling the context again.
- RadixAttention
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.
- Cache eviction
Dropping stored KV cache to make room for new requests.
- KV cache offloading
Spilling evicted KV cache to host DRAM, or further to SSD or a remote store, instead of throwing it away.
- KV cache quantization
Storing the cached keys and values in 8 or 4 bits instead of 16.
Start here. No serving experience assumed.
The mental models the rest of the field takes for granted.
Current work, at the depth practitioners actually argue about.
- paperContext Language Models
An agent's context becomes a file it can edit, not just a history its harness periodically summarizes. Existing models learn to keep notes, remove irrelevant tool output and update state in place. On BrowseComp-Plus with Qwen3.6-27B and a 32K context budget, that beats Codex-style summarization by 11.4% relative accuracy with 21.5% fewer prefix-reuse FLOPs. The serving consequence is the part to read here: editing the middle breaks prefix reuse and forces the suffix through prefill again. Suffix Cache Reuse keeps surviving tokens' old states instead, cutting compute by another 35% in the paper's SGLang comparison at matched task accuracy. Those states still encode the previous prefix, so this is approximate reuse, not an exact cache hit, and FLOPs are not a latency or billing measurement.
Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin and others#agents#long-context#prefix-cache#kv-cache#prefill#sglang - articleScaling KV Cache Compression to Frontier Models
Evicting cache entries gets harder when sparse attention and shared latent KV force several heads and layers to keep the same token set. Ramp adapts KVzip and Attention Matching to GLM 5.3: normalizing each head's scores before selection improves ComplexMCP accuracy by 5 percentage points at 30% prefix compression, while fitting new biases and shared values recovers behavior eviction alone loses. On five QuALITY articles and 90 questions across four runs, Attention Matching reaches 83.9% accuracy with an 80% smaller article cache, against 91.8% for full context. At 95% compression it reaches 77.8%, versus 67.8% for eviction alone. The tradeoff includes preparation: the 80% setting takes about 257 seconds per article for key selection alone. A useful account of what shared-cache compression preserves, what it loses and what it costs, without a measured end-to-end serving speedup.
Jeff Zheng and Alex Chen#kv-cache#attention#long-context#memory#agents - articleToward Recursive Self-Improvement: How GLM Built Its Own Inference Infrastructure
A production stack for a 1M-context multimodal model brought up from scratch on 100,000 Chinese-made accelerators in under two weeks, tripling end-to-end throughput, then serving 62 trillion tokens in six days. The framing is that an agent did much of the work; the substance is the three debugging cases, and they are worth the read on their own. A TF32 default inside tl.dot quietly accumulating error through long-context state merges. An acceptance criterion of 5% on the prefill-to-decode handoff that measured over 20%, because a held interpreter lock kept the transfer thread from being scheduled, until releasing it took the gap under 1%.
#disaggregation#kernels#kv-cache#agents#hardware - paperDeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
Long-horizon agents make the workload input-heavy, so the bytes of cache per token are the bill. V4.1-Flash attacks that in the architecture: the lower 20 of 40 layers run as an encoder and the decoder's global KV is projected from the encoder state, so prefill does about half the layer work; CSA2 shares main KV and indexer keys across layers, and Reindex layers rescore only a candidate pool the first full layer chose, so per-query cost stops growing with context; main KV is MXFP4 and the sliding-window cache is replayed on resume rather than stored. The result is 890 bytes per token of global KV in HBM at 1M context, a quarter of V4-Flash, an eighth in persistent storage. The absolute V4-Flash figure is not in the paper, so the ratios are what you get.
DeepSeek-AI#kv-cache#long-context#attention#quantization#agents - toolNIXL: NVIDIA Inference Xfer Library
Disaggregation only pays if the KV cache moves between pools fast enough. This is the transfer layer underneath Dynamo doing that job.
#disaggregation#kv-cache#networking - articleAgentX and InferenceX v3: does the CUDA moat hold up in agentic inference?
An agent turn is 88K tokens in and 413 out, so the benchmark is really a KV cache hit-rate benchmark: 91% on B300 at 384 sessions, 73% on B200 with the rest spilling to DRAM. Nvidia, AMD, five open models, 70 upstream PRs. Partly paywalled.
Cam Quilici, Bryan Shan, Alec Ibarra and others#agents#prefix-cache#kv-cache#benchmark#hardware - paperCross-model KV cache transfer in LLM families: a closed-form linear mapping for prefill reuse
Every model swap in a cascade or a router throws the prefill away, because the bigger sibling cannot read the smaller one's cache. This paper fits a per-head ridge regression from the small model's keys and values to the large one's, with RoPE stripped so the map is position-free, on 500 calibration sequences. Four of six pairs keep 73 to 98% of standalone accuracy and the map runs 2.7 to 25x faster than re-prefill; two pairs fall apart, which is the honest part. Prefix caching across models, not just across requests.
Taekyung Heo, Rasoul Shafipour, Ritchie Zhao and others#kv-cache#prefix-cache#prefill#routing#cost - articleThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale
An agent that pauses for a tool call gets its cache evicted and pays to rebuild it. Scheduling the whole program instead of each request is the fix.
Hao Kang and colleagues#agents#kv-cache#scheduling