feed/tags/prefix-cache
prefix cache
4 explainers you can run, 6 glossary terms, and 6 readings ordered as a path, each with one line on why it earns your time.
- beginnerCompute the system prompt once
Eight users, one system prompt, several turns each. Step through the requests and watch what each one actually computes once the cache is addressed by prefix.
- intermediateThe hit rate is the workload
Agent sessions taking turns against one card's memory. Add sessions and watch each turn's cost jump from nothing to ten seconds as its history is evicted.
- advancedCross-Model KV Cache Transfer in LLM Families
Escalate a request from a 14B model to its 32B sibling and price the handoff both ways: re-prefill the context, or map the small model's cache across with a per-head linear fit.
- intermediateThe load balancer decides the hit rate
The same sessions against the same fleet under round-robin, sticky sessions and a cache-aware router. One of them throws the prefix cache away.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- KV-cache-aware routing
A load balancer that knows which replica's memory holds which prefixes, and routes a request to the one with the longest match instead of the next one in line.
- Prefix cache hit rate
The share of prompt tokens found already in the cache rather than prefilled.
- Cross-model KV cache transfer
Reusing the KV cache a small model built for a prompt as the starting cache of a larger model in the same family, so a cascade or router can escalate a request without the large model prefilling the context again.
- RadixAttention
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.
- Cache eviction
Dropping stored KV cache to make room for new requests.
Start here. No serving experience assumed.
The mental models the rest of the field takes for granted.
Current work, at the depth practitioners actually argue about.
- paperContext Language Models
An agent's context becomes a file it can edit, not just a history its harness periodically summarizes. Existing models learn to keep notes, remove irrelevant tool output and update state in place. On BrowseComp-Plus with Qwen3.6-27B and a 32K context budget, that beats Codex-style summarization by 11.4% relative accuracy with 21.5% fewer prefix-reuse FLOPs. The serving consequence is the part to read here: editing the middle breaks prefix reuse and forces the suffix through prefill again. Suffix Cache Reuse keeps surviving tokens' old states instead, cutting compute by another 35% in the paper's SGLang comparison at matched task accuracy. Those states still encode the previous prefix, so this is approximate reuse, not an exact cache hit, and FLOPs are not a latency or billing measurement.
Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin and others#agents#long-context#prefix-cache#kv-cache#prefill#sglang - articleAgentX and InferenceX v3: does the CUDA moat hold up in agentic inference?
An agent turn is 88K tokens in and 413 out, so the benchmark is really a KV cache hit-rate benchmark: 91% on B300 at 384 sessions, 73% on B200 with the rest spilling to DRAM. Nvidia, AMD, five open models, 70 upstream PRs. Partly paywalled.
Cam Quilici, Bryan Shan, Alec Ibarra and others#agents#prefix-cache#kv-cache#benchmark#hardware - paperCross-model KV cache transfer in LLM families: a closed-form linear mapping for prefill reuse
Every model swap in a cascade or a router throws the prefill away, because the bigger sibling cannot read the smaller one's cache. This paper fits a per-head ridge regression from the small model's keys and values to the large one's, with RoPE stripped so the map is position-free, on 500 calibration sequences. Four of six pairs keep 73 to 98% of standalone accuracy and the map runs 2.7 to 25x faster than re-prefill; two pairs fall apart, which is the honest part. Prefix caching across models, not just across requests.
Taekyung Heo, Rasoul Shafipour, Ritchie Zhao and others#kv-cache#prefix-cache#prefill#routing#cost - paperSGLang: Efficient Execution of Structured Language Model Programs
Prefix caching as a first-class scheduling primitive. If your agent workload is 80% cache hits, this is the paper explaining your bill.
#prefix-cache#sglang