feed/tags/prefix-cache
prefix cache
2 explainers you can run, 4 glossary terms, and 3 readings ordered as a path, each with one line on why it earns your time.
- beginnerCompute the system prompt once
Eight users, one system prompt, several turns each. Step through the requests and watch what each one actually computes once the cache is addressed by prefix.
- intermediateThe hit rate is the workload
Agent sessions taking turns against one card's memory. Add sessions and watch each turn's cost jump from nothing to ten seconds as its history is evicted.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- Prefix cache hit rate
The share of prompt tokens found already in the cache rather than prefilled.
- RadixAttention
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.
- Cache eviction
Dropping stored KV cache to make room for new requests.
Start here. No serving experience assumed.
Current work, at the depth practitioners actually argue about.
- articleAgentX and InferenceX v3: does the CUDA moat hold up in agentic inference?
An agent turn is 88K tokens in and 413 out, so the benchmark is really a KV cache hit-rate benchmark: 91% on B300 at 384 sessions, 73% on B200 with the rest spilling to DRAM. Nvidia, AMD, five open models, 70 upstream PRs. Partly paywalled.
Cam Quilici, Bryan Shan, Alec Ibarra and others#agents#prefix-cache#kv-cache#benchmark#hardware - paperSGLang: Efficient Execution of Structured Language Model Programs
Prefix caching as a first-class scheduling primitive. If your agent workload is 80% cache hits, this is the paper explaining your bill.
#prefix-cache#sglang