glossary/memory/prefix-caching
Prefix caching
also automatic prefix caching, prompt caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it. Attention over a prefix depends only on the prefix, so the cache is identical for anyone who sends those tokens. System prompts shared by every user and the earlier turns of a conversation are the two big wins. The first token then costs a cache lookup instead of a prefill.
80%
Of prefill compute saved after eleven requests in the explainer: eight users, two system prompts of 2K tokens, three turns each.
See it happen
Related
- glossary/
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- RadixAttention
SGLang's prefix cache: every stored sequence is a path in a radix tree keyed by tokens, so a new request walks the tree as far as its prompt matches and computes only the remainder.
- Cache eviction
Dropping stored KV cache to make room for new requests.
- Prefix cache hit rate
The share of prompt tokens found already in the cache rather than prefilled.
- KV-cache-aware routing
A load balancer that knows which replica's memory holds which prefixes, and routes a request to the one with the longest match instead of the next one in line.
- Cross-model KV cache transfer
Reusing the KV cache a small model built for a prompt as the starting cache of a larger model in the same family, so a cascade or router can escalate a request without the large model prefilling the context again.
- sources/
- SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023)
- vLLM: Automatic Prefix Caching
- topics/