glossary/memory/kv-cache
KV cache
also key-value cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing. It grows with every token of every request in flight, which makes it the memory that scales with users while the weights stay fixed. It is what caps batch size, what paging manages, what a prefix cache shares, and what disaggregation ships between machines.
320 KiB
Per token for Llama 3 70B in BF16: 80 layers, 8 KV heads, 128 dimensions, keys and values, 2 bytes each.
See it happen
Related
- glossary/
- PagedAttention
Storing the KV cache in fixed-size blocks that need not be contiguous, with a per-request table mapping its logical positions to physical blocks, exactly as an operating system pages virtual memory.
- Prefix caching
Keeping the KV cache of a prompt's beginning after its request finishes, so the next request that starts the same way skips computing it.
- Cross-model KV cache transfer
Reusing the KV cache a small model built for a prompt as the starting cache of a larger model in the same family, so a cascade or router can escalate a request without the large model prefilling the context again.
- Cross-layer KV sharing
Letting several transformer layers attend over one set of keys and values instead of each keeping its own, so the cache shrinks along the layer dimension the way grouped-query attention shrinks it along the head dimension.
- High bandwidth memory
The stacked DRAM on the GPU package that holds the weights and the KV cache.
- KV cache quantization
Storing the cached keys and values in 8 or 4 bits instead of 16.
- sources/
- topics/