glossary/precision/kv-cache-quantization
KV cache quantization
also FP8 KV cache, KV cache compression
Storing the cached keys and values in 8 or 4 bits instead of 16. At long context the cache, not the weights, is most of the memory and most of the bytes a decode step reads, so halving it doubles the tokens a GPU can hold and shortens every step. Keys are harder than values because of outlier channels; per-channel scaling for keys and per-token for values is the usual answer.
160 KiB
Per token for Llama 3 70B with an FP8 cache, half of BF16, so the same 180 GB holds twice the context.
See it happen
Related
- glossary/
- KV cache
The keys and values every layer computed for every token so far, kept in GPU memory so the next token can attend to them without recomputing.
- FP8
An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.
- Working set
The amount of KV cache the active sessions need resident to keep hitting.
- Memory bandwidth
Bytes per second the GPU can move between its HBM and its compute units.
- sources/
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (Liu et al., 2024)
- vLLM: Quantized KV Cache
- topics/