glossary/precision/weight-only-quantization
Weight-only quantization
also W4A16, GPTQ, AWQ
Storing the weights in four or eight bits and converting them back to 16 on the way into the multiply, leaving activations at full precision. It cuts the bytes read per decode step without needing low-precision tensor cores, so it speeds up memory-bound decode on any GPU. It does nothing for compute-bound prefill, and the dequantisation adds work, so at large batch it can be slower than BF16.
4x fewer bytes
Per decode step at 4-bit weights, the whole gain, since the arithmetic is unchanged.
See it happen
Related
- glossary/
- Activation quantization
Quantizing the activations flowing between layers as well as the weights, so the matmul itself runs in low precision on the tensor cores.
- FP4
A 4-bit float with sixteen representable values, usable only with fine-grained scaling: a shared scale factor per small block of values restores the range the format lacks.
- Memory-bound
Limited by how fast bytes can be read, not by how fast they can be operated on.
- Decode
Producing output one token per step, each step reading the entire model and the request's cache to compute a single new token.
- sources/
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al., 2022)
- AWQ: Activation-aware Weight Quantization (Lin et al., 2023)
- topics/