feed/tags/quantization
quantization
One explainer you can run, 7 glossary terms, and 3 readings ordered as a path, each with one line on why it earns your time.
- BF16
A 16-bit float with the same 8-bit exponent as FP32 and only 7 bits of mantissa.
- FP8
An 8-bit float in two flavours: E4M3 with more precision for weights and activations, E5M2 with more range for gradients.
- FP4
A 4-bit float with sixteen representable values, usable only with fine-grained scaling: a shared scale factor per small block of values restores the range the format lacks.
- Weight-only quantization
Storing the weights in four or eight bits and converting them back to 16 on the way into the multiply, leaving activations at full precision.
- Activation quantization
Quantizing the activations flowing between layers as well as the weights, so the matmul itself runs in low precision on the tensor cores.
- KV cache quantization
Storing the cached keys and values in 8 or 4 bits instead of 16.
- TensorRT-LLM
NVIDIA's inference library for its own GPUs: compiled engines with hand-tuned kernels, the first place FP8 and FP4 land, in-flight batching, paged KV cache, and the wide expert parallelism and disaggregated serving that its newest racks are built for.
Current work, at the depth practitioners actually argue about.
- paperDeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
Long-horizon agents make the workload input-heavy, so the bytes of cache per token are the bill. V4.1-Flash attacks that in the architecture: the lower 20 of 40 layers run as an encoder and the decoder's global KV is projected from the encoder state, so prefill does about half the layer work; CSA2 shares main KV and indexer keys across layers, and Reindex layers rescore only a candidate pool the first full layer chose, so per-query cost stops growing with context; main KV is MXFP4 and the sliding-window cache is replayed on resume rather than stored. The result is 890 bytes per token of global KV in HBM at 1M context, a quarter of V4-Flash, an eighth in persistent storage. The absolute V4-Flash figure is not in the paper, so the ratios are what you get.
DeepSeek-AI#kv-cache#long-context#attention#quantization#agents - articleOptimizing on-device inference for Apple silicon
The roofline on one laptop. Decode hits 90% of the chip's sustained weight-read rate and removing the arithmetic changes nothing, so speculative decoding comes out 18% slower at batch 1.
Perplexity Engineering#decode#prefill#quantization#kernels#hardware - paperTOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
You are billed for a model you cannot inspect. A 258-byte proof per 32 tokens that catches a swapped model, an altered prompt, or quietly reduced precision.
Jack Min Ong and colleagues#verifiability#trust#quantization