Explainer
Smaller weights run faster, a smaller cache does not
Two different things are called quantization. One stores each parameter in fewer bits. The other stores each cached key and value in fewer bits. They sound like the same idea applied twice, they are sold with the same word, and on the machine in front of you one of them is worth three times the speed and the other is worth nothing. Which is which depends on a number most people never compute.
A decode step is memory-bound: the arithmetic to make one token is trivial, and the time goes on reading every weight once and the whole KV cache once. So tokens per second is bandwidth divided by those bytes, and what any kind of quantization buys in speed is exactly the fraction of the bytes it removes. Below, the same step at full precision and at yours.
At 8K the cache is 18% of the reading. Weight precision is still the bigger lever, 2.88x against the cache’s 1.05x if each is taken to 4-bit on its own, but the cache has stopped being a rounding error.
| Weight precision | Bits/weight | Download | Read/token | Ceiling, tok/s |
|---|---|---|---|---|
| FP16 | 16 | 16.0 GB | 17.1 GB | 4.69 |
| Q8_0 | 8.5 | 8.50 GB | 9.57 GB | 8.36 |
| Q6_K | 6.56 | 6.56 GB | 7.63 GB | 10.5 |
| Q5_K_M | 5.67 | 5.67 GB | 6.74 GB | 11.9 |
| Q4_K_M | 4.85 | 4.85 GB | 5.92 GB | 13.5 |
| MLX 4-bit | 4.5 | 4.50 GB | 5.57 GB | 14.4 |
| Q3_K_M | 3.91 | 3.91 GB | 4.98 GB | 16.1 |
Quantizing the weights is a speed change, and it is not subtle. Every token reads every weight, so halving the bytes per weight halves the time. Llama 3.1 8B at 2K context on a 100 GB/s laptop goes from 4.7 tokens per second at FP16 to 15 at Q4_K_M, a factor of 3.2, which is the difference between a model you cannot use and one you can. This is the entire reason local inference is possible on consumer hardware, and it is why the honest answer to “does quantization speed up inference” is yes, emphatically, when the asker means the weights.
Quantizing the KV cache is a capacity change wearing a speed change’s clothes. At that same 2K context, cutting the cache from 16 bits to 4 is worth 1.01x. Not 1.01x because the idea is bad, but because the cache is 5% of what the step reads and shrinking a twentieth of the work by three quarters is nothing. We measured this on our own machine rather than assuming it: KV quantization took peak memory from 8,235 MB to 7,568 MB at 16K context, an 8% saving that is real and reproduced in both arm orderings, and 1.4% at 4K. It buys context you could not otherwise hold. It does not buy throughput, and the tool that ships it says so too, quoting roughly 0.9x decode speed for a 5x smaller cache.
The crossover is set by the architecture, not by the parameter count. The cache only starts costing more to read than the weights past a context length you can compute: weight bytes divided by cache bytes per token. For Llama 3.2 3B at Q4 that is about 17K tokens. For Llama 3.1 8B, 36K. For Qwen3.5 9B it is 163K, because it keeps a growing cache on only 8 of its 32 layers and a fixed-size state on the rest, and for gpt-oss 20B, which alternates full attention with a 128-token sliding window, it is past half a million. A bigger model does not mean a cache-dominated one. Ask the attention design, not the size.
We published a speed claim for KV quantization and withdrew it. The first run showed 1.24x faster at 16K. Reversing which arm ran first made it disappear: the baseline arm had simply run on a hotter chip. Measured run-to-run spread on decode on that fanless machine reaches 82%, so a 24% effect was never separable from the noise, and the number should not have been published before it was reversed. That is worth knowing whenever you read a quantization benchmark, including this one: the memory numbers here are measurements, and the speed numbers are a bandwidth argument with one measured calibration point.
What the measurements say
Everything above that is a number came from one 16 GB M2 MacBook Air, which is the least forgiving machine you can run a model on, and is quoted here with the caveats attached rather than rounded off.
- Qwen3.5 4B, MLX 4-bit, short context31.4 tok/s
against a ceiling of about 35 tok/s from this arithmetic: the roofline is 13% optimistic, which is about as close as a bandwidth argument gets
- Qwen3.5 4B, GGUF Q4_K_M, llama.cpp17.0 tok/s
same model, same machine, comparable 4-bit quantisation, a different engine. The engine changed decode more than the quantisation scheme did
- KV quantisation at 16K context8235 to 7568 MB
an 8% memory saving, real and reproduced in both arm orderings. At 4K the same change saved 1.4%, because at 4K the cache is a rounding error
- KV quantisation, decode speedwithdrawn
first measured 1.24x faster at 16K, then reversed the arm order and it vanished: the baseline arm had simply run hot. Measured run-to-run spread on this machine reaches 82%, so the claim was never separable from noise
What this model leaves out
Quality, entirely. This page counts bytes and divides by bandwidth. It says nothing about what the model writes, and the ladder showing Q3_K_M as the fastest row is not a recommendation to use it. The rough consensus is that Q8 is indistinguishable from FP16, Q6 and Q5 are safe, Q4 is the default for good reason, and below that degradation becomes visible and gets worse the smaller the model. We did not measure any of that, so we are telling you it is the consensus rather than telling you it is true.
Batch size, which reverses the conclusion. At batch 1 the weights are read once per token, so they dominate. In a server running 64 requests at once, the weights are still read once per step but that step now produces 64 tokens, so the per-token weight cost falls by 64x while every request’s cache is still read in full. The cache dominates almost immediately, and KV quantization becomes the more valuable of the two. That is why FP8 KV cache is standard in datacenter serving and barely discussed on local forums. This page is about the machine on your desk.
Dequantization, which is not free. Quantized weights have to be unpacked before they are multiplied, and the roofline here charges nothing for that. On a machine with plenty of bandwidth and little compute the unpacking can eat the saving, which is one reason the measured 31.4 tokens per second fell 13% short of the 35 the arithmetic predicted.
Prefill, and a third kind of quantization. Reading a prompt is compute-bound, not memory-bound, so none of this describes time to first token. Activation quantization, running the arithmetic itself in FP8 or FP4 rather than merely storing weights that way, is the lever for that half, and it needs hardware with the matching tensor cores. It is a different change with a different answer.
Read next
- at-home/
- Will this model fit on my machine?
The same arithmetic asked the practical way: pick your machine and see what a given quantisation lets you load.
- MLX or llama.cpp on a Mac?
Where the 31.4 and 17.0 tokens per second came from, and why the engine moved decode more than the quantisation did.
- explainers/
- The KV cache is the thing you are actually renting
Why the cache grows with context and batch, which is what decides whether quantising it is worth anything.
- Two phases, two bottlenecks
Why decode is bandwidth divided by bytes, the assumption every number on this page rests on.
- feed/
- llama.cpp k-quants
Where Q4_K_M and its siblings are defined, and why the average is 4.85 bits per weight rather than 4.
- mlx-lm
Apple's MLX language model server, whose 4-bit format carries a scale and bias per group of 64 weights.
- Optimizing on-device inference for Apple silicon
Perplexity, 2026. Decode on a laptop reaches 90% of the chip's sustained weight-read rate, which is the roofline this page assumes.
- glossary/
- BF16
140 GB, for Llama 3 70B's weights at two bytes each, and 320 KiB of cache per token
- FP8
70 GB, for Llama 3 70B's weights in FP8, halving the read per decode step and the floor on inter-token latency with it
- FP4
35 GB, for Llama 3 70B's weights at half a byte each; the KV cache, if left in BF16, then dominates memory at long context
- Weight-only quantization
4x fewer bytes, per decode step at 4-bit weights, the whole gain, since the arithmetic is unchanged
- Activation quantization
2x compute, the tensor core rate gain from 8-bit matmuls, on top of half the bytes, which is why it helps prefill where weight-only does not
- KV cache quantization
160 KiB, per token for Llama 3 70B with an FP8 cache, half of BF16, so the same 180 GB holds twice the context