Explainer
Cross-Model KV Cache Transfer in LLM Families
A cascade answers with a 14B model and escalates the hard cases to a 32B one. Every escalation used to mean the 32B model reading the whole conversation again before it could say a word, because a KV cache belongs to the model that wrote it. Within one family, it turns out, the big model’s keys and values are close to a linear function of the small model’s. Below, the same handoff priced both ways.
Prefix caching saves the prefill when the next request starts the same way, for the same model. Routing between sizes, cost-quality cascades and mid-conversation upgrades all break that condition on purpose, and each swap pays a full prefill on the receiving side. The map here is fit once per pair, per target layer and head, from 500 calibration sequences, and is then a small multiply per token.
At 32K tokens the map is 372x cheaper than re-prefilling on the roofline, Qwen3 32B’s first token comes 2.62 s sooner, and a cascade escalating 30% of requests spends 41% less on prefill than one that re-prefills, 57% less than sending everything to the large model. It keeps 97.6% of Qwen3 32B’s accuracy: 97.6% on ARC-C and HellaSwag, 98.5% WinoGrande, 95.0% MMLU, 95.6% GSM8K.
The cache is linear across sizes, mostly. Matched pairs in a family share the number of KV heads and the head dimension, so a head in the large model can be predicted from heads in the small one. On Qwen3 14B to 32B, a single source layer explains 56% of the variance in the target’s keys and 32% in its values; the best eight source layers together explain 79% and 65%. That is enough: a per-head ridge regression on those layers keeps 97.6% of the 32B model’s accuracy across five benchmarks. Keys are mapped with their rotary position encoding stripped and re-applied afterwards, which is what lets one fit serve every context length.
The map is a rounding error next to the prefill it replaces. Re-prefilling 32K tokens on Qwen3 32B is two FLOPs per parameter per token plus the attention over the context, which at that length is a third of the arithmetic. The map is, per token, a matrix the size of eight head dimensions by one, for every target layer and head, keys and values: nearly four hundred times less work on the roofline. The paper’s implementation measured 25x at 32K and 4x at 64 tokens, where fixed overhead dominates. Either number turns an escalation from seconds of silence into a handoff nobody notices.
Two of six pairs break, and the paper says so. Ministral 3 into 14B, from either the 3B or the 8B, keeps only 42 to 44% of accuracy on average, which after normalising to the random-guess floor is 11 to 15%. A small nonlinear map recovers 24 and 37 points of HellaSwag. The predictor of whether a pair works is not how well the regression fits, which correlates negatively with retention, but whether attention outputs still point the same way afterwards. Some families are linear across sizes and some are not, and the only way to know is to test the pair.
The six pairs, as the paper reports them
Retention is the mapped cache’s accuracy as a share of the large model’s own, averaged over the benchmarks the paper ran. k is how many of the small model’s layers feed each layer of the map.
| pair | layers | k | retained | what the average hides |
|---|---|---|---|---|
| Qwen3 14B to 32B | 40 to 64 | 8 | 97.6% | 97.6% on ARC-C and HellaSwag, 98.5% WinoGrande, 95.0% MMLU, 95.6% GSM8K |
| Qwen3 8B to 32B | 36 to 64 | 12 | 87.5% | 94.0% ARC-C, 95.2% HellaSwag, 91.0% WinoGrande, 88.5% MMLU, and GSM8K down to 68.8% |
| Llama 3.1 8B to 70B | 32 to 80 | 20 | 72.8% | 90.9% HellaSwag and 94.4% WinoGrande, but the reasoning benchmarks pull the average to 72.8% |
| Ministral 3 3B to 8B | 26 to 34 | 26 | 76.2% | 90.6% HellaSwag, 93.3% WinoGrande, 91.3% MMLU, every one of the small model's 26 layers used as a source |
| Ministral 3 3B to 14B | 26 to 40 | 20 | 43% | 42 to 44% average, 11 to 15% once normalised to the random-guess floor; a small nonlinear map recovers 24 points of HellaSwag |
| Ministral 3 8B to 14B | 34 to 40 | 20 | 43% | 42 to 44% average, 11 to 15% floor-normalised; the nonlinear map recovers 37 points of HellaSwag |
What this model leaves out
The speedup here is a ratio of roofline times, and the roofline is kind to the map: a dense per-head multiply reaches peak more easily than the paper’s unfused implementation did. The hollow marks on the chart are what they measured, on a different machine, and the gap between the two is engineering not yet done. The fixed cost that made the 64-token handoff only 4x faster is the same kind of thing.
Accuracy is quoted, not simulated. No roofline says whether a mapped cache produces the right answer, and the paper’s retention numbers come from single-turn benchmarks with the question in the prompt. Over ten turns of a conversation the small-to-large gap widened by 1.7 points; the reverse direction, large to small, drifted a third of a point per turn.
Decode is not counted in the cascade cost, because it is the same work however the large model got its cache. The escalated request’s time to first token is the honest comparison: a handoff that takes seconds is a stall the user sees, and one that takes milliseconds is not. Both models also share the same four GPUs here, which a real cascade would not do; the cache has to cross a network first, which is the disaggregation page’s problem and not a small one at 32K tokens.
Read next
- explainers/
- Compute the system prompt once
Prefix caching within one model, the condition this page breaks on purpose.
- Two phases, two bottlenecks
The roofline the prefill costs come from.
- The KV cache is the thing you are actually renting
Why the cache is per model, per layer, per head, and how big it gets.
- feed/
- glossary/