Can a laptop do real inference experiments?

The result: 5.6 times faster with the cache, 95% interval 4.9 to 6.3, in 24 of 24 blocks. The design is the whole point. Each block runs the same eight-turn transcript twice, once with a fixed prefix so every turn extends the last, once with a fresh nonce at the head of every turn so nothing can be reused. Which arm runs first is randomised per block. Blocks alternate between an idle machine and one with three CPU workers grinding beside the server. The analysis, non-parametric throughout, was written down before the data existed, along with four predictions and the confidence attached to each. The cache removed a median 81% of prefill time, and the paired test came out at p of six in a hundred million. That claim now has an interval, and it holds on the worst machine most people own.

Two of the four predictions were wrong, and the wrong ones are the useful part. Contention slowed the uncached arm 75% and the cached arm 25%, as predicted; a broken cache costs most exactly when the machine is busiest. But the prediction that the speedup ratio would be larger under contention did not survive: arm order turned out to be a large nuisance factor, 6.6 times when the cached arm ran first against 4.2 times when it ran second, because the second arm of every block runs on a hotter chip, and randomisation had not balanced that across conditions. Stratified by order, the contention effect on the ratio disappeared. Without the order diagnostic, written into the plan because an earlier experiment had been fooled the same way, it would have been reported as real. And the prediction that memory-traffic signals would predict slow rounds better than CPU load was simply refuted: load average was the strongest predictor by a wide margin.

Five rules, each learned by getting it wrong. Randomise arm order and test for it; a KV-quantisation speedup of 1.24 times on this machine was published and withdrawn when it vanished with the arms reversed. Compare within a run, never across days; identical work took 16.6 seconds cold and 28.6 after an hour. Never report a difference smaller than the within-run spread, which reached 82% on decode under sustained load. Treat a crashed trial as data, recorded and excluded for cause, never silently re-run. And interleave conditions round-robin rather than running each condition’s trials together: a first attempt at a prompt ablation ran them in blocks, the server died partway, and the later variants were scored on crashed trials, producing a clean monotonic decline that was entirely an artefact of the clock.

What a laptop can measure is anything large and paired. A 5 times effect, an 8 times memory saving, whether a cache is being hit at all: these survive thermal noise because each block is its own control, run minutes apart. What it cannot measure is a 10% difference between two engines, or between two quantisations, or anything that needs the machine to be the same on Tuesday as it was on Monday. The whole study took 1.8 hours of wall time and a few hundred lines of Python. The expensive part was the discipline.