Running models at home

Pick your Mac or GPU, a model, a quantisation and a context length. Get a yes or no, the headroom left, the longest context that fits, and a decode speed estimate calibrated on a measured laptop.

The same 4B model through both engines on a 16 GB MacBook Air, measured with the order reversed. Plus the llama.cpp flag that took the server from 9.4 GB to 1.2 GB, and why RSS lies about memory on Apple silicon.

A 5.6x prefix-cache result with a confidence interval, measured on a fanless M2 Air. The five rules it cost to get there, and the two published claims that had to be withdrawn on the way.