Running models at home
You have a Mac, or a gaming GPU, and a model you want to run on it. The questions are the same every time: will it fit, how fast will it go, which of the three engines to use, and why the number you got yesterday is not the number you got today.
Everything here was measured on a fanless 16 GB MacBook Air, the least forgiving machine you can run a model on, which is the point. If a result holds there it holds on yours. Each page gives you a number for your own machine and then points into the explainers for the reason behind it.
- calculatorWill this model fit on my machine?
Pick your Mac or GPU, a model, a quantisation and a context length. Get a yes or no, the headroom left, the longest context that fits, and a decode speed estimate calibrated on a measured laptop.
- measuredMLX or llama.cpp on a Mac?
The same 4B model through both engines on a 16 GB MacBook Air, measured with the order reversed. Plus the llama.cpp flag that took the server from 9.4 GB to 1.2 GB, and why RSS lies about memory on Apple silicon.
- measuredCan a laptop do real inference experiments?
A 5.6x prefix-cache result with a confidence interval, measured on a fanless M2 Air. The five rules it cost to get there, and the two published claims that had to be withdrawn on the way.