Skip to content
inference.academy

at-home/run-on/rtx3060-12

What can I run on an RTX 3060 with 12 GB?

Of its 12 GB of VRAM, about 10 GB is left for a model's weights and cache once the driver and the engine have taken theirs. The largest model below that runs with room for a 8K conversation is Qwen3 14B at Q3_K_M, decoding at up to 40 tokens a second, though only at Q3, where small models visibly lose quality.

ModelBest fitDownloadLongest contextDecode, up to
Llama 3.2 3BQ8_03.4 GB56K tokens85 tok/s
Qwen3.5 4BQ8_04.3 GB171K tokens68 tok/s
Llama 3.1 8BQ8_08.5 GB11K tokens34 tok/s
Qwen3 8BQ8_08.7 GB9K tokens33 tok/s
Qwen3.5 9BQ8_09.6 GB13K tokens30 tok/s
Qwen3 14BQ3_K_M7.2 GB16K tokens40 tok/s
gpt-oss 20Bdoes not fit
Qwen3 32Bdoes not fit
Llama 3.3 70Bdoes not fit

Best fit is the highest-quality quantisation that still leaves room for a 8K context. It stops at Q8 rather than FP16, because Q8 is indistinguishable from FP16 in practice and reads half the bytes, so it decodes about twice as fast on a machine like this.

Decode is a ceiling: 80% of this machine’s 360 GB/s divided by the bytes a token reads, at the start of a conversation. On a measured 16 GB MacBook Air that arithmetic predicted 35.6 tokens a second and MLX delivered 31.4; llama.cpp delivered 17 on the same model. Expect your engine to land at or below it.

Longest context assumes an unquantised KV cache. Quantising the cache to Q8 roughly doubles it. Reading a long prompt also needs working memory the table does not count, so leave a margin.

Read next