Skip to content
inference.academy

at-home/run-on/rtx4060ti-16

What can I run on an RTX 4060 Ti with 16 GB?

Of its 16 GB of VRAM, about 14 GB is left for a model's weights and cache once the driver and the engine have taken theirs. The largest model below that runs with room for a 8K conversation is gpt-oss 20B at MXFP4, as shipped, decoding at up to 18 tokens a second.

ModelBest fitDownloadLongest contextDecode, up to
Llama 3.2 3BQ8_03.4 GB90K tokens68 tok/s
Qwen3.5 4BQ8_04.3 GB291K tokens54 tok/s
Llama 3.1 8BQ8_08.5 GB41K tokens27 tok/s
Qwen3 8BQ8_08.7 GB35K tokens26 tok/s
Qwen3.5 9BQ8_09.6 GB132K tokens24 tok/s
Qwen3 14BQ6_K12.1 GB11K tokens19 tok/s
gpt-oss 20BMXFP4, as shipped12.8 GB48K tokens18 tok/s
Qwen3 32Bdoes not fit
Llama 3.3 70Bdoes not fit

Best fit is the highest-quality quantisation that still leaves room for a 8K context. It stops at Q8 rather than FP16, because Q8 is indistinguishable from FP16 in practice and reads half the bytes, so it decodes about twice as fast on a machine like this.

Decode is a ceiling: 80% of this machine’s 288 GB/s divided by the bytes a token reads, at the start of a conversation. On a measured 16 GB MacBook Air that arithmetic predicted 35.6 tokens a second and MLX delivered 31.4; llama.cpp delivered 17 on the same model. Expect your engine to land at or below it.

Longest context assumes an unquantised KV cache. Quantising the cache to Q8 roughly doubles it. Reading a long prompt also needs working memory the table does not count, so leave a margin.

Read next