Skip to content
inference.academy

at-home/run-on/rtx4090-24

What can I run on an RTX 4090 with 24 GB?

Of its 24 GB of VRAM, about 22 GB is left for a model's weights and cache once the driver and the engine have taken theirs. The largest model below that runs with room for a 8K conversation is Qwen3 32B at Q3_K_M, decoding at up to 50 tokens a second, though only at Q3, where small models visibly lose quality.

ModelBest fitDownloadLongest contextDecode, up to
Llama 3.2 3BQ8_03.4 GB158K tokens237 tok/s
Qwen3.5 4BQ8_04.3 GB529K tokens190 tok/s
Llama 3.1 8BQ8_08.5 GB101K tokens95 tok/s
Qwen3 8BQ8_08.7 GB88K tokens93 tok/s
Qwen3.5 9BQ8_09.6 GB371K tokens84 tok/s
Qwen3 14BQ8_015.7 GB37K tokens51 tok/s
gpt-oss 20BMXFP4, as shipped12.8 GB366K tokens63 tok/s
Qwen3 32BQ3_K_M16.0 GB22K tokens50 tok/s
Llama 3.3 70Bdoes not fit

Best fit is the highest-quality quantisation that still leaves room for a 8K context. It stops at Q8 rather than FP16, because Q8 is indistinguishable from FP16 in practice and reads half the bytes, so it decodes about twice as fast on a machine like this.

Decode is a ceiling: 80% of this machine’s 1008 GB/s divided by the bytes a token reads, at the start of a conversation. On a measured 16 GB MacBook Air that arithmetic predicted 35.6 tokens a second and MLX delivered 31.4; llama.cpp delivered 17 on the same model. Expect your engine to land at or below it.

Longest context assumes an unquantised KV cache. Quantising the cache to Q8 roughly doubles it. Reading a long prompt also needs working memory the table does not count, so leave a margin.

Read next