at-home/run-on/rtx5090-32

What can I run on an RTX 5090 with 32 GB?

Of its 32 GB of VRAM, about 30 GB is left for a model's weights and cache once the driver and the engine have taken theirs. The largest model below that runs with room for a 8K conversation is Qwen3 32B at Q6_K, decoding at up to 53 tokens a second.

Model	Best fit	Download	Longest context	Decode, up to
Llama 3.2 3B	Q8_0	3.4 GB	226K tokens	422 tok/s
Qwen3.5 4B	Q8_0	4.3 GB	767K tokens	337 tok/s
Llama 3.1 8B	Q8_0	8.5 GB	160K tokens	169 tok/s
Qwen3 8B	Q8_0	8.7 GB	141K tokens	165 tok/s
Qwen3.5 9B	Q8_0	9.6 GB	609K tokens	150 tok/s
Qwen3 14B	Q8_0	15.7 GB	85K tokens	91 tok/s
gpt-oss 20B	MXFP4, as shipped	12.8 GB	683K tokens	112 tok/s
Qwen3 32B	Q6_K	26.9 GB	12K tokens	53 tok/s
Llama 3.3 70B	does not fit

Best fit is the highest-quality quantisation that still leaves room for a 8K context. It stops at Q8 rather than FP16, because Q8 is indistinguishable from FP16 in practice and reads half the bytes, so it decodes about twice as fast on a machine like this.

Decode is a ceiling: 80% of this machine’s 1792 GB/s divided by the bytes a token reads, at the start of a conversation. On a measured 16 GB MacBook Air that arithmetic predicted 35.6 tokens a second and MLX delivered 31.4; llama.cpp delivered 17 on the same model. Expect your engine to land at or below it.

Longest context assumes an unquantised KV cache. Quantising the cache to Q8 roughly doubles it. Reading a long prompt also needs working memory the table does not count, so leave a margin.

What can I run on an RTX 5090 with 32 GB?

Read next