at-home/run-on/rtx3090-24

What can I run on an RTX 3090 with 24 GB?

Of its 24 GB of VRAM, about 22 GB is left for a model's weights and cache once the driver and the engine have taken theirs. The largest model below that runs with room for a 8K conversation is Qwen3 32B at Q3_K_M, decoding at up to 47 tokens a second, though only at Q3, where small models visibly lose quality.

Model	Best fit	Download	Longest context	Decode, up to
Llama 3.2 3B	Q8_0	3.4 GB	158K tokens	220 tok/s
Qwen3.5 4B	Q8_0	4.3 GB	529K tokens	176 tok/s
Llama 3.1 8B	Q8_0	8.5 GB	101K tokens	88 tok/s
Qwen3 8B	Q8_0	8.7 GB	88K tokens	86 tok/s
Qwen3.5 9B	Q8_0	9.6 GB	371K tokens	78 tok/s
Qwen3 14B	Q8_0	15.7 GB	37K tokens	48 tok/s
gpt-oss 20B	MXFP4, as shipped	12.8 GB	366K tokens	59 tok/s
Qwen3 32B	Q3_K_M	16.0 GB	22K tokens	47 tok/s
Llama 3.3 70B	does not fit

Best fit is the highest-quality quantisation that still leaves room for a 8K context. It stops at Q8 rather than FP16, because Q8 is indistinguishable from FP16 in practice and reads half the bytes, so it decodes about twice as fast on a machine like this.

Decode is a ceiling: 80% of this machine’s 936 GB/s divided by the bytes a token reads, at the start of a conversation. On a measured 16 GB MacBook Air that arithmetic predicted 35.6 tokens a second and MLX delivered 31.4; llama.cpp delivered 17 on the same model. Expect your engine to land at or below it.

Longest context assumes an unquantised KV cache. Quantising the cache to Q8 roughly doubles it. Reading a long prompt also needs working memory the table does not count, so leave a margin.

What can I run on an RTX 3090 with 24 GB?

Read next