Home / Best models for 32 GB GPUs
The best local AI models for a 32 GB GPU are the largest that fit fully in VRAM at Q4_K_M or better: Hermes 4.3 36B (Q5_K_M, about 42 tokens/sec on an RTX 5090), Qwen3.6 35B-A3B (Q6_K, over 150 tokens/sec on an RTX 5090), Gemma 4 31B (Q6_K, about 43 tokens/sec on an RTX 5090), Nemotron 3.5 Lightning 30B-A3B (Q6_K, over 150 tokens/sec on an RTX 5090). Estimates assume 8K context.
Check your exact card and RAM| Model | Best quant | Memory needed | Est. speed (RTX 5090) |
|---|---|---|---|
| Hermes 4.3 36B | Q5_K_M | 27.7 GB | 42 tok/s |
| Qwen3.6 35B-A3B | Q6_K | 29.7 GB | 150+ tok/s |
| Gemma 4 31B | Q6_K | 26.4 GB | 43 tok/s |
| Nemotron 3.5 Lightning 30B-A3B | Q6_K | 25.4 GB | 150+ tok/s |
| Granite 4.2 30B | Q6_K | 24.8 GB | 45 tok/s |
| Qwen3.8 27B | Q8_0 | 30.6 GB | 36 tok/s |
| Qwen3.6 27B | Q8_0 | 29.8 GB | 37 tok/s |
| Gemma 4 26B A4B | Q8_0 | 27.9 GB | 150+ tok/s |
| Mistral Small 3.1 24B | Q8_0 | 27.3 GB | 42 tok/s |
| gpt-oss-20b | Q8_0 | 23.2 GB | 150+ tok/s |
| Phi-4 14B | Q8_0 | 17.7 GB | 69 tok/s |
| Gemma 4 12B | Q8_0 | 13.8 GB | 84 tok/s |
| Qwen3.5 9B | Q8_0 | 11.3 GB | 104 tok/s |
| Gemma 4 E4B | Q8_0 | 9.3 GB | 150+ tok/s |
| Gemma 4 E2B | Q8_0 | 6.1 GB | 150+ tok/s |
| Ministral 3 3B | Q8_0 | 4.8 GB | 150+ tok/s |