All hardware / RTX 4060 Ti 8GB
With 8 GB of VRAM, the RTX 4060 Ti 8GB runs 4 of 22 popular open models fully in fast memory at Q4_K_M or better (8K context, assuming 32 GB of DDR5 system RAM). The largest that fit are Qwen3.5 9B (Q4_K_M), Gemma 4 E4B (Q6_K), Gemma 4 E2B (Q8_0). Fastest: Gemma 4 E2B at about 71 tokens/sec and Gemma 4 E4B at about 47 tokens/sec. 10 more run with offload to system RAM, but slower.
Check with your own RAM and context length| Model | Result | Best quant | Memory needed | Est. speed |
|---|---|---|---|---|
| Gemma 4 E2B | Runs on GPU | Q8_0 | 6.1 GB | 71 tok/s |
| Gemma 4 E4B | Runs on GPU | Q6_K | 7.4 GB | 47 tok/s |
| Ministral 3 3B | Runs on GPU | Q8_0 | 4.8 GB | 43 tok/s |
| Qwen3.5 9B | Runs on GPU | Q4_K_M | 6.9 GB | 29 tok/s |
| gpt-oss-20b | Needs offload | Q4_K_M | 13.6 GB | 33 tok/s |
| Qwen3.6 35B-A3B | Needs offload | Q4_K_M | 22.2 GB | 31 tok/s |
| Gemma 4 12B | Lower quality fit | Q3_K_M | 6.9 GB | 29 tok/s |
| Gemma 4 26B A4B | Needs offload | Q4_K_M | 16.4 GB | 28 tok/s |
| Phi-4 14B | Needs offload | Q4_K_M | 11.0 GB | 9.6 tok/s |
| Mistral Small 3.1 24B | Needs offload | Q4_K_M | 16.4 GB | 4.4 tok/s |
| Qwen3.6 27B | Needs offload | Q4_K_M | 17.5 GB | 3.8 tok/s |
| Qwen3.8 27B | Needs offload | Q4_K_M | 18.0 GB | 3.6 tok/s |
| Granite 4.2 30B | Needs offload | Q4_K_M | 18.6 GB | 3.4 tok/s |
| Gemma 4 31B | Needs offload | Q4_K_M | 19.8 GB | 3.1 tok/s |
| Hermes 4.3 36B | Needs offload | Q4_K_M | 23.9 GB | 2.5 tok/s |
| Llama 3.3 70B | Too big | Q4_K_M | 45.9 GB | — |
| Llama 4 Scout | Too big | Q4_K_M | 68.1 GB | — |
| gpt-oss-120b | Too big | Q4_K_M | 72.0 GB | — |
| Qwen3.5 122B-A10B | Too big | Q4_K_M | 75.3 GB | — |
| Qwen3.8-Flash-Next | Too big | Q4_K_M | 107.8 GB | — |
| GLM-5.3-Flash | Too big | Q4_K_M | 196.0 GB | — |
| DeepSeek V4.1-Flash | Too big | Q4_K_M | 335.8 GB | — |
OllamaLM StudioJanGPT4AllAnythingLLMllama.cppvLLMSGLangKoboldCppText Generation WebUILocalAI