All models / Qwen3.5 9B
Qwen3.5 9B has 9.7B parameters. At Q4_K_M with 8K context it needs about 6.9 GB of memory. The smallest graphics card that runs it fully on the GPU is about 8 GB, such as the RTX 4060.
Check your own hardware| Quant | Bits per weight | Memory needed |
|---|---|---|
| Q8_0 | 8.5 | 11.3 GB |
| Q6_K | 6.56 | 9.0 GB |
| Q5_K_M | 5.69 | 7.9 GB |
| Q4_K_M | 4.85 | 6.9 GB |
| Q3_K_M | 3.91 | 5.7 GB |
| IQ2_M | 2.7 | 4.3 GB |
| Hardware | Memory | Best quant | Est. speed |
|---|---|---|---|
| RTX 5060 Ti 8GB | 8 GB | Q4_K_M | 46 tok/s |
| RTX 5060 | 8 GB | Q4_K_M | 46 tok/s |
| RTX 3070 | 8 GB | Q4_K_M | 46 tok/s |
| RTX 3060 Ti | 8 GB | Q4_K_M | 46 tok/s |
| RTX 5050 | 8 GB | Q4_K_M | 33 tok/s |
| RTX 4060 Ti 8GB | 8 GB | Q4_K_M | 29 tok/s |
| RX 7600 | 8 GB | Q4_K_M | 29 tok/s |
| RTX 4060 | 8 GB | Q4_K_M | 28 tok/s |
| RTX 3080 10GB | 10 GB | Q6_K | 57 tok/s |
| RTX 2080 Ti | 11 GB | Q6_K | 46 tok/s |
| RTX 3080 Ti | 12 GB | Q8_0 | 53 tok/s |
| RTX 3080 12GB | 12 GB | Q8_0 | 53 tok/s |
| RTX 5070 | 12 GB | Q8_0 | 39 tok/s |
| RTX 4070 Ti | 12 GB | Q8_0 | 29 tok/s |
| RTX 4070 Super | 12 GB | Q8_0 | 29 tok/s |
| RTX 4070 | 12 GB | Q8_0 | 29 tok/s |
| Arc B580 | 12 GB | Q8_0 | 27 tok/s |
| RX 7700 XT | 12 GB | Q8_0 | 25 tok/s |
| RTX 3060 12GB | 12 GB | Q8_0 | 21 tok/s |
| RTX 5080 | 16 GB | Q8_0 | 56 tok/s |
| RTX 5070 Ti | 16 GB | Q8_0 | 52 tok/s |
| RTX 4080 Super | 16 GB | Q8_0 | 43 tok/s |
| RTX 4080 | 16 GB | Q8_0 | 42 tok/s |
| RTX 4070 Ti Super | 16 GB | Q8_0 | 39 tok/s |
| RX 9070 XT | 16 GB | Q8_0 | 37 tok/s |
| RX 9070 | 16 GB | Q8_0 | 37 tok/s |
| RX 7800 XT | 16 GB | Q8_0 | 36 tok/s |
| Arc A770 16GB | 16 GB | Q8_0 | 33 tok/s |
| RX 6800 XT | 16 GB | Q8_0 | 30 tok/s |
| RTX 5060 Ti 16GB | 16 GB | Q8_0 | 26 tok/s |
| RX 9060 XT 16GB | 16 GB | Q8_0 | 19 tok/s |
| RTX 4060 Ti 16GB | 16 GB | Q8_0 | 17 tok/s |
| M5 16GB | 16 GB | Q8_0 | 8.9 tok/s |
| M4 16GB | 16 GB | Q8_0 | 7.0 tok/s |
| RX 7900 XT | 20 GB | Q8_0 | 47 tok/s |
| RTX 4090 | 24 GB | Q8_0 | 59 tok/s |
| RTX 3090 Ti | 24 GB | Q8_0 | 59 tok/s |
| RX 7900 XTX | 24 GB | Q8_0 | 56 tok/s |
| RTX 3090 | 24 GB | Q8_0 | 54 tok/s |
| M4 Pro 24GB | 24 GB | Q8_0 | 16 tok/s |
| RTX 5090 | 32 GB | Q8_0 | 104 tok/s |
| M5 32GB | 32 GB | Q8_0 | 8.9 tok/s |
| M4 32GB | 32 GB | Q8_0 | 7.0 tok/s |
| M4 Max 36GB | 36 GB | Q8_0 | 24 tok/s |
| M4 Pro 48GB | 48 GB | Q8_0 | 16 tok/s |
| M4 Max 64GB | 64 GB | Q8_0 | 32 tok/s |
| Ryzen AI Max+ 395 64GB | 64 GB | Q8_0 | 15 tok/s |
| RTX PRO 6000 Blackwell | 96 GB | Q8_0 | 104 tok/s |
| M3 Ultra 96GB | 96 GB | Q8_0 | 48 tok/s |
| M4 Max 128GB | 128 GB | Q8_0 | 32 tok/s |
| DGX Spark 128GB | 128 GB | Q8_0 | 16 tok/s |
| Ryzen AI Max+ 395 128GB | 128 GB | Q8_0 | 15 tok/s |
| M3 Ultra 256GB | 256 GB | Q8_0 | 48 tok/s |
| M3 Ultra 512GB | 512 GB | Q8_0 | 48 tok/s |