All models / Gemma 4 12B
Gemma 4 12B has 12B parameters. At Q4_K_M with 8K context it needs about 8.3 GB of memory. The smallest graphics card that runs it fully on the GPU is about 10 GB, such as the RTX 3080 10GB.
Check your own hardware| Quant | Bits per weight | Memory needed |
|---|---|---|
| Q8_0 | 8.5 | 13.8 GB |
| Q6_K | 6.56 | 10.8 GB |
| Q5_K_M | 5.69 | 9.5 GB |
| Q4_K_M | 4.85 | 8.3 GB |
| Q3_K_M | 3.91 | 6.9 GB |
| IQ2_M | 2.7 | 5.1 GB |
| Hardware | Memory | Best quant | Est. speed |
|---|---|---|---|
| RTX 3080 10GB | 10 GB | Q4_K_M | 63 tok/s |
| RTX 2080 Ti | 11 GB | Q5_K_M | 43 tok/s |
| RTX 3080 Ti | 12 GB | Q6_K | 56 tok/s |
| RTX 3080 12GB | 12 GB | Q6_K | 56 tok/s |
| RTX 5070 | 12 GB | Q6_K | 41 tok/s |
| RTX 4070 Ti | 12 GB | Q6_K | 31 tok/s |
| RTX 4070 Super | 12 GB | Q6_K | 31 tok/s |
| RTX 4070 | 12 GB | Q6_K | 31 tok/s |
| Arc B580 | 12 GB | Q6_K | 28 tok/s |
| RX 7700 XT | 12 GB | Q6_K | 26 tok/s |
| RTX 3060 12GB | 12 GB | Q6_K | 22 tok/s |
| RTX 5080 | 16 GB | Q8_0 | 45 tok/s |
| RTX 5070 Ti | 16 GB | Q8_0 | 42 tok/s |
| RTX 4080 Super | 16 GB | Q8_0 | 35 tok/s |
| RTX 4080 | 16 GB | Q8_0 | 34 tok/s |
| RTX 4070 Ti Super | 16 GB | Q8_0 | 32 tok/s |
| RX 9070 XT | 16 GB | Q8_0 | 30 tok/s |
| RX 9070 | 16 GB | Q8_0 | 30 tok/s |
| RX 7800 XT | 16 GB | Q8_0 | 29 tok/s |
| Arc A770 16GB | 16 GB | Q8_0 | 26 tok/s |
| RX 6800 XT | 16 GB | Q8_0 | 24 tok/s |
| RTX 5060 Ti 16GB | 16 GB | Q8_0 | 21 tok/s |
| RX 9060 XT 16GB | 16 GB | Q8_0 | 15 tok/s |
| RTX 4060 Ti 16GB | 16 GB | Q8_0 | 14 tok/s |
| M5 16GB | 16 GB | Q6_K | 9.3 tok/s |
| M4 16GB | 16 GB | Q6_K | 7.3 tok/s |
| RX 7900 XT | 20 GB | Q8_0 | 38 tok/s |
| RTX 4090 | 24 GB | Q8_0 | 47 tok/s |
| RTX 3090 Ti | 24 GB | Q8_0 | 47 tok/s |
| RX 7900 XTX | 24 GB | Q8_0 | 45 tok/s |
| RTX 3090 | 24 GB | Q8_0 | 44 tok/s |
| M4 Pro 24GB | 24 GB | Q8_0 | 13 tok/s |
| RTX 5090 | 32 GB | Q8_0 | 84 tok/s |
| M5 32GB | 32 GB | Q8_0 | 7.2 tok/s |
| M4 32GB | 32 GB | Q8_0 | 5.6 tok/s |
| M4 Max 36GB | 36 GB | Q8_0 | 19 tok/s |
| M4 Pro 48GB | 48 GB | Q8_0 | 13 tok/s |
| M4 Max 64GB | 64 GB | Q8_0 | 26 tok/s |
| Ryzen AI Max+ 395 64GB | 64 GB | Q8_0 | 12 tok/s |
| RTX PRO 6000 Blackwell | 96 GB | Q8_0 | 84 tok/s |
| M3 Ultra 96GB | 96 GB | Q8_0 | 39 tok/s |
| M4 Max 128GB | 128 GB | Q8_0 | 26 tok/s |
| DGX Spark 128GB | 128 GB | Q8_0 | 13 tok/s |
| Ryzen AI Max+ 395 128GB | 128 GB | Q8_0 | 12 tok/s |
| M3 Ultra 256GB | 256 GB | Q8_0 | 39 tok/s |
| M3 Ultra 512GB | 512 GB | Q8_0 | 39 tok/s |