All models / gpt-oss-120b
gpt-oss-120b has 117B parameters (5.1B active per token, mixture-of-experts). At Q4_K_M with 8K context it needs about 72.0 GB of memory. The smallest graphics card that runs it fully on the GPU is about 96 GB, such as the RTX PRO 6000 Blackwell.
Check your own hardware| Quant | Bits per weight | Memory needed |
|---|---|---|
| Q8_0 | 8.5 | 125.4 GB |
| Q6_K | 6.56 | 97.0 GB |
| Q5_K_M | 5.69 | 84.3 GB |
| Q4_K_M | 4.85 | 72.0 GB |
| Q3_K_M | 3.91 | 58.3 GB |
| IQ2_M | 2.7 | 40.6 GB |
| Hardware | Memory | Best quant | Est. speed |
|---|---|---|---|
| RTX PRO 6000 Blackwell | 96 GB | Q5_K_M | 150+ tok/s |
| M4 Max 128GB | 128 GB | Q5_K_M | 90 tok/s |
| DGX Spark 128GB | 128 GB | Q5_K_M | 45 tok/s |
| Ryzen AI Max+ 395 128GB | 128 GB | Q5_K_M | 42 tok/s |
| M3 Ultra 256GB | 256 GB | Q8_0 | 91 tok/s |
| M3 Ultra 512GB | 512 GB | Q8_0 | 91 tok/s |