Pick your graphics card or Mac. See which open models fit in your memory, at what quantization, and about how many tokens per second you'll get.
One short email when a new open model drops, with what it needs and who can run it.
Memory needed = model weights + context cache + about 0.5 GB of runtime overhead.
Weights = parameters × bits per weight ÷ 8. A 27B model at Q4_K_M (≈4.85 bits) is about 16.4 GB.
Speed comes from memory bandwidth. Each new token reads every active weight once, so tokens/sec ≈ usable bandwidth ÷ active weight size. Mixture-of-experts (MoE) models only read their active experts, which is why a 35B-A3B model runs like a 3B one.
When a model doesn't fit, llama.cpp and Ollama can keep part of it in system RAM. The part in RAM runs at RAM speed, so we blend the two. On Macs and other unified-memory machines, about 72% of RAM is usable for the model.
| Quant | Bits | Quality |
|---|---|---|
| Q8_0 | 8.5 | Near lossless |
| Q6_K | 6.6 | Excellent |
| Q5_K_M | 5.7 | Very good |
| Q4_K_M | 4.85 | Good, the usual pick |
| Q3_K_M | 3.9 | Noticeable loss |
| IQ2_M | 2.7 | Big loss, last resort |
These are estimates. Real numbers vary with the runtime, drivers and settings, usually within about 20%.
At Q4_K_M, plan on about 0.6 GB per billion parameters plus context. A 9B model fits in 8 GB, 27B–31B models need about 20 GB, and a 70B model needs about 45 GB.
Qwen3.6 35B-A3B and Gemma 4 26B A4B run well with light offload because they're MoE. Gemma 4 12B fits fully at Q8.
Qwen3.8 27B, Gemma 4 31B and Qwen3.6 27B at Q4_K_M, with room for 8K–32K of context.
Yes, with offload to system RAM, but it slows down. MoE models handle offload far better than dense models.
Yes. llama.cpp supports AMD through ROCm and Vulkan, and Intel Arc through Vulkan and SYCL. Speeds land a bit below NVIDIA at the same bandwidth.