RunsLocalGPU checkerLocal AI toolsv2.1 · data Oct 16, 2026

All models / Mellum2.1 12B-A2.5B Thinking

What GPU do I need to run Mellum2.1 12B-A2.5B Thinking?

Mellum2.1 12B-A2.5B Thinking has 12B parameters (2.5B active per token, mixture-of-experts). At Q4_K_M with 8K context it needs about 7.9 GB of memory. The smallest graphics card that runs it fully on the GPU is about 10 GB, such as the RTX 3080 10GB. Because only 2.5B parameters are active, it still runs usably with offload to system RAM on cards like the RTX 5060 Ti 8GB.

Check your own hardware

Memory by quantization (8K context)

Hardware that runs it fully in fast memory

Runs with offload to system RAM (slower)

RTX 5060 Ti 8GB ~134 tok/s · RTX 5060 ~134 tok/s · RTX 3070 ~134 tok/s · RTX 3060 Ti ~134 tok/s · RTX 5050 ~104 tok/s · RTX 4060 Ti 8GB ~96 tok/s · RX 7600 ~96 tok/s · RTX 4060 ~92 tok/s

Assumes 32 GB of DDR5 system RAM for offload. Estimates only; real speeds usually land within about 20%.