Can a 35B MoE model really run on 12GB VRAM?

Yes, with the right quantization (IQ4_NL), native ROCm backend, and careful layer placement using --n-cpu-moe 26, it runs stable at up to 28 tok/s decode and 581 tok/s prefill without ever running out of memory.

What was the biggest bottleneck you found?

Not VRAM size but KV-cache and thread scheduling. Using --threads 4 and --no-kv-unified for a single user kept junction thermal at max 85°C and made 128K context stable.

Should I copy your llama.cpp config directly?

No. The config is a reference tuned for i5-11400F + RX 6700 XT. You must retest --n-cpu-moe, --threads, and batch sizes on your own hardware.

The 35B MoE model runs on an older architecture with 12GB of VRAM.

Home Profile Experiments Contact Experiment 001 Experiment 002 Experiment 003 Experiment 004 Experiment 005 Experiment 006 Experiment 007 Experiment 008 Experiment 009 Experiment 010 Experiment 011 Experiment 012 Experiment 013 Experiment 014 Experiment 015 Experiment 016 Experiment 017