24+ tok/s from ~30B MoE models on an old GTX 1080 (8 GB VRAM, 128k context)

I got Qwen 3.6 35B-A3B and Gemma 4 26B-A4B running on a $200 secondhand machine (i7-6700 / GTX 1080 / 32 GB RAM) using llama.cpp (the TurboQuant/RotorQuant KV cache quantisation allows 128k context within the 8 GB VRAM).

Results (Q4_K_M models, 128k context):

The trick is MoE offloading: llama.cpp can park the cold expert weights in system RAM, and stream over PCIe to the GPU, while keeping hot layers + KV cache on GPU. The system is fully PCIe bandwidth-limited (GPU sits at ~40-50% utilisation while PCIe 3.0 x16 is maxed out).

Biggest finding: Gemma 4's MTP speculative decoding barely helps out of the box (~5% gain). Turns out llama.cpp unconditionally keeps the token embedding table on CPU. Normally that's fine (just a get_rows lookup), but Gemma 4's MTP assistant has a tied LM head - so every draft token does a full 262k×1024 matmul across PCIe. Forcing it onto GPU with --override-tensor-draft gives the real ~22% speedup and ~79% draft acceptance rate.

Setup pain points (Fedora 42 + Pascal GPU):

• Pin akmod-nvidia to 580xx branch (Pascal is going legacy)
• Force gcc-14 for CUDA 12.9 (newer gcc rejected)
• Patch CUDA's math_functions.h for glibc 2.41 compatibility
• Used the AtomicBot-ai/atomic-llama-cpp-turboquant fork for both TurboQuant cache + Gemma MTP support

Full blog post with all the grindy build details (every command, and the debugging deep-dive into the MTP embedding table issue)

I'm also planning a YouTube video walkthrough soon - I'll update when that's live.

Happy to answer questions about the setup.