Running Qwen3 30B A3B at 50 tok/s on RTX 5060 Ti

Experimented with some custom CUDA and C++ code that can now run a Qwen3-30B-A3B at 50-54 tok/s at float 8 on an RTX 5060 Ti with only 16 GB of VRAM. This speed is roughly 50% improvement to llama.cpp which runs at around 33-34 tok/s (with n-cpu-moe). These speedups come mostly from combining SOTA solutions I saw in papers in NeurIPS, ICML, and EuroSys

Engines like these allow for new local inference oppurtunities on consumer hardware, offering more private, cheaper, and greener alternative to centralized datacenters!

REPO: https://github.com/NikolayBlagoev/garlic-inference