Flyweight: open-source C++/CUDA engine for running MoE models bigger than your VRAM on one GPU + system RAM. First PyPI release, looking for contributors.

Been building this for a few months, mostly for myself, and it just got a proper release so figured I'd post it.

It's a native GGUF inference runtime with OpenAI/Anthropic-compatible APIs and a chat UI. The whole point is one consumer NVIDIA card + lots of RAM: MoE models that don't fit in VRAM run their experts on the CPU, or split with a hot set cached on the card. It figures out what fits at startup instead of you guessing offload layer counts.

Runs Qwen 3.x dense and MoE (incl. Qwen3.8-Flash-Next), DeepSeek-V4-Flash, Ling 3.0, K2-Horizon, Gemma 4, Laguna, Muse Glimmer. Image input via mmproj on the Qwen models. Also does Z-Image-Turbo image gen next to a chat model on the same card.

Numbers from my laptop (5070 Ti 12 GB, 60 GB RAM):

- Qwen3.8-Flash-Next IQ1_S: ~35 tok/s decode, ~475 tok/s prefill

- Qwen3.8-27B IQ2_XXS: ~40 tok/s

- DeepSeek-V4-Flash: 6-7 tok/s (that's basically the DRAM bandwidth limit)

- Z-Image 1024x1024 in ~15 s with a 35B loaded beside it

Stuff I think is neat:

- Kernels are compiled at runtime by NVRTC, so no CUDA toolkit in the wheel and no nvcc. Same kernel source compiles as plain C++ for a CPU-only backend.

- KV cache in f16 / q8_0 / TurboQuant 4-bit. On the dense 27B at 32K that's the difference between 10.8 and 22.9 tok/s, because it's what keeps the weights on the card.

- Tool calls are enforced by a sampler grammar, and temperature/penalties are clamped inside a call so edit tools reproduce file text exactly. Makes Claude Code / opencode a lot less flaky on small quants.

- Thinking budget is a hard cap, plus a stop_thinking endpoint to cut a stream over to the answer.

Full disclosure: a lot of this was written with AI help (Claude Code, mostly). I did the design, the measuring, and the arguing about what's actually faster; the AI did a lot of the typing. Every kernel is parity-tested against a reference and the perf numbers are real measurements, but if that's a dealbreaker for you, fair enough.

pip install flyweight-llm flyweight doctor flyweight serve model.gguf 

Linux x86-64 and Windows wheels. Needs the NVIDIA driver + CUDA toolkit (for NVRTC and headers), or --backend cpu.

https://github.com/yairpatch/flyweight (Apache-2.0)

Where I could use help

One-person project, one-person blind spots. PRs and issues welcome, especially:

- Runs on other hardware. Everything was measured on one Blackwell laptop and one Intel box. Ampere/Ada, AMD CPUs, 8 GB cards, all untested. Even just flyweight doctor output + a tok/s number in an issue helps.

- Windows users. CI passes, real usage is thin.

- macOS / ARM. No wheels, nobody's tried. The CPU backend should build.

- Models llama.cpp runs that this doesn't. Open an issue with the GGUF metadata.

- CPU expert kernels, especially the 1-3 bit IQ formats. That's the bottleneck on everything MoE.

- Anything you tripped over installing.

There's a plans/ dir with notes on what's been tried and what got dropped, so check there before proposing something. CONTRIBUTING.md has the rest.

Happy to answer questions.