We open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)

I'm one of the developers. We said in August it would go open source in September and it did last night. MIT or Apache-2.0, pick one. The repo you see is our internal repo, kernels included, so from now on everything happens in public.

It's an inference engine in Rust and C++ with our own CUDA kernels. One binary with OpenAI and Anthropic style APIs, loads GGUF and safetensors. We run about 300B tokens a year through it at work.

Some numbers: Qwen3.8-27B FP8 on one RTX PRO 6000, spec decoding off on every engine:

• vs vLLM faster in 13 of 13 cells, 1.02x to 1.19x (so not huge)
• vs SGLang faster in 10 of 13, behind in 2, level in 1
• vs llama.cpp Q8_0 faster in 13 of 13, 1.5x to 37x
• 32 clients at 1024 in / 1024 out: 1062 tok/s, vLLM 958, SGLang 844

Full board with the losses: https://truespar.com/paddock/benchmarks/qwen38-27b

What it does not do yet: No Mac, no ROCm, no Vulkan.

One model per GPU, no tensor parallel. CUDA only, Windows and Linux. Validated on Blackwell (5090, RTX PRO 4500/5000/6000, B200) and Ampere (an A6000 was the bring-up card, 30-series works). Ada kernels ship but nobody has run a board on them so the engine refuses to start unless you set PADDOCK_UNVALIDATED_ARCH=1. Hopper and A100 kernels are in the tree without a board.

https://github.com/truespar/paddock

Thankful for any help and input!