An example

TL;DR

I have open-sourced NInfer, a from-scratch C++/CUDA inference engine currently specialized for two exact Qwen3.6 checkpoints on a single RTX 5090. Both the engine and the converted model artifacts are publicly available:

Github: https://github.com/Neroued/ninfer

The main result:

Qwen3.6-35B-A3B sustained 542 tok/s while generating a full 65,536 token completion, on a single RTX 5090, single request.

My goal was to find out how fast inference can get on a single GPU (in my case RTX 5090), with a fixed model and fixed weights, after deep, end-to-end optimization. To that end, I threw everything I could at it and built the entire pipeline from scratch: custom quantization, weight layout design, per op kernel optimization, kernel fusion, a dedicated LM head draft, and so on. NInfer is not a general inference engine, it's designed just for certain model artifacts.
The currently supported models are:

• Qwen3.6-27B
• Qwen3.6-35B-A3B

Both converted model artifacts are available on Hugging Face. Under NInfer's quantization scheme, the published artifacts are 16.29 GiB (~5.03 bpw) for Qwen3.6-27B and 20.84 GiB (~4.97 bpw) for Qwen3.6-35B-A3B.

The Qwen3.6-35B-A3B results:

All MTP results below use a draft window of 3 and NInfer’s optimized LM-head draft path. Each result is the mean ± sample standard deviation across five fixed seeds, after one warm-up run.

Long-reasoning runs:

Completion length	Decode speed	MTP acceptance
65,536 tokens	542.8 ± 12.5 tok/s	73.0%
~55,171 tokens	572.9 ± 9.1 tok/s	77.7%
~8,675 tokens	634.3 ± 14.2 tok/s	82.7%

I also ran a mixed set of code, translation, story, and structured output prompts:

Workload	Decode speed	MTP acceptance
Code	576.5 ± 21.7 tok/s	71.0%
Translation	559.3 ± 28.1 tok/s	66.6%
Story	395.9 ± 30.9 tok/s	37.7%
Structured output	661.2 ± 29.5 tok/s	87.2%

MTP0 context-length scaling:

Prompt length	Prefill speed	Decode speed
7,680	15,544 tok/s	271.1 tok/s
64,512	10,809 tok/s	242.9 tok/s
130,048	7,828 tok/s	219.4 tok/s
260,096	5,157 tok/s	188.2 tok/s

The Qwen3.6-27B results:

NInfer also performs strongly on the 27B dense model:

Workload	Decode speed	MTP acceptance
Long-reasoning	174.2 ± 3.3 tok/s	79.9%
Code	163.9 ± 6.2 tok/s	72.5%
Translation	153.6 ± 11.7 tok/s	65.7%
Story	110.4 ± 9.2 tok/s	37.9%
Structured output	189.1 ± 15.7 tok/s	88.9%

Capability scores:

I also ran the published artifacts through AIME25, AIME26, and GPQA-Diamond (0-shot, rule scoring, single sample, thinking enabled, MTP=3).

Model	AIME25	AIME26	GPQA-Diamond
Qwen3.6-27B-NInfer	26/30	28/30	172/198
Qwen3.6-35B-A3B-NInfer	27/30	27/30	169/198

Full evaluation configurations are availble in the repository.

Capabilities & limitations

For both supported models, NInfer handles text, image, and video input, with OpenAI- and Anthropic-compatible HTTP endpoints. It supports limited prefix caching and a range of sampling parameters. With INT8 KV cache enabled on the RTX 5090's 32 GB, both models can reach their full native context length of 262,144 tokens.

Known limitations:

• Only the two listed models are supported. If a stronger, locally-suitable model drops, I'll jump on it immediately.
• Only RTX 5090 (sm_120a). RTX PRO 6000 should also work, though some kernel tuning may be suboptimal.
• No continuous batching. (If no new models land soon, I may look into adding it.)

I'd genuinely like to see another inference engine match or beat these numbers — similar quantization size, single request, single RTX 5090, Qwen3.6-35B-A3B. Bring it on.