With hopes of a Qwen3.8-35B-A3B release now mostly dashed, many people including myself are looking at fine-tunes and other variants of Qwen3.6-35B-A3B to run on VRAM-limited hardware. I decided to try to benchmark some of the top contenders: KAT-Coder, Ornith 1.5 and the very recent Tiel-Coder. I used the tool-eval-bench utility by SeraphimSerapis as the benchmark suite. It measures how well the different models handle tool calls, including some very hard scenarios.

TL;DR: Ornith 1.5 and Tiel-Coder (which is based on Ornith) were the tied winners in this benchmark. They scored well above Qwen3.6-27B and got pretty close to 3.8-27B. KAT Coder was also slightly better than the original 35B-A3B. Ornith-1.5-Heretic was a disappointment.

Some time ago I posted a similar tool evaluation benchmark of different Qwen3.6-35B-A3B quants. In hindsight, that didn't work so well, mainly because I was looking at too many variables (GGUF quant, KV quant, context depth/pressure) and the benchmark itself was quite noisy so it was hard to get clear results. I hope I did better this time!

Materials

I had access to a cluster of 32GB V100s. For this comparison, I selected 2-3 different quants per model, if possible from different providers. For comparison, I also included original Qwen3.6-35B-A3B as well as the dense 3.6-27B and 3.8-27B Qwens. I picked different quants around Q4 (15GB to 22GB GGUF files) because that's what many people seem to use. For the original Qwen models, I chose Unsloth UD-Q4 quants because they are well known. I also included the ByteShape CPU-5 quant of Qwen3.6-35B-A3B because that's the quant I've been using recently. Altogether I benchmarked 13 different GGUF files, with 5 runs per file for a total of 65 runs. Each run took around 4.5 hours GPU time, except the 27B ones took 7 hours or so. Total GPU time spent was well over 300 hours, including a few failed runs.

To run the models, I used llama.cpp version 0.1.0-dev (build 10433, commit 9b05354ec) dated 2026-08-14 and built with CUDA support. I used q8_0 KV cache (that's what VRAM-limited people like me often do) and set ubatch-size to 2048 because the benchmark does a lot of prompt processing. I did not bother with MTP or other speculative decoding. This is not a speed benchmark.

llama.cpp parameters: -m $GGUF --temperature 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 -ngl 99 --ubatch-size 2048 --fit-target 256 -ctk q8_0 -ctv q8_0 --port $PORT --seed $SEED

For the benchmark, I used tool-eval-bench 2.6.0. I set the context length to 262144 and context pressure to 50%. This means that the models were benchmarked at 50% context depth, i.e., around 128k of possibly distracting chat and tool call history.

tool-eval-bench parameters: --base-url $BASE\_URL --hardmode --weight-by-difficulty --backend llamacpp --context-size 262144 --context-pressure $CONTEXT\_PRESSURE --seed $SEED

Scoring metric

The metric I looked at is what tool-eval-bench reports as "total points". With --hardmode enabled, this version of tool-eval-bench performs 88 separate tests. Each test gives 2 points for a succesful tool use, 1 point for a partially correct tool use, 0 for failure. The theoretical maximum is in this case 88 * 2 = 176 points. tool-eval-bench also returns an overall score, but this is just a rounded percentage of total points and the rounding loses some precision, so I opted for the raw total points instead.

Results by model (averaged over all quants)

Here are the benchmark scores by model. I have averaged them over all the quants of the same model and all 5 runs per quant.

model_id	repetitions (n)	avg total_score	CI (95%)
Qwen3.8-27B	5	152.6	[149.4, 155.8]
Ornith-1.5	10	144.2	[141.7, 146.7]
Tiel-Coder	10	144.0	[141.8, 146.2]
Qwen3.6-27B	5	134.8	[131.2, 138.4]
KAT-Coder-V2.5-Dev	15	133.8	[131.8, 135.8]
Ornith-1.5-Heretic	10	132.2	[130.6, 133.8]
Qwen3.6-35B-A3B	10	131.5	[129.9, 133.1]

Results by specific quant

See the images. There are no big differences between quants of the same model, except possibly KAT-Coder, where the mudler APEX quants were somewhat better than bartowski's. Also, the ByteShape quant of Qwen3.6-35B-A3B was a bit better than Unsloth's, which was a nice surprise.

Raw results

If someone wants to take a deeper look, I've shared the CSV with the tool-eval-bench results here. This includes e.g. category-specific scores (i.e. how well the model did on specific kinds of tool calls) and total tokens; I did not look at those in my analysis.

Findings

• Of the original Qwen models, 3.6-35B-A3B gets the lowest score, 3.8-27B the highest, with 3.6-27B landing in between. This is as expected and indicates that the benchmark is giving a useful signal.
• Ornith-1.5 and Tiel-Coder are practically tied. They are the clear winners out of the 35B-A3B variants. They scored above 3.6-27B but below 3.8-27B.
• KAT-Coder was possibly a bit better than the original 35B-A3B, but the confidence intervals overlap.
• Ornith-1.5-Heretic was a disappointment, much worse than plain Ornith.

Caveats

This benchmark relies entirely on the tool-eval-bench tasks and how the results are graded. It may or may not be representative of real tool use performance. To me it seems that the author or tool-eval-bench has done a great job in coming up with realistic looking tool call tasks, including some really hard ones enabled using --hardmode. I relied on the --context-pressure setting in tool-eval-bench, which (in my limited understanding) populates the context with realistic looking conversation and tool call history that could confuse the model.

Tool calls are not everything. If you are doing agentic coding, also the coding quality matters a lot. I did not measure it in this benchmark except very indirectly. There are other benchmarks for that purpose.

There was substantial variation and noise in the benchmark scores, which I tried to alleviate by repeating the runs with different seeds, averaging, and calculating confidence intervals.

In the X/Y plot where the X axis represents size, I did not check whether the model includes MTP heads or not, I just looked at raw GGUF file size. This is slightly unfair to the MTP-enabled models because their files are larger but MTP does not increase quality, only generation speed.

No AI was used for writing this post. I did use Tiel-Coder to help me with plotting the results. Also reused some of my own earlier writing. I am not in any way affiliated with the model or quant makers or the benchmark suite.