I previously posted the first results of my VLM benchmark. There were a few useful comments and observations I took into account, to revise and expand my benchmark:

• I initially did not take into account the Gemma 4 vision budget which defaults to 280, essentially making it useless. I have increased it to maximum level, with the following optimal setttings which were posted here recently: --image-min-tokens 560 --image-max-tokens 2240
• I used the -b 4096 -ub 4096 parameters to avoid splitting the image tokens into multiple blocks (default value is 512)
• Switched from ollama to llama.cpp
• I expanded my dataset from 20 to 30 images, to cover more use cases
• I expanded the benchmark to test the impact of thinking vs non-thinking
• The first benchmark only included Q4 quants; I expanded it to Q8 quants for small models
• The first benchmark only tested each image once; now 3x tests per image

In total, 23 models x 30 images x 3 tests = 2,070 tests (not including failures, tunings, re-runs), 60 to 70 inference hours.

I have three recommendations this time, one per hardware tier:

I noticed a few interesting outcomes, which I did not expect:

Thinking mode hurts vision. Every Qwen hybrid thinker scored higher with enable_thinking=false. This is because vision is perception, not reasoning. Thinking adds instability, timeouts, and empty outputs.

MoE size is misleading for vision. MoE models tie with much smaller dense models, and perform worse than equivalent dense models. It makes sense in retrospect if when you see that a MoE is a collection of small models. Their big total parameter count buys knowledge breadth, not perception depth which scales with density.

Q8 is not a guaranteed improvement. It improves Gemma 4 (more consistent, less hallucinations), cripples Qwen hybrid thinkers (they spend too long thinking, resulting in frequent timeouts). The only Q8 that's a strict win is Qwen3-VL 8B-Q8.

Here are the full quality ranking, sorted by effective score (raw × completion rate). σ = stability across 3 runs.

Here is bit more info about some of those models, that the above numbers cannot express, based on reading their actual output:

Qwen3.6-27B (Q4=16.9GB) : Best quality, best stability, no failures with thinking disabled. The no-thinking mode has a huge beneficial on speed, and avoids the timeouts due to reasoning too long. Gives very direct answers.

Qwen3.6-35B-A3B (Q4=21.9GB) : Based on the numbers it might appear like a good speedy alternatives, but it rarely performs better than smaller models. Biggest problem, apart from its size, is the huge variance and unpredictability of its responses. Skip it, not worth using MoE for vision.

Qwen3-VL-8B-Instruct (Q4=5.8GB Q8=8.1GB) : The only model with 100% reliability on Q8. Q8 brings big over Q4, for both quality and consistency.

Qwen3.5-4B (Q4=3.2GB) : Use with thinking disabled; when enabled, on dense images, it can easily exhaust its token budget and error, or timeout. Q8 was a lot worse than Q4, with again timeouts on dense images. None of those problems with Q4 non-thinking.

Test methodology

• specs: Apple M2 Max, 96GB RAM
• runtime: llama.cpp b9690 via llama-server
• models: 11 base models, Q4_K_M; Q8_0 added for 7 of the smaller ones
• hybrid thinking models (Qwen3.5/3.6) tested both with and without thinking enabled
• 30 images across screenshots, photos, posters, art, medical, scientific graphs, dense scenes, and multilingual content
• 3 runs per (model × image), median run scored
• hybrid scoring: 40% deterministic probes (OCR, counts, hallucination checks) + 60% LLM judge based on human created detailed ground truth description for each image
• timeout: 300s per call (fail fast on runaway thinking)