Qwen3-VL 8B on a laptop vs Opus 5.5 / Sonnet 5 / GPT-5.6 on 137 messy documents: beat GPT-5.6 on tax forms, lost badly on Indian date formats[R]

I benchmarked Qwen3-VL 8B Instruct (Q4_K_M, Ollama, M5 24GB, ~30s/doc) against Claude Opus 5.5, Sonnet 5 and GPT-5.6 Terra on:

- receipts: CORD (Indonesia) and SROIE (Malaysia), 30 each

- 20 scanned 1980s-90s invoices , answer keys human-verified

- 32 real IRS forms, 4 damage levels (generated this week, so not in anyone's training data)

- 10 synthetic Indian bank statements, 15 CUAD contracts

Results (documents fully right):
- Opus 89%,
- Sonnet 85%,
- Qwen 8B 59%,
- GPT-5.6 Terra 57%.

Qwen-specific findings:

- W-2s: 21/32 fully right vs GPT-5.6 Terra 7/32

- Indian bank statements: 2/10. Every amount and balance correct, but dd-mm-yyyy read as mm-dd.

- Contracts (long text): 2/15, mostly wrong expiry dates

- The default qwen3-vl:8b tag in Ollama is the thinking variant and ignores think:false. On long contracts it spent all 4,096 tokens thinking and returned nothing. Use :8b-instruct.

Other things I didn't expect:

- GPT-5.6 Terra "corrects" unusual spellings (Rachael -> Rachel, Kelleyland -> Kellyland)

- Asking a model to check its own output changed almost nothing (119/137 identical)

- At least 4 of the 30 SROIE receipts have a wrong published answer key (e.g. B1750 where the receipt prints 81750)

Next I'm fine-tuning the 8B to fix the date and spelling failures and will post the result either way.

https://preview.redd.it/vmwm4920v8sh1.png?width=2200&format=png&auto=webp&s=173dcd046cdcb7c2d26ba9027f1ef4b9c24486e0

Prompts, keys, scorer tests and every raw output: https://github.com/TashonBraganca/messy-docs-bench