Chalk one up for the frontier model...

I just spent the last 2 hours of my life on a Friday night debugging a strange error in a prod CLI app. EF core was receive a readonlyspan during a Contains query. Normally, this query converted to a WHERE [col] IN (...), but for some reason, after an update, it started choking, despite no code change. The same exact code runs in a separate website docker image fine, no problem.

I put Qwen 3.8 27b (q8 model, f16 kv) on it and it spun its wheels going down 4 different paths. Finally I got sick of it and switched models to GPT Sol with the full context available. It found the fix in 2 minutes.

The issue was that I had recently installed .NET 10 SDK on this machine, and the lack of a global.json file pinning the SDK meant that when the CLI was rebuilt locally, it used the c# 14 compiler, which introduced first-class Span<T> support, thus borking the EF query.

The wasted time isn't what bothers me here. It's that Sol was able to pinpoint the issue so much incredibly faster than Qwen, shattering my image of Qwen 3.8 as a fairly competent model. Benchmarks aren't everything folks. Real world use cases are the final say here.

I'm posting this in /r/LocalLLaMA because I'm a big local LLM fan, but sometimes, it's worth reminding ourselves of the gap that really exists, no matter how much we might want to wish it away.

Edit: Folks, some of you are missing the point. I tried Sol because it was next in my favorites list. Yes, I could have tried GLM, Kimi, or a number of others. The point would stand that no matter how great 27b is, it's not even remotely "near-frontier" in many cases, despite claims otherwise. Optimism has clouded our vision, somewhat. This was not meant to be a Sol promotion.