Lesson learned. Don't blindly trust repos and make sure everything is stable for a long running (multi weeks) benchmark.

I posted previously my swe-verified django 100 tasks benchmark comparing different local models and quantization.

No new models for now, but a fix in my evaluation workflow that was unfortunately not stable during the weeks/months of me using it. I redid the evaluation on all runs and here are some noticeable changes:

• Flash Next is still king, but the benefit of xhigh vs medium reasoning effort is now properly showing.
• Same for 3.8 27B (however in everyday tasks I personally still prefer using medium)