Benchmarking or benchmarketing?

Maybe I’m getting cynical, but LLM benchmarking is starting to feel less like measurement and more like marketing and positioning. Every week there’s a new leaderboard score, new chart, new eval suite, or some claim that a model is suddenly the best.

It feels like benchmarks have become part of the launch narrative and we've become hyper-tuned to them as framing not for “here’s how this model performs,” but “here’s the scoreboard that makes this release look inevitable.”

I’m not saying benchmarks are useless. I still look at them, but more and more skeptically every day. Wondering from y'all: which benchmarks do you find to actually be meaningful for local models - not just benchmarketing slop?