Why decontamination reports can't fix benchmark contamination, and what an evaluator has to do instead [D]

In February OpenAI stopped reporting SWE-bench Verified and recommended other labs stop too. Every frontier model they tested could reproduce the human-written reference fix, or verbatim details of the problem statement, for some tasks. Progress had slowed to six points in six months and it wasn't clear how much of the remaining score was capability at all. The lab that built the benchmark, and had every reason to keep it, is the one that retired it.

The usual answer is a decontamination report: the lab searched its training data and found nothing. I wrote up why that can't work, for three reasons that don't go away with better search:

- The lab checks itself. Nobody outside the lab has the corpus, so nobody can rerun the search.

- The corpus can't be disclosed. It's a list of every copyrighted work in it, and publishing that is litigation exposure.

- Matching misses most of it. Paraphrases, forum walkthroughs, solutions on GitHub, synthetic data generated from the benchmark. A model learns the answers from any of those without sharing an n-gram.

Commitments and private set intersection help less than they look: they prove things about the corpus the lab declared, not what the model was actually trained on, and proof-of-training schemes so far have been shown to be spoofable.

So the argument is to flip it. The evaluator controls the test: the submission never receives the labels, evaluation runs with no network, the evaluator builds the code itself from a named commit and reproduces the score, and where possible the test data is generated after submissions freeze. A result counts only if it was reproduced.

I've built a small version of this (tabular models, private test sets, funder posts a problem and a bar). The post is honest about what it doesn't prove: that the benchmark is any good, that a hidden test set can't be squeezed through repeated submissions, that the funder didn't leak labels, and that a third party can re-run it without the data. The second of those is the gap I'd close first.

https://holdoutlabs-ai.github.io/reproduce-it-or-it-doesnt-count/

If the argument is wrong somewhere I'd rather hear it now.