[PAPER] GPQA, MMLU-Pro, and MMMU-Pro were audited for broken questions, and up to 12% of them had to be removed. New drop in clean versions released

I was very curious why all the models were topping out on GPQA-Diamond around 92 or 93% (AA) and spent the last few weeks pouring over GPQA (Diamond and Extended), and then expanded to auditing MMLU-Pro and MMMU-Pro.

It was quite frankly shocking just how many questions were malformed, had wrong answer keys, or questions with more than one realistic answer. In fact, on GPQA-Extended, MMLU-Pro and MMMU-Pro, ~12% of questions were verifiably broken! Once fixed, the top models hit around 98%.

Full paper released here: https://github.com/adamallcock/answer-key-audit

As part of this process, I have also shopped -Clean versions of all four benchmarks with the broken items removed, but also a full flagged-candidate ledger so you can see exactly why. I've also included dual original-vs-cleaned scoring, lm-eval-harness tasks and Hugging Face datasets.

I would love any feedback you have on the paper, and what benchmarks I should look at next. PS: There are some verbatim examples of broken questions on page 28 if you want to take a look.