Did blatant AI Slop just win a 25K USD Deepmind / Kaggle Grand Prize? [D]
Mirrored from r/MachineLearning for archival readability. Support the source by reading on the original site.
The Google DeepMind-sponsored Kaggle challenge "Measuring Progress Toward AGI - Cognitive Abilities" asked participants to design new cognitive-science-based AI benchmarks and they just announced the results this week.
In my two posts I present evidence that deepmind & kaggle rewarded a nonsensical number generation machine and a litany of unfounded claims with 25k and a grand prize stamp.
What the authors of the work I analyze intended to do was to present an LLM with alternative viewpoints of other LLMs on 5 claims regarding a tricky situation and see whether the model changes its own assessment. It's an interesting question. However, it turned into a vibed pile of spaghetti 10 times the size of the requested submission format which it seems neither the authors nor the judges were able to (or minded to?) give a cursory reading.
Here's the original posts in the competition forum, if you are looking for some AI research slop detective work / rant please help yourselves. But beware, some of the "universal findings" or "core insights" of the authors might continue to haunt you. You might even question your own sanity (as I did).
Part 1: The Smoke: cursory review of the writeup
Part 2: The Fire: looking at the methodology, code, and data
The organizers' stance has been that review was done properly and this is just a matter of subjectivity. What do you think?
[link] [comments]
More from r/MachineLearning
-
A collision-entropy floor for watermark/retrieval AI-text detection. Looking for a sanity check before I take this further [D]
Aug 14
-
Are supervised and unsupervised learning still relevant today? [D]
Aug 14
-
TMLR Relevance and Prestige [D]
Aug 13
-
Reproducible canvas-aligned low-level patterns in somerandomllm-generated images and their possible relation to iterative editing artifacts [D]
Aug 13
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.