Paper: <a href=\"https://arxiv.org/abs/2606.06462\" rel=\"nofollow\">https://arxiv.org/abs/2606.06462</a><br>Code: <a href=\"https://github.com/Shiyun-x/Benchmark-Agent\" rel=\"nofollow\">https://github.com/Shiyun-x/Benchmark-Agent</a></p>\n","updatedAt":"2026-06-05T15:11:35.174Z","author":{"_id":"644784e219538c015b2531f5","avatarUrl":"/avatars/748507a8164d99a126ff0b21990240bd.svg","fullname":"Shiyun Xiong","name":"Christinexx","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6399125456809998},"editors":["Christinexx"],"editorAvatarUrls":["/avatars/748507a8164d99a126ff0b21990240bd.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.06462","authors":[{"_id":"6a225fd83490a593e87b15cf","user":{"_id":"644784e219538c015b2531f5","avatarUrl":"/avatars/748507a8164d99a126ff0b21990240bd.svg","isPro":false,"fullname":"Shiyun Xiong","user":"Christinexx","type":"user","name":"Christinexx"},"name":"Shiyun Xiong","status":"claimed_verified","statusLastChangedAt":"2026-06-05T15:06:08.751Z","hidden":false},{"_id":"6a225fd83490a593e87b15d0","name":"Dongming Wu","hidden":false},{"_id":"6a225fd83490a593e87b15d1","name":"Peiwen Sun","hidden":false},{"_id":"6a225fd83490a593e87b15d2","name":"Yuang Ai","hidden":false},{"_id":"6a225fd83490a593e87b15d3","name":"Bokang Yang","hidden":false},{"_id":"6a225fd83490a593e87b15d4","name":"Wencheng Han","hidden":false},{"_id":"6a225fd83490a593e87b15d5","name":"Xiao-Hui Li","hidden":false},{"_id":"6a225fd83490a593e87b15d6","name":"Xiangyu Yue","hidden":false}],"publishedAt":"2026-06-04T00:00:00.000Z","submittedOnDailyAt":"2026-06-05T00:00:00.000Z","title":"Benchmark Everything Everywhere All at Once","submittedOnDailyBy":{"_id":"644784e219538c015b2531f5","avatarUrl":"/avatars/748507a8164d99a126ff0b21990240bd.svg","isPro":false,"fullname":"Shiyun Xiong","user":"Christinexx","type":"user","name":"Christinexx"},"summary":"Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.","upvotes":1,"discussionId":"6a225fd83490a593e87b15d7","projectPage":"https://benchmarkagent.github.io/","githubRepo":"https://github.com/Shiyun-x/Benchmark-Agent","githubRepoAddedBy":"user","ai_summary":"Automated benchmark creation system generates diverse evaluation datasets with minimal human intervention, enabling continuous model assessment across multiple domains.","ai_keywords":["benchmark construction","LLMs","MLLMs","automated systems","benchmark generation","performance evaluation","domain-specific reasoning","human evaluation","LLM-as-a-judge assessment","consistency checks"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":2},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"644784e219538c015b2531f5","avatarUrl":"/avatars/748507a8164d99a126ff0b21990240bd.svg","isPro":false,"fullname":"Shiyun Xiong","user":"Christinexx","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.06462.md"}">
Benchmark Everything Everywhere All at Once
Abstract
Automated benchmark creation system generates diverse evaluation datasets with minimal human intervention, enabling continuous model assessment across multiple domains.
Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.06462 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.06462 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.06462 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.