-- Your medical AI agent didn't fail because it lacked medical knowledge. It failed because it didn't verify its own work.</p>\n<p>A strong agent produces fewer errors and recovers gracefully from the ones it makes.</p>\n<p>📖Our Paper -- AutoMedBench, is now online: <a href=\"https://arxiv.org/pdf/2606.01961\" rel=\"nofollow\">https://arxiv.org/pdf/2606.01961</a><br>🌍 Leaderboard: <a href=\"https://automedbench.github.io\" rel=\"nofollow\">https://automedbench.github.io</a></p>\n<p>AutoMedBench is a long-horizon medical imaging + multimodal benchmark with 5 tracks, averaging 33 agent turns per run. Tasks come in Lite and Standard tiers and are scored across 5 stages: Plan → Setup → Validate → Inference → Submit.</p>\n<p>Main finding: Validate is the weakest stage, while Setup is the strongest. Current agents are better at making pipelines executable than at verifying reliability. ⚠️</p>\n<p>Error analysis confirms it:<br>🔍 verification/recovery errors: 37.7%<br>📦 deliverable/submission errors: 38.1%<br>🧠 task-understanding errors: only 0.9%</p>\n<p>Runs with one fired error code have a 48% lower overall score than clean runs.</p>\n","updatedAt":"2026-06-03T05:29:19.065Z","author":{"_id":"6371b9bb3d1bd47a4ec73ec5","avatarUrl":"/avatars/ed92874a85efcd9ea487a5b59959eb46.svg","fullname":"Tianhao Qi","name":"qth","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.906527042388916},"editors":["qth"],"editorAvatarUrls":["/avatars/ed92874a85efcd9ea487a5b59959eb46.svg"],"reactions":[{"reaction":"🔥","users":["MitakaKuma","qth"],"count":2}],"isReport":false}},{"id":"6a1fc17cd0801677b936c1ba","author":{"_id":"671b8822cca657cc83e23281","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/671b8822cca657cc83e23281/blQLbiPQe-MeY1YoAU7uV.png","fullname":"Kuma Kuma","name":"MitakaKuma","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false},"createdAt":"2026-06-03T05:54:04.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"❓The problem: Most medical agent benchmarks score the final answer. But real medical-AI research is a workflow, and failures are often hidden *inside* that workflow.\n\n❕The solution: AutoMedBench provides the workflow-aware evaluation of LLM agents in AutoResearch tasks, and covering wild range of tasks from segmentation, image enhancement, VQA, report generation and lesion detection\n\n\n\n🤔 What makes AutoMedBench different:\n\nAutoMedBench ranks LLM based on agentic and task scores, as:\n\n1. **Agentic score**: stage-level S1-S5 workflow scoring \n2. **Task score**: held-out medical metric for the final artifact \n3. **Overall score**: 0.5 * Agentic score + 0.5 * Task score\n\nFurthurmore, AutoMedBench applies the combination of online LLM judge and determinstic score, where S1-S3 are judged from saved artifacts + execution traces; S4-S5 use deterministic checks for inference completeness and submission validity.\n\n\n\nThis makes the benchmark useful beyond ranking: it produces feedback signals for agent self-refinement.\n\nDid the agent write a valid plan? Did it run a pilot? Did it inspect outputs? Did it debug and rerun? Did it submit the right schema?\n\n‼️ AutoMedBench tells us *where* to refine the agent, not just *whether* it failed.\n\n\n\n","html":"<p>❓The problem: Most medical agent benchmarks score the final answer. But real medical-AI research is a workflow, and failures are often hidden <em>inside</em> that workflow.</p>\n<p>❕The solution: AutoMedBench provides the workflow-aware evaluation of LLM agents in AutoResearch tasks, and covering wild range of tasks from segmentation, image enhancement, VQA, report generation and lesion detection</p>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/671b8822cca657cc83e23281/_2MziLBJLQqa0tQNjYRgL.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/671b8822cca657cc83e23281/_2MziLBJLQqa0tQNjYRgL.png\" alt=\"fig_method\"></a></p>\n<p>🤔 What makes AutoMedBench different:</p>\n<p>AutoMedBench ranks LLM based on agentic and task scores, as:</p>\n<ol>\n<li><strong>Agentic score</strong>: stage-level S1-S5 workflow scoring </li>\n<li><strong>Task score</strong>: held-out medical metric for the final artifact </li>\n<li><strong>Overall score</strong>: 0.5 * Agentic score + 0.5 * Task score</li>\n</ol>\n<p>Furthurmore, AutoMedBench applies the combination of online LLM judge and determinstic score, where S1-S3 are judged from saved artifacts + execution traces; S4-S5 use deterministic checks for inference completeness and submission validity.</p>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/671b8822cca657cc83e23281/nzc46tWqEMNicydu0ULWe.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/671b8822cca657cc83e23281/nzc46tWqEMNicydu0ULWe.png\" alt=\"fig_scoring_rubrics\"></a></p>\n<p>This makes the benchmark useful beyond ranking: it produces feedback signals for agent self-refinement.</p>\n<p>Did the agent write a valid plan? Did it run a pilot? Did it inspect outputs? Did it debug and rerun? Did it submit the right schema?</p>\n<p>‼️ AutoMedBench tells us <em>where</em> to refine the agent, not just <em>whether</em> it failed.</p>\n","updatedAt":"2026-06-03T05:54:49.085Z","author":{"_id":"671b8822cca657cc83e23281","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/671b8822cca657cc83e23281/blQLbiPQe-MeY1YoAU7uV.png","fullname":"Kuma Kuma","name":"MitakaKuma","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.7646273970603943},"editors":["MitakaKuma"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/671b8822cca657cc83e23281/blQLbiPQe-MeY1YoAU7uV.png"],"reactions":[{"reaction":"🔥","users":["qth"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.01961","authors":[{"_id":"6a1e69d2808ddbc3c7d43e74","name":"Junqi Liu","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e75","name":"Salena Song","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e76","name":"Yuhan Wang","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e77","name":"Jiawei Mao","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e78","name":"Hardy Chen","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e79","name":"Xiaoke Huang","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e7a","name":"Tianhao Qi","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e7b","name":"Pengfei Guo","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e7c","name":"Yucheng Tang","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e7d","name":"Yufan He","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e7e","name":"Can Zhao","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e7f","name":"Andriy Myronenko","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e80","name":"Dong Yang","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e81","name":"Daguang Xu","hidden":false},{"_id":"6a1e69d2808ddbc3c7d43e82","name":"Yuyin Zhou","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/6371b9bb3d1bd47a4ec73ec5/tIdwvZMr6zWXt9aJOV5-M.png"],"publishedAt":"2026-06-01T00:00:00.000Z","submittedOnDailyAt":"2026-06-03T00:00:00.000Z","title":"AutoMedBench: Towards Medical AutoResearch with Agentic AI Models","submittedOnDailyBy":{"_id":"6371b9bb3d1bd47a4ec73ec5","avatarUrl":"/avatars/ed92874a85efcd9ea487a5b59959eb46.svg","isPro":false,"fullname":"Tianhao Qi","user":"qth","type":"user","name":"qth"},"summary":"Autonomous agents are increasingly expected to support end-to-end medical-AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering. However, existing medical agent benchmarks primarily evaluate final outputs, providing limited visibility into agent behavior within the research process. To address this gap, we present AutoMedBench, a workflow-aware benchmark for autonomous medical-AI research across diverse medical imaging and multimodal inference tasks, organizing agent execution into a unified five-stage workflow (S1-S5): Plan, Setup, Validate, Inference, and Submit. It comprises long-horizon tasks with each run averaging 33 agent turns, spanning five research tracks: segmentation, image enhancement, visual question answering (VQA), report generation, and lesion detection. Each task is evaluated under two difficulty tiers, Lite and Standard, which use the same data and metrics but differ in the amount of task-brief scaffolding, and each run is scored using both final task performance and S1-S5 stage scores, enabling stage-level analysis from the initial task brief to the final submitted artifact. Across thousands of recorded runs, stage-level scoring reveals that Validate is the weakest workflow stage on average, whereas Setup is the strongest, suggesting that current agents are better at making pipelines executable than at verifying their reliability. Post-run error analysis further shows that verification and submission failures dominate tagged errors, accounting for 37.7% and 38.1% of fired codes respectively, whereas task-understanding errors are rare at 0.9%, and runs with one fired error code have a 48% lower overall score than runs with no error code on average.","upvotes":15,"discussionId":"6a1e69d3808ddbc3c7d43e83","projectPage":"https://automedbench.github.io/","githubRepo":"https://github.com/AutoMedBench/AutoMedBench","githubRepoAddedBy":"user","ai_summary":"AutoMedBench presents a comprehensive benchmark for autonomous medical-AI research that evaluates agent performance across five workflow stages, revealing validation as the weakest stage and highlighting the importance of reliable pipeline execution and verification in medical AI workflows.","ai_keywords":["autonomous agents","medical-AI research","workflow-aware benchmark","five-stage workflow","Plan","Setup","Validate","Inference","Submit","long-horizon tasks","research tracks","segmentation","image enhancement","visual question answering","report generation","lesion detection","stage-level analysis","error analysis","verification","submission failures","task-understanding errors"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":32,"organization":{"_id":"641e3d47921e9c5d8481da5d","name":"UCSC","fullname":"University of California, Santa Cruz","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/68e396f2b5bb631e9b2fac9a/EjFSifKRyMKbLHB59AYbp.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"671b8822cca657cc83e23281","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/671b8822cca657cc83e23281/blQLbiPQe-MeY1YoAU7uV.png","isPro":false,"fullname":"Kuma Kuma","user":"MitakaKuma","type":"user"},{"_id":"66f23f77353887e95e958ed2","avatarUrl":"/avatars/a394600629867add16783e15da60b0bb.svg","isPro":false,"fullname":"yzhou284","user":"yzhou284","type":"user"},{"_id":"6604d7b2398b0e15bc4fb848","avatarUrl":"/avatars/033c42902f12877d8ac08a38cf4ece90.svg","isPro":false,"fullname":"Sergiu Han","user":"hgsg","type":"user"},{"_id":"643f9e2288d9d4488fd81c52","avatarUrl":"/avatars/e589c9cbd47022883cf33d7555bee89c.svg","isPro":false,"fullname":"Tinghui Zhu","user":"DarthZhu","type":"user"},{"_id":"660026b7573abbcdb975a34f","avatarUrl":"/avatars/93defd0e6274cfe8f124220c59ec2bed.svg","isPro":false,"fullname":"Juncheng Wu","user":"Chtholly17","type":"user"},{"_id":"6371b9bb3d1bd47a4ec73ec5","avatarUrl":"/avatars/ed92874a85efcd9ea487a5b59959eb46.svg","isPro":false,"fullname":"Tianhao Qi","user":"qth","type":"user"},{"_id":"680710c4240820c2f5d0f5b2","avatarUrl":"/avatars/6137bc0ab24bdb6ba08fbc8cc1bc0f78.svg","isPro":false,"fullname":"JJ","user":"JJ02HF","type":"user"},{"_id":"68e70a5d311f55603f9bbdcc","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/V8KuwjLWhPZoRvNDl4ucT.png","isPro":false,"fullname":"Geyifan","user":"yfge","type":"user"},{"_id":"650d13f241521dec20fa7725","avatarUrl":"/avatars/14897818e5f2232f52e1ca04ac4525ae.svg","isPro":false,"fullname":"JIANJUN XU","user":"KAICLIFE","type":"user"},{"_id":"692855eabdadcb130bd0fb35","avatarUrl":"/avatars/dd9a608d6ac196e349eb3b48ca8ffaea.svg","isPro":false,"fullname":"XMY","user":"showmeon","type":"user"},{"_id":"683aff59920f5940bbfb068d","avatarUrl":"/avatars/56bd1a6d18252e8879a2ec639f7e84d1.svg","isPro":true,"fullname":"jiawei mao","user":"JiaMao","type":"user"},{"_id":"6307292dcada14cdcedb0a09","avatarUrl":"/avatars/5b5defd64d3b237e1da97426da4cfabf.svg","isPro":false,"fullname":"Fang","user":"Shancheng","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"641e3d47921e9c5d8481da5d","name":"UCSC","fullname":"University of California, Santa Cruz","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/68e396f2b5bb631e9b2fac9a/EjFSifKRyMKbLHB59AYbp.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.01961.md"}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2606.01961 in a model README.md to link it from this page.

Cite arxiv.org/abs/2606.01961 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2606.01961 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.