Meta-evaluating LLMJ critiques on agentic tasks, such as Tau-Bench</p>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/633c4fb100732349209f2aad/uS3W33Cr55xK55Nu8TGjU.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/633c4fb100732349209f2aad/uS3W33Cr55xK55Nu8TGjU.png\" alt=\"Screenshot 2026-06-23 at 10.13.42\"></a></p>\n","updatedAt":"2026-06-23T09:14:08.264Z","author":{"_id":"633c4fb100732349209f2aad","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/633c4fb100732349209f2aad/CjcKdqDmsdOkpT-Z-lWPs.jpeg","fullname":"Sashank Pisupati","name":"spisupat","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":17,"isUserFollowing":false,"primaryOrg":{"avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61149f7d265ca9d17ada7408/X2rIRD_H6h-JYtLVvfg6B.jpeg","fullname":"Reflection AI","name":"reflectionai","type":"org","isHf":false,"plan":"team"}}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5886921882629395},"editors":["spisupat"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/633c4fb100732349209f2aad/CjcKdqDmsdOkpT-Z-lWPs.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.21627","authors":[{"_id":"6a3a3724fdcd3514343bb720","user":{"_id":"633c4fb100732349209f2aad","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/633c4fb100732349209f2aad/CjcKdqDmsdOkpT-Z-lWPs.jpeg","isPro":false,"fullname":"Sashank Pisupati","user":"spisupat","type":"user","name":"spisupat"},"name":"Sashank Pisupati","status":"claimed_verified","statusLastChangedAt":"2026-06-23T13:55:50.284Z","hidden":false},{"_id":"6a3a3724fdcd3514343bb721","user":{"_id":"66c70e83072fca1a89507263","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c70e83072fca1a89507263/LWBAsL9D6KaiAnRRp0zSU.jpeg","isPro":false,"fullname":"Henry Broomfield","user":"HennersBro98","type":"user","name":"HennersBro98"},"name":"Henry Broomfield","status":"claimed_verified","statusLastChangedAt":"2026-06-23T13:55:53.470Z","hidden":false},{"_id":"6a3a3724fdcd3514343bb722","user":{"_id":"660ba79bfb554841ab76ddd0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/660ba79bfb554841ab76ddd0/CAwaGw9fFe4nf_XR36a6y.jpeg","isPro":false,"fullname":"Eujeong Choi","user":"EujeongChoi","type":"user","name":"EujeongChoi"},"name":"Eujeong Choi","status":"claimed_verified","statusLastChangedAt":"2026-06-23T13:55:48.535Z","hidden":false},{"_id":"6a3a3724fdcd3514343bb723","name":"Antonia Calvi","hidden":false},{"_id":"6a3a3724fdcd3514343bb724","name":"Charlie Wang","hidden":false},{"_id":"6a3a3724fdcd3514343bb725","name":"Roman Engeler","hidden":false},{"_id":"6a3a3724fdcd3514343bb726","name":"Max Bartolo","hidden":false},{"_id":"6a3a3724fdcd3514343bb727","name":"Patrick Lewis","hidden":false}],"publishedAt":"2026-06-19T00:00:00.000Z","submittedOnDailyAt":"2026-06-23T00:00:00.000Z","title":"Counsel: A Meta-Evaluation Dataset for Agentic Tasks","submittedOnDailyBy":{"_id":"633c4fb100732349209f2aad","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/633c4fb100732349209f2aad/CjcKdqDmsdOkpT-Z-lWPs.jpeg","isPro":false,"fullname":"Sashank Pisupati","user":"spisupat","type":"user","name":"spisupat"},"summary":"As agentic systems tackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agentic benchmarks can take hours, making it difficult to scale evaluations for measuring performance or curating training data. This has driven widespread reliance on automated approaches such as LLM-as-a-judge (LLMJ) to critique agents at the process and outcome-levels at scale, however, the soundness of LLMJ critiques often goes unmeasured. Here, we introduce Counsel, the first public dataset of meta-evaluations for agentic tasks. Counsel consists of process-level critiques from open-weight LLMJs on two agent benchmarks: tau-bench (customer support agents) and DA-Code (coding agents), and human meta-evaluations of these critiques. Human annotators label critiques on each flagged error as \"spot on\", \"correct location but poor reasoning\", or \"should not have flagged\", achieving reliable inter-annotator agreement (Krippendorff's alpha of 0.78). The resulting dataset stratifies LLMJ critiques by human alignment across both error location within a trajectory and reasoning quality, serving as valuable data to calibrate, improve, or train LLMJs for agents. Comparing open-weight judges, we find that more capable judge models and more reasoning effort both enabled improved human agreement, with the strongest judge reaching ~88% agreement on location and ~65% on reasoning. Counsel is generated using open-weight models and is permissively licensed for broad community use, which we hope will enable rigorous study and improved alignment of LLM-based evaluators for agentic systems.","upvotes":5,"discussionId":"6a3a3724fdcd3514343bb728","projectPage":"https://huggingface.co/spaces/AtlaAI/counsel-tech-report","ai_summary":"A large-scale dataset of human-metaevaluations of LLM critiques for agentic tasks is introduced to improve the calibration and reliability of automated evaluation methods.","ai_keywords":["agentic systems","LLM-as-a-judge","meta-evaluation","trajectory evaluation","human alignment","inter-annotator agreement","Krippendorff's alpha","agent benchmarks","tau-bench","DA-Code","open-weight models","automated evaluation"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"66718df94d81aba4c985b002","name":"AtlaAI","fullname":"Atla","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/659fc8832cb13cede03047bb/rN1cwZNEyJqGQyBwwtLN-.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"633c4fb100732349209f2aad","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/633c4fb100732349209f2aad/CjcKdqDmsdOkpT-Z-lWPs.jpeg","isPro":false,"fullname":"Sashank Pisupati","user":"spisupat","type":"user"},{"_id":"66c70e83072fca1a89507263","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c70e83072fca1a89507263/LWBAsL9D6KaiAnRRp0zSU.jpeg","isPro":false,"fullname":"Henry Broomfield","user":"HennersBro98","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"},{"_id":"66d08d5c952f5e4e64bd6be0","avatarUrl":"/avatars/2fc3a6e3813718f0c001fb26337dab45.svg","isPro":false,"fullname":"Maurice","user":"MauriceBurg","type":"user"},{"_id":"66197a8afeb55cbe39e50ae8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pcSS1TsCoHxRcAEkcMNm0.png","isPro":false,"fullname":"Ollie Liu","user":"oliu-io","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"66718df94d81aba4c985b002","name":"AtlaAI","fullname":"Atla","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/659fc8832cb13cede03047bb/rN1cwZNEyJqGQyBwwtLN-.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.21627.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2606.21627 in a model README.md to link it from this page.