When an AI agent fails, what actually needs fixing—the model, the harness, or something else?</p>\n<p>In this work, we introduce an interaction-centric taxonomy of 41 agent failure modes to help localize where failures originate and where the fix belongs.</p>\n<p>Curious how others think about the boundary between model failures and harness failures in practice.</p>\n","updatedAt":"2026-08-04T11:31:18.762Z","author":{"_id":"60ec9b1fea2a0aeecd4a7d85","avatarUrl":"/avatars/1d1860f63b523021c7a5eddfcbaf6838.svg","fullname":"Harsh Raj","name":"Harsh1729","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9214110970497131},"editors":["Harsh1729"],"editorAvatarUrls":["/avatars/1d1860f63b523021c7a5eddfcbaf6838.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.28802","authors":[{"_id":"6a71173cec5082b9f872cc6a","name":"Harsh Raj","hidden":false},{"_id":"6a71173cec5082b9f872cc6b","name":"Vipul Gupta","hidden":false},{"_id":"6a71173cec5082b9f872cc6c","name":"Anas Mahmoud","hidden":false},{"_id":"6a71173cec5082b9f872cc6d","name":"Razvan-Gabriel Dumitru","hidden":false},{"_id":"6a71173cec5082b9f872cc6e","name":"Darvin Yi","hidden":false},{"_id":"6a71173cec5082b9f872cc6f","name":"Aakash Sabharwal","hidden":false},{"_id":"6a71173cec5082b9f872cc70","name":"Yunzhong He","hidden":false}],"publishedAt":"2026-07-30T00:00:00.000Z","submittedOnDailyAt":"2026-08-04T00:00:00.000Z","title":"Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures","submittedOnDailyBy":{"_id":"60ec9b1fea2a0aeecd4a7d85","avatarUrl":"/avatars/1d1860f63b523021c7a5eddfcbaf6838.svg","isPro":false,"fullname":"Harsh Raj","user":"Harsh1729","type":"user","name":"Harsh1729"},"summary":"Existing evaluations often reduce agent failures to system-level outcomes, obscuring where the fault originated and which intervention would improve the agent system. This creates a repair-assignment problem: the same visible failure may call for model post-training, harness engineering, environment redesign, or benchmark repair depending on its source. Because agent behavior emerges from interactions among models, harnesses, users, tools, memory, and environments, outcome-level labels are often insufficient for improvement. Most failure taxonomies do little to resolve this problem because they are benchmark-specific and lack a shared structure. We introduce an interaction-centric taxonomy that localizes failures to the interactions in which they originate and identifies the responsible component. It organizes 41 failure modes by assigning each to an edge between two components and a fault side indicating where the repair belongs. This makes the taxonomy actionable: model-side failures identify targets for post-training, harness-side failures point to scaffolding and tool-integration fixes, and environment or grader failures reveal evaluation conditions requiring redesign. The schema applies across agent architectures, from coding assistants to long-horizon personal assistants and multi-agent systems. We ground the taxonomy in worked examples from public benchmarks, model system cards, published reports, and logged agent trajectories, and evaluate its reproducibility using independent reasoning agents as judges. Across four frontier models, the strongest judge reaches Cohen's κ=0.76 against human category labels, suggesting that the categories capture shared structure rather than annotator-specific preferences.","upvotes":3,"discussionId":"6a71173cec5082b9f872cc71","organization":{"_id":"6677220f8a4064c02bc81217","name":"ScaleAI","fullname":"Scale AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d6a5f94c28026a003581b4/uqHyTuNQ8fX7LheVhzPeO.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64834b399b352597e41816ac","avatarUrl":"/avatars/63d9d123bffa90f43186a0bdc4455cbd.svg","isPro":false,"fullname":"Shaobai Jiang","user":"shaobaij","type":"user"},{"_id":"60ec9b1fea2a0aeecd4a7d85","avatarUrl":"/avatars/1d1860f63b523021c7a5eddfcbaf6838.svg","isPro":false,"fullname":"Harsh Raj","user":"Harsh1729","type":"user"},{"_id":"6a1687595b32ef37b413a8a6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/wdU3bWwrxWGAUlYA8sGcy.png","isPro":false,"fullname":"Lincoln Taylor","user":"lincolnta","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6677220f8a4064c02bc81217","name":"ScaleAI","fullname":"Scale AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d6a5f94c28026a003581b4/uqHyTuNQ8fX7LheVhzPeO.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.28802.md","query":{}}">
Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
Abstract
Existing evaluations often reduce agent failures to system-level outcomes, obscuring where the fault originated and which intervention would improve the agent system. This creates a repair-assignment problem: the same visible failure may call for model post-training, harness engineering, environment redesign, or benchmark repair depending on its source. Because agent behavior emerges from interactions among models, harnesses, users, tools, memory, and environments, outcome-level labels are often insufficient for improvement. Most failure taxonomies do little to resolve this problem because they are benchmark-specific and lack a shared structure. We introduce an interaction-centric taxonomy that localizes failures to the interactions in which they originate and identifies the responsible component. It organizes 41 failure modes by assigning each to an edge between two components and a fault side indicating where the repair belongs. This makes the taxonomy actionable: model-side failures identify targets for post-training, harness-side failures point to scaffolding and tool-integration fixes, and environment or grader failures reveal evaluation conditions requiring redesign. The schema applies across agent architectures, from coding assistants to long-horizon personal assistants and multi-agent systems. We ground the taxonomy in worked examples from public benchmarks, model system cards, published reports, and logged agent trajectories, and evaluate its reproducibility using independent reasoning agents as judges. Across four frontier models, the strongest judge reaches Cohen's κ=0.76 against human category labels, suggesting that the categories capture shared structure rather than annotator-specific preferences.
Community
When an AI agent fails, what actually needs fixing—the model, the harness, or something else?
In this work, we introduce an interaction-centric taxonomy of 41 agent failure modes to help localize where failures originate and where the fix belongs.
Curious how others think about the boundary between model failures and harness failures in practice.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.28802 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.28802 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.28802 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.