An End-to-End Agent Auditing Engine<br>Our code is available at <a href=\"https://github.com/datamllab/A2E\" rel=\"nofollow\">https://github.com/datamllab/A2E</a></p>\n","updatedAt":"2026-08-11T06:50:40.260Z","author":{"_id":"69181250e7201807ff11ee2f","avatarUrl":"/avatars/07f3a6c5bfc9fa8379e59fa1b546912c.svg","fullname":"Mingxuan Zhang","name":"xuan269","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8467903137207031},"editors":["xuan269"],"editorAvatarUrls":["/avatars/07f3a6c5bfc9fa8379e59fa1b546912c.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.07346","authors":[{"_id":"6a7956538e9301703eaa5f5d","name":"Haoning Wang","hidden":false},{"_id":"6a7956538e9301703eaa5f5e","name":"Mingxun Zhang","hidden":false},{"_id":"6a7956538e9301703eaa5f5f","name":"Chenyue Yu","hidden":false},{"_id":"6a7956538e9301703eaa5f60","name":"Yingjun Shang","hidden":false},{"_id":"6a7956538e9301703eaa5f61","name":"Xia Hu","hidden":false},{"_id":"6a7956538e9301703eaa5f62","name":"Guanchu Wang","hidden":false},{"_id":"6a7956538e9301703eaa5f63","name":"Na Zou","hidden":false}],"publishedAt":"2026-08-10T00:00:00.000Z","submittedOnDailyAt":"2026-08-11T00:00:00.000Z","title":"A^2E : An End-to-End Agent Auditing Engine","submittedOnDailyBy":{"_id":"69181250e7201807ff11ee2f","avatarUrl":"/avatars/07f3a6c5bfc9fa8379e59fa1b546912c.svg","isPro":false,"fullname":"Mingxuan Zhang","user":"xuan269","type":"user","name":"xuan269"},"summary":"With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce A^2E (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. A^2E leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, A^2E systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with A^2E further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.","upvotes":6,"discussionId":"6a7956538e9301703eaa5f64","githubRepo":"https://github.com/datamllab/A2E","githubRepoAddedBy":"user","ai_summary":"A2E is an end-to-end evaluation engine for agent harnesses that uses a standardized task protocol and execution traces to assess capabilities across efficiency, tool use, planning, and error recovery.","ai_keywords":["large language models","agent harnesses","Agent Auditing Engine","Agent Task Protocol","Monitor","execution traces","multidimensional metrics","execution efficiency","tool use","task planning","error recovery"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":24,"organization":{"_id":"6a4fb75a1c66dbf208e7ddb6","name":"Shanghai-AI-Laboratory","fullname":"Shanghai AI Laboratory","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/65cd955637be1841d0b75397/Rao_Kq6NMtTVfSqLUIR4k.webp"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"69181250e7201807ff11ee2f","avatarUrl":"/avatars/07f3a6c5bfc9fa8379e59fa1b546912c.svg","isPro":false,"fullname":"Mingxuan Zhang","user":"xuan269","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"65c4eb7cd1dcbd30d86febec","avatarUrl":"/avatars/001c8f02e8ce794b2c21883628b2da72.svg","isPro":false,"fullname":"free-bit","user":"free-bit","type":"user"},{"_id":"6270324ebecab9e2dcf245de","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6270324ebecab9e2dcf245de/cMbtWSasyNlYc9hvsEEzt.jpeg","isPro":false,"fullname":"Kye Gomez","user":"kye","type":"user"},{"_id":"66c1f01cea476bea053a3fe8","avatarUrl":"/avatars/fa6fa5c50a8bc327e49c6cda28ad69ba.svg","isPro":false,"fullname":"Wang Haoning","user":"stevewithjobs","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6a4fb75a1c66dbf208e7ddb6","name":"Shanghai-AI-Laboratory","fullname":"Shanghai AI Laboratory","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/65cd955637be1841d0b75397/Rao_Kq6NMtTVfSqLUIR4k.webp"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.07346.md","query":{}}">
A^2E : An End-to-End Agent Auditing Engine
Abstract
A2E is an end-to-end evaluation engine for agent harnesses that uses a standardized task protocol and execution traces to assess capabilities across efficiency, tool use, planning, and error recovery.
With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce A^2E (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. A^2E leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, A^2E systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with A^2E further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.07346 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.07346 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.07346 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.