AI evaluation is entering an interactive benchmark era. Across tool-use agents, web/OS benchmarks, multi-agent systems, and reliability evaluations, interaction is becoming central to how modern AI systems are tested.</p>\n<p>But the field risks adding interaction faster than it develops the scientific principles for evaluating interaction.</p>\n<p>In this position paper, we argue that interactive evaluation is not just longer tasks, tool use, or multi-turn interaction. It requires a design science for mapping trajectories to valid evaluative claims.</p>\n<p>We define interactive evaluation by the evaluation problem itself: what trajectory evidence enters evaluation, how that evidence is mapped to system-level judgments, and what claims the resulting scores can support.</p>\n<p>The takeaway: do not just add interaction to benchmarks. Design the evaluation around the claim.</p>\n","updatedAt":"2026-05-20T22:32:06.879Z","author":{"_id":"687ebb87416fcff56959d817","avatarUrl":"/avatars/8f13090dd6179bb18e8c8d205fd20131.svg","fullname":"Keyang Xuan","name":"keyangx3","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8540875911712646},"editors":["keyangx3"],"editorAvatarUrls":["/avatars/8f13090dd6179bb18e8c8d205fd20131.svg"],"reactions":[{"reaction":"👍","users":["p-song1"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.17829","authors":[{"_id":"6a0e34b1164dbbc68a26c3aa","name":"Keyang Xuan","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3ab","name":"Peiyang Song","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3ac","name":"Pan Lu","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3ad","name":"Pengrui Han","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3ae","name":"Wenkai Li","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3af","name":"Zhenyu Zhang","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3b0","name":"Zexue He","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3b1","name":"Wenyue Hua","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3b2","name":"Manling Li","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3b3","name":"Jiaxuan You","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3b4","name":"Adrian Weller","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3b5","name":"Yizhong Wang","hidden":false},{"_id":"6a0e34b1164dbbc68a26c3b6","name":"Jiaxin Pei","hidden":false}],"publishedAt":"2026-05-18T00:00:00.000Z","submittedOnDailyAt":"2026-05-20T00:00:00.000Z","title":"Interactive Evaluation Requires a Design Science","submittedOnDailyBy":{"_id":"687ebb87416fcff56959d817","avatarUrl":"/avatars/8f13090dd6179bb18e8c8d205fd20131.svg","isPro":false,"fullname":"Keyang Xuan","user":"keyangx3","type":"user","name":"keyangx3"},"summary":"AI evaluation is undergoing a structural change. Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents, while many evaluation practices still inherit assumptions from response-centered benchmarks (e.g., fixed inputs, isolated outputs, and outcome judgments that can be made from a single response). The field has begun to build interactive benchmarks, but the resulting landscape is fragmented: benchmarks differ in what interaction artifacts they admit, how trajectories are scored, and what claims their results support. This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm, not merely a new family of agent benchmarks. Simply adopting previous evaluation paradigms does not suffice. We define evaluation as an autonomous mapping from evidence to judgments, and show that interactive evaluation changes both sides of this mapping: the evidence becomes interaction-generated trajectories, while the evaluation procedure must assess process, recoverability, coordination, robustness, and system-level performance. Building on this definition, we propose a two-axis taxonomy, derive design principles and reporting standards, examine representative scenarios, and analyze how longstanding evaluation challenges reappear at the trajectory level.","upvotes":11,"discussionId":"6a0e34b1164dbbc68a26c3b7","githubRepo":"https://github.com/keyangds/interactive_evaluation","githubRepoAddedBy":"user","ai_summary":"Interactive evaluation represents a principled paradigm shift requiring new frameworks for assessing system behavior through dynamic trajectories rather than static responses.","ai_keywords":["large language models","interactive benchmarks","evaluation paradigm","interaction-generated trajectories","process assessment","recoverability","coordination","robustness","system-level performance","two-axis taxonomy","design principles","reporting standards"],"githubStars":10},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"649c5cf5c1ae48cf4d7dda34","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/649c5cf5c1ae48cf4d7dda34/bSJXATqkqBn8ypUy0OezY.jpeg","isPro":false,"fullname":"Peiyang Song","user":"p-song1","type":"user"},{"_id":"63624ffd2a84d82a8c8d3f60","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63624ffd2a84d82a8c8d3f60/NhDFHhlCWJeMevXN7aQUX.png","isPro":false,"fullname":"Chumeng Liang","user":"chumengl","type":"user"},{"_id":"62c68fe4d9bb7ca028dc7323","avatarUrl":"/avatars/5114e6102c68da7ef367e63e61ce47dc.svg","isPro":false,"fullname":"Wenkai Li","user":"wenkai-li","type":"user"},{"_id":"687ebb87416fcff56959d817","avatarUrl":"/avatars/8f13090dd6179bb18e8c8d205fd20131.svg","isPro":false,"fullname":"Keyang Xuan","user":"keyangx3","type":"user"},{"_id":"664263ea8b2d38e53f04079c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/664263ea8b2d38e53f04079c/_7og0ggbPpEHcH6g__DF0.jpeg","isPro":false,"fullname":"Pengrui Han","user":"barryhpr","type":"user"},{"_id":"5f9e1a68a13e063b8b2b5b15","avatarUrl":"/avatars/754c6339f77ebe6bf39a4606c037431a.svg","isPro":false,"fullname":"Jiaxin Pei","user":"pedropei","type":"user"},{"_id":"65621fd68631d43d2baf33b2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1JemNCnPkS1mE3SNygsE2.png","isPro":false,"fullname":"siqi zhu","user":"zsqzz","type":"user"},{"_id":"66269e8b3385ba4e6ed75834","avatarUrl":"/avatars/a3188fbb4526f62db92dc65ce42598bc.svg","isPro":false,"fullname":"Tao Feng","user":"taofeng","type":"user"},{"_id":"65334279a78e70d19c7ccdaa","avatarUrl":"/avatars/1f73738347829e0eb8ed5f2245b0bfc6.svg","isPro":false,"fullname":"Yifan Sun","user":"Yifan99","type":"user"},{"_id":"64588ef3aa36a63c42a1a47a","avatarUrl":"/avatars/c69e89d79f42ed3c77fabfd75cfa970a.svg","isPro":false,"fullname":"Ziheng Cai","user":"MonoLeon","type":"user"},{"_id":"679fff15d09622fbebbe1395","avatarUrl":"/avatars/109e6756cedb7d2b2d519564813db895.svg","isPro":false,"fullname":"Zhanyang Jin","user":"HolmesS","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.17829.md"}">
Interactive Evaluation Requires a Design Science
Authors: ,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Interactive evaluation represents a principled paradigm shift requiring new frameworks for assessing system behavior through dynamic trajectories rather than static responses.
AI-generated summary
AI evaluation is undergoing a structural change. Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents, while many evaluation practices still inherit assumptions from response-centered benchmarks (e.g., fixed inputs, isolated outputs, and outcome judgments that can be made from a single response). The field has begun to build interactive benchmarks, but the resulting landscape is fragmented: benchmarks differ in what interaction artifacts they admit, how trajectories are scored, and what claims their results support. This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm, not merely a new family of agent benchmarks. Simply adopting previous evaluation paradigms does not suffice. We define evaluation as an autonomous mapping from evidence to judgments, and show that interactive evaluation changes both sides of this mapping: the evidence becomes interaction-generated trajectories, while the evaluation procedure must assess process, recoverability, coordination, robustness, and system-level performance. Building on this definition, we propose a two-axis taxonomy, derive design principles and reporting standards, examine representative scenarios, and analyze how longstanding evaluation challenges reappear at the trajectory level.
Community
AI evaluation is entering an interactive benchmark era. Across tool-use agents, web/OS benchmarks, multi-agent systems, and reliability evaluations, interaction is becoming central to how modern AI systems are tested.
But the field risks adding interaction faster than it develops the scientific principles for evaluating interaction.
In this position paper, we argue that interactive evaluation is not just longer tasks, tool use, or multi-turn interaction. It requires a design science for mapping trajectories to valid evaluative claims.
We define interactive evaluation by the evaluation problem itself: what trajectory evidence enters evaluation, how that evidence is mapped to system-level judgments, and what claims the resulting scores can support.
The takeaway: do not just add interaction to benchmarks. Design the evaluation around the claim.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.17829 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.17829 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.17829 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.