A CUA trajectory is the interleaved record of an agent's screens, actions, and reasoning. Deciding whether it fulfilled the instruction is the reward signal behind evaluation, data curation and RL. How well the models we now use as judges actually make that call has gone unexamined. We provide:</p>\n<ol>\n<li>A human-gold benchmark of CUA trajectories across web, Windows, Ubuntu and mobile, with an OSReward-Hard variant and OSReward-Multi for fine-grained grading.</li>\n<li>The most comprehensive judge evaluation to date, characterising models along multiple dimensions, with extensive analysis and the insights it yields</li>\n<li>OS-Shepherd-100K: the largest reasoning-annotated corpus of CUA judge samples to date, built by a pipeline shaped by those findings</li>\n<li>OS-Shepherd 9B / 35B: open-weight, open-data CUA reward models trained on that corpus, level with commercial judges at 30–60× lower cost</li>\n</ol>\n","updatedAt":"2026-08-07T05:29:02.296Z","author":{"_id":"6064a0eeb1703ddba0d458b9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png","fullname":"Qiushi","name":"QiushiSun","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":21,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9091192483901978},"editors":["QiushiSun"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.28609","authors":[{"_id":"6a704b5b02c90f968f48a149","name":"Qiushi Sun","hidden":false},{"_id":"6a704b5b02c90f968f48a14a","name":"Kanzhi Cheng","hidden":false},{"_id":"6a704b5b02c90f968f48a14b","name":"Yian Wang","hidden":false},{"_id":"6a704b5b02c90f968f48a14c","name":"Bowen Yang","hidden":false},{"_id":"6a704b5b02c90f968f48a14d","name":"Hang Yan","hidden":false},{"_id":"6a704b5b02c90f968f48a14e","name":"Liheng Chen","hidden":false},{"_id":"6a704b5b02c90f968f48a14f","name":"Fangzhi Xu","hidden":false},{"_id":"6a704b5b02c90f968f48a150","name":"Zichen Ding","hidden":false},{"_id":"6a704b5b02c90f968f48a151","name":"Nuo Chen","hidden":false},{"_id":"6a704b5b02c90f968f48a152","name":"Jialin Cao","hidden":false},{"_id":"6a704b5b02c90f968f48a153","name":"Xingdong Gong","hidden":false},{"_id":"6a704b5b02c90f968f48a154","name":"Zehao Li","hidden":false},{"_id":"6a704b5b02c90f968f48a155","name":"Kaiming Jin","hidden":false},{"_id":"6a704b5b02c90f968f48a156","name":"Xinfeng Yuan","hidden":false},{"_id":"6a704b5b02c90f968f48a157","name":"Zhoumianze Liu","hidden":false},{"_id":"6a704b5b02c90f968f48a158","name":"Jingyang Gong","hidden":false},{"_id":"6a704b5b02c90f968f48a159","name":"Zhangyue Yin","hidden":false},{"_id":"6a704b5b02c90f968f48a15a","name":"Jiahui Gao","hidden":false},{"_id":"6a704b5b02c90f968f48a15b","name":"Zhiyong Wu","hidden":false},{"_id":"6a704b5b02c90f968f48a15c","name":"Tianbao Xie","hidden":false},{"_id":"6a704b5b02c90f968f48a15d","name":"Jianbing Zhang","hidden":false},{"_id":"6a704b5b02c90f968f48a15e","name":"Ben Kao","hidden":false},{"_id":"6a704b5b02c90f968f48a15f","name":"Lingpeng Kong","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/6064a0eeb1703ddba0d458b9/rOPmYJDoU7iKVmz0ytY_v.png","https://cdn-uploads.huggingface.co/production/uploads/6064a0eeb1703ddba0d458b9/9IInd13x1Ei5-BCEc_t3g.png","https://cdn-uploads.huggingface.co/production/uploads/6064a0eeb1703ddba0d458b9/NrKMZX4HQcfXLsLYHZt6p.png"],"publishedAt":"2026-07-30T00:00:00.000Z","submittedOnDailyAt":"2026-08-07T00:00:00.000Z","title":"OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models","submittedOnDailyBy":{"_id":"6064a0eeb1703ddba0d458b9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png","isPro":false,"fullname":"Qiushi","user":"QiushiSun","type":"user","name":"QiushiSun"},"summary":"Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.","upvotes":44,"discussionId":"6a704b5b02c90f968f48a160","projectPage":"https://os-copilot.github.io/OSReward-Home/","githubRepo":"https://github.com/OS-Copilot/OSReward","githubRepoAddedBy":"user","githubStars":6,"organization":{"_id":"61bb0986699d29d369eba1b2","name":"hkunlp","fullname":"NLP Group of The University of Hong Kong","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1639647572687-618767e4238063b4615d042b.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6597ec3afd17ceb1d4e9ea9a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6597ec3afd17ceb1d4e9ea9a/VYYLkHjfNVHNx84DtO1Rt.png","isPro":false,"fullname":"Hongli Zhou","user":"Joe-Hall-Lee","type":"user"},{"_id":"642b9861bb77f8456634b048","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/VrNmmcdgX7FufQmdP5YaG.jpeg","isPro":false,"fullname":"Zichen Ding","user":"heroding77","type":"user"},{"_id":"6064a0eeb1703ddba0d458b9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png","isPro":false,"fullname":"Qiushi","user":"QiushiSun","type":"user"},{"_id":"64ca227b667f4f80850e6bd1","avatarUrl":"/avatars/db083cbc9d1540cdaa8b270c7e3f0304.svg","isPro":false,"fullname":"Hang Yan","user":"beatccjiang","type":"user"},{"_id":"628c5da32f09ccf530204dbe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1653366416287-628c5da32f09ccf530204dbe.jpeg","isPro":false,"fullname":"Zhangyue Yin","user":"yinzhangyue","type":"user"},{"_id":"666ff82ddc348adcabc2e6bd","avatarUrl":"/avatars/3cb86a3d7ab154fc169024c9d39c7238.svg","isPro":false,"fullname":"John Gong","user":"gonggongjohn","type":"user"},{"_id":"6355473d525beaee688b7ba1","avatarUrl":"/avatars/1fb0d57ed5f1a9b872a1ada8b2973ffb.svg","isPro":false,"fullname":"Wei Tao","user":"itaowe","type":"user"},{"_id":"636f526a6cd69d9a36ff2b53","avatarUrl":"/avatars/8f2271a193fcac609d9be270552b5afa.svg","isPro":false,"fullname":"Qiguang Chen","user":"LightChen2333","type":"user"},{"_id":"6776ae0c91b4c75dac91249c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6776ae0c91b4c75dac91249c/uJk3ZnRrzjPCcBNjmrWLI.png","isPro":false,"fullname":"Oran Feng","user":"xiachongfeng","type":"user"},{"_id":"6374f8bb6ea8da14f800a558","avatarUrl":"/avatars/66e1f741e054a6b56f152df5ca229804.svg","isPro":false,"fullname":"Xijia Tao","user":"Cie1","type":"user"},{"_id":"61669c456916c52acd5a1aa3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61669c456916c52acd5a1aa3/HnZTwRaXgTeTG3ljO3ITb.jpeg","isPro":false,"fullname":"jianbo dai","user":"jbd","type":"user"},{"_id":"6214e4ee1e35c843d42d1f88","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6214e4ee1e35c843d42d1f88/fj-9wuIdPhvogh3BrcXTB.jpeg","isPro":false,"fullname":"Longxu Dou","user":"dreamerdeo","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":3,"organization":{"_id":"61bb0986699d29d369eba1b2","name":"hkunlp","fullname":"NLP Group of The University of Hong Kong","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1639647572687-618767e4238063b4615d042b.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.28609.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2607.28609 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.