Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.</p>\n","updatedAt":"2026-06-26T03:29:46.032Z","author":{"_id":"685b93ba200d37039025b89d","avatarUrl":"/avatars/e8963b6ab8e6a248d701891e5cedd43e.svg","fullname":"Xiao Zhou","name":"rebeccazzzz","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9185006022453308},"editors":["rebeccazzzz"],"editorAvatarUrls":["/avatars/e8963b6ab8e6a248d701891e5cedd43e.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.24551","authors":[{"_id":"6a3b4b060a86ac3098d5d715","user":{"_id":"685b93ba200d37039025b89d","avatarUrl":"/avatars/e8963b6ab8e6a248d701891e5cedd43e.svg","isPro":false,"fullname":"Xiao Zhou","user":"rebeccazzzz","type":"user","name":"rebeccazzzz"},"name":"Xiao Zhou","status":"claimed_verified","statusLastChangedAt":"2026-06-25T09:30:22.665Z","hidden":false},{"_id":"6a3b4b060a86ac3098d5d716","name":"Siyue Zhang","hidden":false},{"_id":"6a3b4b060a86ac3098d5d717","name":"Yilun Zhao","hidden":false},{"_id":"6a3b4b060a86ac3098d5d718","name":"Jinbiao Wei","hidden":false},{"_id":"6a3b4b060a86ac3098d5d719","name":"Tingyu Song","hidden":false},{"_id":"6a3b4b060a86ac3098d5d71a","name":"Arman Cohan","hidden":false},{"_id":"6a3b4b060a86ac3098d5d71b","name":"Chen Zhao","hidden":false}],"publishedAt":"2026-06-22T00:00:00.000Z","submittedOnDailyAt":"2026-06-26T00:00:00.000Z","title":"GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents","submittedOnDailyBy":{"_id":"685b93ba200d37039025b89d","avatarUrl":"/avatars/e8963b6ab8e6a248d701891e5cedd43e.svg","isPro":false,"fullname":"Xiao Zhou","user":"rebeccazzzz","type":"user","name":"rebeccazzzz"},"summary":"Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.","upvotes":27,"discussionId":"6a3b4b070a86ac3098d5d71c","githubRepo":"https://github.com/rebeccaz4/gui-vs-cli","githubRepoAddedBy":"user","githubStars":1},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64dc29d9b5d625e0e9a6ecb9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QxGBsnk1cNsBEPqSx4ae-.jpeg","isPro":false,"fullname":"Tingyu Song","user":"songtingyu","type":"user"},{"_id":"683c642b02c1a474a867964e","avatarUrl":"/avatars/63e44a9cf788ee7b3ad236407700ceca.svg","isPro":false,"fullname":"Jinbiao Wei","user":"mikeweii","type":"user"},{"_id":"638f1803c67af472d317a922","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/638f1803c67af472d317a922/9BMVXqHa-AsdZPmBprcbd.jpeg","isPro":false,"fullname":"siyue zhang","user":"siyue","type":"user"},{"_id":"685b93ba200d37039025b89d","avatarUrl":"/avatars/e8963b6ab8e6a248d701891e5cedd43e.svg","isPro":false,"fullname":"Xiao Zhou","user":"rebeccazzzz","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"646def60df618b303b419323","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646def60df618b303b419323/JLJGYen4-5M8ivsLsSk0w.jpeg","isPro":false,"fullname":"Lei Wang","user":"demolei","type":"user"},{"_id":"62f662bcc58915315c4eccea","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg","isPro":true,"fullname":"Yilun Zhao","user":"yilunzhao","type":"user"},{"_id":"6a2f314bed2345e45e4e2129","avatarUrl":"/avatars/43e4cc657b99708a045d84c76069600e.svg","isPro":false,"fullname":"Yutong Ding","user":"yutong45","type":"user"},{"_id":"6a2f2e96d5560ff540676f39","avatarUrl":"/avatars/ab8a604f2a74c0917ae50676bbfc9f2d.svg","isPro":false,"fullname":"Zhang Wei","user":"zhangwei-hf","type":"user"},{"_id":"6a2f0c5f63c271161df37e79","avatarUrl":"/avatars/d6598ca54b33fd2895fbfdefdfa0574e.svg","isPro":false,"fullname":"Lijun Tan","user":"lijuntan","type":"user"},{"_id":"6a2f0ca98971f84f78246fb9","avatarUrl":"/avatars/79a6bab6d4f352c23f92ec314ef05e8d.svg","isPro":false,"fullname":"Kevin Li","user":"kaiweili","type":"user"},{"_id":"6a2f32c2a14a4189799755e0","avatarUrl":"/avatars/bf3e4ecdfc251c9280cbeddf9cd65f87.svg","isPro":false,"fullname":"Yifan Gao","user":"gao-yifan","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.24551.md","query":{}}">
GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents
Abstract
Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.
Community
Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.24551 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.24551 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.24551 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.