We introduce SkillHarness, a framework for safer skill learning and reuse in computer-use agents. Existing skill-learning methods usually extract reusable skills from successful trajectories, but in dynamic environments this can encode unsafe behaviors from prompt injections, policy violations, pop-ups, or brittle UI-specific action flows.<br>SkillHarness treats skills as context-dependent capabilities rather than fixed scripts. It builds explicit skill boundaries from multiple supervision signals, including successful trajectories, failures, and detected risks, then uses a selective reuse mechanism that activates skills only when their safety and applicability conditions are satisfied.<br>Across ST-WebAgentBench, WASP, OS-Harm, and OpenApps, SkillHarness reduces unsafe learned skills and improves robustness under adversarial and changing environments. We hope this work provides a useful step toward continual skill learning for computer-use agents that is not only effective, but also safer and more reliable in real-world settings.</p>\n<p>GitHub: <a href=\"https://github.com/YurunChen/SkillHarness\" rel=\"nofollow\">https://github.com/YurunChen/SkillHarness</a><br>Arxiv: <a href=\"https://arxiv.org/abs/2606.20636\" rel=\"nofollow\">https://arxiv.org/abs/2606.20636</a></p>\n","updatedAt":"2026-06-23T08:04:35.650Z","author":{"_id":"6747285ed34bd6f05080ddda","avatarUrl":"/avatars/ea63de7348aaabc8cff44e76207ba65c.svg","fullname":"Yurun Chen","name":"yurun-chen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8936313986778259},"editors":["yurun-chen"],"editorAvatarUrls":["/avatars/ea63de7348aaabc8cff44e76207ba65c.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.20636","authors":[{"_id":"6a39fcccfdcd3514343bb549","name":"Yurun Chen","hidden":false},{"_id":"6a39fcccfdcd3514343bb54a","name":"Biao Yi","hidden":false},{"_id":"6a39fcccfdcd3514343bb54b","name":"Keting Yin","hidden":false},{"_id":"6a39fcccfdcd3514343bb54c","name":"Shengyu Zhang","hidden":false}],"publishedAt":"2026-06-02T00:00:00.000Z","submittedOnDailyAt":"2026-06-23T00:00:00.000Z","title":"SkillHarness: Harnessing Safe Skills for Computer-Use Agents","submittedOnDailyBy":{"_id":"6747285ed34bd6f05080ddda","avatarUrl":"/avatars/ea63de7348aaabc8cff44e76207ba65c.svg","isPro":false,"fullname":"Yurun Chen","user":"yurun-chen","type":"user","name":"yurun-chen"},"summary":"Computer-Use Agents (CUAs) are increasingly deployed in dynamic interactive environments, creating a growing need for continual skill learning during interaction. Recent approaches address this challenge by learning reusable skills from successful trajectories. However, these skill learning methods largely assume static and safe environments, overlooking risks from adversarial interactions (e.g., prompt injections) and environmental dynamics (e.g., pop-ups). In dynamic settings, such assumptions can lead to risky skill learning and brittle execution, undermining the reliability of CUAs. This raises the question: how can CUAs learn and use skills safely in dynamic environments? To address this problem, we propose SkillHarness, a framework for safe skill harnessing in dynamic environments. SkillHarness moves beyond static skill abstractions by modeling skill learning and utilization as a safety-constrained interaction process. Specifically, we introduce the skill boundary that leverages multi-source supervision signals to identify safe skills from interaction trajectories, and construct self-improving safety constraints throughout the skill lifecycle. In addition, SkillHarness introduces selective skill reuse, where tasks are guided to decompose according to context and completed through the selective activation of skill subsets. Our experiments demonstrate that SkillHarness significantly reduces the unsafe rate of learned skills by 57.1% and consistently improves execution stability under dynamic environmental changes, outperforming existing baselines.","upvotes":16,"discussionId":"6a39fcccfdcd3514343bb54d","githubRepo":"https://github.com/YurunChen/SkillHarness","githubRepoAddedBy":"user","ai_summary":"SkillHarness is a framework that enables computer-use agents to safely learn and execute skills in dynamic environments by incorporating safety constraints and adaptive skill selection mechanisms.","ai_keywords":["computer-use agents","continual skill learning","dynamic environments","adversarial interactions","environmental dynamics","skill learning","safety-constrained interaction","skill boundary","multi-source supervision signals","self-improving safety constraints","selective skill reuse","skill lifecycle"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":9,"organization":{"_id":"6345aadf5efccdc07f1365a5","name":"ZhejiangUniversity","fullname":"Zhejiang University","avatar":"https://www.gravatar.com/avatar/d1d414628877bec2958f95ad283c15e7?d=retro&size=100"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6747285ed34bd6f05080ddda","avatarUrl":"/avatars/ea63de7348aaabc8cff44e76207ba65c.svg","isPro":false,"fullname":"Yurun Chen","user":"yurun-chen","type":"user"},{"_id":"655abdfe5507a50772d81e39","avatarUrl":"/avatars/664ff16d2476bae84c45608aa98a0ac5.svg","isPro":false,"fullname":"QuZhan","user":"QuZZZZZZZ","type":"user"},{"_id":"671a6de96a427e75e07b0425","avatarUrl":"/avatars/cd16c46389639b9f6b74afa52709de6e.svg","isPro":false,"fullname":"易标","user":"EaseJimmy","type":"user"},{"_id":"6a01d742551818f067f292d5","avatarUrl":"/avatars/9e39640c5a40fb2e617fe298174e5387.svg","isPro":false,"fullname":"xinchen xu","user":"b7277","type":"user"},{"_id":"652264ab578e7da0d75730db","avatarUrl":"/avatars/1fc1afc1bbb06c212eb33e28eb62f0d3.svg","isPro":false,"fullname":"lyh","user":"sadnessll","type":"user"},{"_id":"66a697248b85fa8a34005aed","avatarUrl":"/avatars/3d7f26676ad7e91e540929f1f04b33fd.svg","isPro":false,"fullname":"Tao Xiong","user":"YuanDaozeiii","type":"user"},{"_id":"69169b5342da384f7cb308a8","avatarUrl":"/avatars/8f5b42c0c92ea01fd4a22b1e41a98d4c.svg","isPro":false,"fullname":"li","user":"li1402359854","type":"user"},{"_id":"6703b17955012f023b427699","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6703b17955012f023b427699/G0mJpWR-d--aIMDVjV15g.jpeg","isPro":false,"fullname":"Zhouzhou Shen","user":"zhouzhoushen","type":"user"},{"_id":"68674e845e518d29eddf4175","avatarUrl":"/avatars/bfaf72c472c281b8a9e2cf672b7e8b5f.svg","isPro":false,"fullname":"xqb","user":"caklsjdlkamdkam","type":"user"},{"_id":"685bb4155d06780768d949d0","avatarUrl":"/avatars/db652e14d5a29001cd7b1651c5b77f96.svg","isPro":false,"fullname":"yj","user":"jackvae","type":"user"},{"_id":"6736b996e02ecde8509eba7f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/i1WayUAX1N_T7_31WDQtA.png","isPro":false,"fullname":"Junhao Chen","user":"chenjunhao100","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6345aadf5efccdc07f1365a5","name":"ZhejiangUniversity","fullname":"Zhejiang University","avatar":"https://www.gravatar.com/avatar/d1d414628877bec2958f95ad283c15e7?d=retro&size=100"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.20636.md","query":{}}">
SkillHarness: Harnessing Safe Skills for Computer-Use Agents
Abstract
SkillHarness is a framework that enables computer-use agents to safely learn and execute skills in dynamic environments by incorporating safety constraints and adaptive skill selection mechanisms.
Computer-Use Agents (CUAs) are increasingly deployed in dynamic interactive environments, creating a growing need for continual skill learning during interaction. Recent approaches address this challenge by learning reusable skills from successful trajectories. However, these skill learning methods largely assume static and safe environments, overlooking risks from adversarial interactions (e.g., prompt injections) and environmental dynamics (e.g., pop-ups). In dynamic settings, such assumptions can lead to risky skill learning and brittle execution, undermining the reliability of CUAs. This raises the question: how can CUAs learn and use skills safely in dynamic environments? To address this problem, we propose SkillHarness, a framework for safe skill harnessing in dynamic environments. SkillHarness moves beyond static skill abstractions by modeling skill learning and utilization as a safety-constrained interaction process. Specifically, we introduce the skill boundary that leverages multi-source supervision signals to identify safe skills from interaction trajectories, and construct self-improving safety constraints throughout the skill lifecycle. In addition, SkillHarness introduces selective skill reuse, where tasks are guided to decompose according to context and completed through the selective activation of skill subsets. Our experiments demonstrate that SkillHarness significantly reduces the unsafe rate of learned skills by 57.1% and consistently improves execution stability under dynamic environmental changes, outperforming existing baselines.
Community
We introduce SkillHarness, a framework for safer skill learning and reuse in computer-use agents. Existing skill-learning methods usually extract reusable skills from successful trajectories, but in dynamic environments this can encode unsafe behaviors from prompt injections, policy violations, pop-ups, or brittle UI-specific action flows.
SkillHarness treats skills as context-dependent capabilities rather than fixed scripts. It builds explicit skill boundaries from multiple supervision signals, including successful trajectories, failures, and detected risks, then uses a selective reuse mechanism that activates skills only when their safety and applicability conditions are satisfied.
Across ST-WebAgentBench, WASP, OS-Harm, and OpenApps, SkillHarness reduces unsafe learned skills and improves robustness under adversarial and changing environments. We hope this work provides a useful step toward continual skill learning for computer-use agents that is not only effective, but also safer and more reliable in real-world settings.
GitHub: https://github.com/YurunChen/SkillHarness
Arxiv: https://arxiv.org/abs/2606.20636
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.20636 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.20636 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.20636 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.