54% harmful safety-violation rate, even for the best model, is a strong signal that current alignment is still not sufficient for realistic project settings.\n","html":"<p>SABER shifts coding-agent safety evaluation from single-turn refusal behavior to the final state of a realistic, stateful workspace after multi-step agent actions. This is an important benchmark direction because many safety failures in coding agents emerge operationally, through file edits, commands, and environment changes, rather than in isolated model responses. The reported &gt;54% harmful safety-violation rate, even for the best model, is a strong signal that current alignment is still not sufficient for realistic project settings.</p>\n","updatedAt":"2026-06-05T16:18:19.432Z","author":{"_id":"6a1e7928cae1ff6e422adeba","avatarUrl":"/avatars/aa6cc7828e8a17a835f119f1ed1e00c8.svg","fullname":"Qi HU","name":"lingfengzhe","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.9011849164962769},"editors":["lingfengzhe"],"editorAvatarUrls":["/avatars/aa6cc7828e8a17a835f119f1ed1e00c8.svg"],"reactions":[],"isReport":false}},{"id":"6a237b5998f0490648ff44f7","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":362,"isUserFollowing":false},"createdAt":"2026-06-06T01:43:53.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents](https://huggingface.co/papers/2606.02302) (2026)\n* [Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents](https://huggingface.co/papers/2605.16282) (2026)\n* [SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces](https://huggingface.co/papers/2605.12015) (2026)\n* [When Agents Overtrust Environmental Evidence: An Extensible Agentic Framework for Benchmarking Evidence-Grounding Defects in LLM Agents](https://huggingface.co/papers/2605.08828) (2026)\n* [Measuring Safety Alignment Effects in Autonomous Security Agents](https://huggingface.co/papers/2605.19722) (2026)\n* [Benchmarking Autonomous Agents against Temporal, Spatial, and Semantic Evasions](https://huggingface.co/papers/2605.22321) (2026)\n* [Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw](https://huggingface.co/papers/2605.11047) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"<p>This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. </p>\n<p>The following papers were recommended by the Semantic Scholar API </p>\n<ul>\n<li><a href=\"https://huggingface.co/papers/2606.02302\">SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.16282\">Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.12015\">SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.08828\">When Agents Overtrust Environmental Evidence: An Extensible Agentic Framework for Benchmarking Evidence-Grounding Defects in LLM Agents</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.19722\">Measuring Safety Alignment Effects in Autonomous Security Agents</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.22321\">Benchmarking Autonomous Agents against Temporal, Spatial, and Semantic Evasions</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.11047\">Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw</a> (2026)</li>\n</ul>\n<p> Please give a thumbs up to this comment if you found it helpful!</p>\n<p> If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space</p>\n<p> You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code><span class=\"SVELTE_PARTIAL_HYDRATER contents\" data-target=\"UserMention\" data-props=\"{&quot;user&quot;:&quot;librarian-bot&quot;}\"><span class=\"inline-block\"><span class=\"contents\"><a href=\"/librarian-bot\">@<span class=\"underline\">librarian-bot</span></a></span> </span></span> recommend</code></p>\n","updatedAt":"2026-06-06T01:43:53.484Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":362,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7271891236305237},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.01317","authors":[{"_id":"6a20d70115100c5272a84630","user":{"_id":"6a1e7928cae1ff6e422adeba","avatarUrl":"/avatars/aa6cc7828e8a17a835f119f1ed1e00c8.svg","isPro":false,"fullname":"Qi HU","user":"lingfengzhe","type":"user","name":"lingfengzhe"},"name":"Qi Hu","status":"claimed_verified","statusLastChangedAt":"2026-06-05T15:08:48.548Z","hidden":false},{"_id":"6a20d70115100c5272a84631","name":"Yifeng Tang","hidden":false},{"_id":"6a20d70115100c5272a84632","name":"Qinghua Wang","hidden":false},{"_id":"6a20d70115100c5272a84633","name":"Lanyang Zhao","hidden":false},{"_id":"6a20d70115100c5272a84634","name":"Pengji Zhang","hidden":false},{"_id":"6a20d70115100c5272a84635","name":"Yuhao Qing","hidden":false},{"_id":"6a20d70115100c5272a84636","name":"Xin Yao","hidden":false},{"_id":"6a20d70115100c5272a84637","name":"Dong Huang","hidden":false},{"_id":"6a20d70115100c5272a84638","name":"Lin Zhang","hidden":false},{"_id":"6a20d70115100c5272a84639","user":{"_id":"69e054ca938fe0bcb78c6912","avatarUrl":"/avatars/2e94517639122c4884f1f5e265d9e773.svg","isPro":false,"fullname":"zhuoran Ji","user":"jizhuoran","type":"user","name":"jizhuoran"},"name":"Zhuoran Ji","status":"claimed_verified","statusLastChangedAt":"2026-06-04T12:39:59.966Z","hidden":false}],"publishedAt":"2026-05-31T00:00:00.000Z","submittedOnDailyAt":"2026-06-05T00:00:00.000Z","title":"SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces","submittedOnDailyBy":{"_id":"6a1e7928cae1ff6e422adeba","avatarUrl":"/avatars/aa6cc7828e8a17a835f119f1ed1e00c8.svg","isPro":false,"fullname":"Qi HU","user":"lingfengzhe","type":"user","name":"lingfengzhe"},"summary":"Large language models are increasingly deployed as coding agents, shifting safety from individual responses to action sequences. Existing benchmarks, however, primarily assess whether models refuse unsafe prompts, leaving impacts on stateful workspaces largely unexamined. We present SABER, a benchmark for environment-aware operational safety that places models in realistic agent-style projects and evaluates safety from the final environment state after a sequence of actions. Beyond binary safety-violation reports, SABER categorizes violations by cause, enabling analysis of model-specific safety profiles. Our evaluations show that even the best-performing model has more than a 54% harmful safety-violation rate (HSR), suggesting that current alignment remains insufficient for realistic project environments. SABER further reveals distinct safety profiles across models. Our benchmark is publicly available at https://github.com/sssr-lab/saber.","upvotes":0,"discussionId":"6a20d70215100c5272a8463a","ai_summary":"Large language models deployed as coding agents exhibit significant safety violations in realistic project environments, necessitating new evaluation approaches beyond simple prompt refusal assessments.","ai_keywords":["large language models","coding agents","safety violations","environment-aware operational safety","agent-style projects","stateful workspaces","alignment","harmful safety-violation rate"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.01317.md"}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2606.01317 in a model README.md to link it from this page.

Cite arxiv.org/abs/2606.01317 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.