TLDR: LLM agents are most unsafe at the very beginning of a session.</p>\n","updatedAt":"2026-06-12T15:33:13.037Z","author":{"_id":"64b6076821c601b486d217a3","avatarUrl":"/avatars/bfd680028e10683b6c0544eb24006246.svg","fullname":"Chung-En, Sun","name":"cesun","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9533320069313049},"editors":["cesun"],"editorAvatarUrls":["/avatars/bfd680028e10683b6c0544eb24006246.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.07867","authors":[{"_id":"6a2b25174957fcdd3aac052d","name":"Chung-En Sun","hidden":false},{"_id":"6a2b25174957fcdd3aac052e","name":"Linbo Liu","hidden":false},{"_id":"6a2b25174957fcdd3aac052f","name":"Tsui-Wei Weng","hidden":false}],"publishedAt":"2026-06-05T00:00:00.000Z","submittedOnDailyAt":"2026-06-12T00:00:00.000Z","title":"The Cold-Start Safety Gap in LLM Agents","submittedOnDailyBy":{"_id":"64b6076821c601b486d217a3","avatarUrl":"/avatars/bfd680028e10683b6c0544eb24006246.svg","isPro":false,"fullname":"Chung-En, Sun","user":"cesun","type":"user","name":"cesun"},"summary":"Are tool-calling LLM agents equally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a session and become substantially safer after a few regular agentic tasks -- a phenomenon we term the cold-start safety gap. To study this systematically, we introduce Safety Over Depth for Agents (SODA), a benchmark that controls how many regular agentic tasks the agent completes before encountering a safety threat, supporting up to 20 preceding tasks. Evaluating 7 models from 4 families, safety improves by 9--52% as the number of preceding regular agentic tasks increases from zero to twenty. Representation analysis confirms that model hidden states gradually shift toward a safety-aligned region as more preceding tasks are present. By systematically studying which part of the preceding conversation matters most, we find that the regular agentic tasks themselves are the primary driver of safety, while the agent's own prior responses have less effect on safety but are essential for preserving later utility. This conclusion is further supported by evaluation on open-source safety benchmarks (AgentHarm, Agent Safety Bench) and utility benchmarks (BFCL, API-Bank), confirming that warming up the agent with regular agentic tasks before deployment makes it safer and preserves full capability. Based on these findings, we recommend a simple deployment strategy: having the agent complete a few regular agentic tasks before possible exposure to safety-critical requests mitigates the cold-start safety gap. Our code is available at https://github.com/Trustworthy-ML-Lab/Agent-Cold-Start-Safety-Gap","upvotes":1,"discussionId":"6a2b25174957fcdd3aac0530","projectPage":"https://lilywenglab.github.io/Agent-Cold-Start-Safety-Gap/","githubRepo":"https://github.com/Trustworthy-ML-Lab/Agent-Cold-Start-Safety-Gap","githubRepoAddedBy":"user","ai_summary":"Tool-calling language model agents exhibit improved safety after initial interactions, with a systematic benchmark demonstrating enhanced security through prior task completion.","ai_keywords":["tool-calling LLM agents","cold-start safety gap","Safety Over Depth for Agents","representation analysis","hidden states","agentic tasks","safety benchmarks","utility benchmarks","deployment strategy"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64b6076821c601b486d217a3","avatarUrl":"/avatars/bfd680028e10683b6c0544eb24006246.svg","isPro":false,"fullname":"Chung-En, Sun","user":"cesun","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.07867.md","query":{}}">
The Cold-Start Safety Gap in LLM Agents
Abstract
Tool-calling language model agents exhibit improved safety after initial interactions, with a systematic benchmark demonstrating enhanced security through prior task completion.
Are tool-calling LLM agents equally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a session and become substantially safer after a few regular agentic tasks -- a phenomenon we term the cold-start safety gap. To study this systematically, we introduce Safety Over Depth for Agents (SODA), a benchmark that controls how many regular agentic tasks the agent completes before encountering a safety threat, supporting up to 20 preceding tasks. Evaluating 7 models from 4 families, safety improves by 9--52% as the number of preceding regular agentic tasks increases from zero to twenty. Representation analysis confirms that model hidden states gradually shift toward a safety-aligned region as more preceding tasks are present. By systematically studying which part of the preceding conversation matters most, we find that the regular agentic tasks themselves are the primary driver of safety, while the agent's own prior responses have less effect on safety but are essential for preserving later utility. This conclusion is further supported by evaluation on open-source safety benchmarks (AgentHarm, Agent Safety Bench) and utility benchmarks (BFCL, API-Bank), confirming that warming up the agent with regular agentic tasks before deployment makes it safer and preserves full capability. Based on these findings, we recommend a simple deployment strategy: having the agent complete a few regular agentic tasks before possible exposure to safety-critical requests mitigates the cold-start safety gap. Our code is available at https://github.com/Trustworthy-ML-Lab/Agent-Cold-Start-Safety-Gap
Community
TLDR: LLM agents are most unsafe at the very beginning of a session.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.07867 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.