We introduce <strong>SecRespond</strong>, a benchmark for evaluating AI agents on real-world post-compromise incident response. It includes 10 reproducible cyber ranges across Linux and Windows, frozen forensic disk snapshots, synthetic security-product evidence, and expert-authored evaluation checklists that separately assess detection and remediation planning.</p>\n<p>Paper: <a href=\"https://arxiv.org/abs/2607.26791\" rel=\"nofollow\">https://arxiv.org/abs/2607.26791</a><br>Code and data: <a href=\"https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond\" rel=\"nofollow\">https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond</a><br>Dataset: <a href=\"https://huggingface.co/datasets/Alibaba-NLP/SecRespond\">https://huggingface.co/datasets/Alibaba-NLP/SecRespond</a></p>\n","updatedAt":"2026-07-30T06:17:10.049Z","author":{"_id":"61454d930aac1efe3e8d842e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61454d930aac1efe3e8d842e/sQELHgJP6c_547OrT8JZJ.jpeg","fullname":"boli","name":"bcol","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7730923891067505},"editors":["bcol"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/61454d930aac1efe3e8d842e/sQELHgJP6c_547OrT8JZJ.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.26791","authors":[{"_id":"6a6aeaab4463a8a84bdc402e","name":"Lehan Wang","hidden":false},{"_id":"6a6aeaab4463a8a84bdc402f","name":"Boli Chen","hidden":false},{"_id":"6a6aeaab4463a8a84bdc4030","name":"Ruixue Ding","hidden":false},{"_id":"6a6aeaab4463a8a84bdc4031","name":"Pengjun Xie","hidden":false},{"_id":"6a6aeaab4463a8a84bdc4032","name":"Jinwei Huang","hidden":false},{"_id":"6a6aeaab4463a8a84bdc4033","name":"Zhendong Liu","hidden":false},{"_id":"6a6aeaab4463a8a84bdc4034","name":"Shuo Wang","hidden":false},{"_id":"6a6aeaab4463a8a84bdc4035","name":"Tao Lei","hidden":false},{"_id":"6a6aeaab4463a8a84bdc4036","name":"Xin Ouyang","hidden":false},{"_id":"6a6aeaab4463a8a84bdc4037","name":"Xiaomeng Li","hidden":false}],"publishedAt":"2026-07-29T00:00:00.000Z","submittedOnDailyAt":"2026-07-30T00:00:00.000Z","title":"SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response","submittedOnDailyBy":{"_id":"61454d930aac1efe3e8d842e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61454d930aac1efe3e8d842e/sQELHgJP6c_547OrT8JZJ.jpeg","isPro":false,"fullname":"boli","user":"bcol","type":"user","name":"bcol"},"summary":"Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond.","upvotes":0,"discussionId":"6a6aeaac4463a8a84bdc4038","projectPage":"https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond","organization":{"_id":"661f98de142a51d630dbbcc4","name":"Alibaba-NLP","fullname":"Alibaba-NLP","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/63fc4c00a3c067e62899d32b/dfd_EcIfylvu3sdc2WMqX.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"organization":{"_id":"661f98de142a51d630dbbcc4","name":"Alibaba-NLP","fullname":"Alibaba-NLP","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/63fc4c00a3c067e62899d32b/dfd_EcIfylvu3sdc2WMqX.png"},"query":{}}">

Cite arxiv.org/abs/2607.26791 in a model README.md to link it from this page.

Cite arxiv.org/abs/2607.26791 in a Space README.md to link it from this page.