The code is available at: <a href=\"https://github.com/wang-yanting/PIMiner\" rel=\"nofollow\">https://github.com/wang-yanting/PIMiner</a></p>\n","updatedAt":"2026-08-06T04:29:35.872Z","author":{"_id":"656f7bf329b40892e83819c4","avatarUrl":"/avatars/4dace5c2516fb89947b509596e617f13.svg","fullname":"yanting wang","name":"SecureLLMSys","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.729256808757782},"editors":["SecureLLMSys"],"editorAvatarUrls":["/avatars/4dace5c2516fb89947b509596e617f13.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.05108","authors":[{"_id":"6a740ce8c5e410d076869b6b","name":"Yanting Wang","hidden":false},{"_id":"6a740ce8c5e410d076869b6c","name":"Chenlong Yin","hidden":false},{"_id":"6a740ce8c5e410d076869b6d","name":"Runpeng Geng","hidden":false},{"_id":"6a740ce8c5e410d076869b6e","name":"Jinyuan Jia","hidden":false}],"publishedAt":"2026-08-05T00:00:00.000Z","submittedOnDailyAt":"2026-08-06T00:00:00.000Z","title":"Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming","submittedOnDailyBy":{"_id":"656f7bf329b40892e83819c4","avatarUrl":"/avatars/4dace5c2516fb89947b509596e617f13.svg","isPro":true,"fullname":"yanting wang","user":"SecureLLMSys","type":"user","name":"SecureLLMSys"},"summary":"Prompt injection poses significant security risks to LLM agents. Efficient and effective red-teaming is therefore critical, both for evaluating these risks and for collecting training data to improve defenses. Existing state-of-the-art prompt injection red-teaming methods primarily rely on reinforcement learning (RL), producing attacker models that often generalize poorly to new target LLMs. In this work, we develop PIMiner, an agentic system for prompt injection red-teaming. During training, PIMiner is trained on a sequence of (dataset, target model) pairs and builds a strategy library from scratch. At test time, the learned strategy library can be directly transferred to a previously unseen target LLM without additional training. PIMiner requires only a small number of queries to a target agent (e.g., 10) per test sample. Experimental results demonstrate that PIMiner achieves strong performance. On IPIArena, it attains a 76.2% ASR against Gemini-2.5-Pro, 61.9% ASR against GPT-5.1, and 42.9% ASR against Claude-Sonnet-4.5. On AgentDojo, it achieves an 86.7% ASR against Gemini-2.5-Pro, 53.3% ASR against GPT-5.1, and 40.0% ASR against Claude-Sonnet-4.5.","upvotes":5,"discussionId":"6a740ce9c5e410d076869b6f","githubRepo":"https://github.com/wang-yanting/PIMiner","githubRepoAddedBy":"user","githubStars":1,"organization":{"_id":"623c72b6483fb88b35620a27","name":"PennState","fullname":"Pennsylvania State University","avatar":"https://www.gravatar.com/avatar/b3578c3ac0fec6a791bb71c68bf8a92a?d=retro&size=100"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6a6a925739a7f0911330c546","avatarUrl":"/avatars/21568a0081650e63921f1f70775f8bdd.svg","isPro":false,"fullname":"Timothy Lee","user":"timothylee","type":"user"},{"_id":"6a6c7a6302e1b4f71ce174ab","avatarUrl":"/avatars/459ff1a5509c507ce6b66c7284231b37.svg","isPro":false,"fullname":"Andrew Hernandez","user":"Ember-Andrew","type":"user"},{"_id":"6a6da7e008e6705013bc9e63","avatarUrl":"/avatars/c3d8bc73df06931ba549417b8af48e6b.svg","isPro":false,"fullname":"Linda Gonzalez","user":"NimbusLens","type":"user"},{"_id":"6a6dec00b82bf75bcb1f75fb","avatarUrl":"/avatars/cb67a57dc68a8cc508413bdc7cc59e49.svg","isPro":false,"fullname":"Brian Gonzalez","user":"david-8964122","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"623c72b6483fb88b35620a27","name":"PennState","fullname":"Pennsylvania State University","avatar":"https://www.gravatar.com/avatar/b3578c3ac0fec6a791bb71c68bf8a92a?d=retro&size=100"},"query":{}}">
Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming
Abstract
Prompt injection poses significant security risks to LLM agents. Efficient and effective red-teaming is therefore critical, both for evaluating these risks and for collecting training data to improve defenses. Existing state-of-the-art prompt injection red-teaming methods primarily rely on reinforcement learning (RL), producing attacker models that often generalize poorly to new target LLMs. In this work, we develop PIMiner, an agentic system for prompt injection red-teaming. During training, PIMiner is trained on a sequence of (dataset, target model) pairs and builds a strategy library from scratch. At test time, the learned strategy library can be directly transferred to a previously unseen target LLM without additional training. PIMiner requires only a small number of queries to a target agent (e.g., 10) per test sample. Experimental results demonstrate that PIMiner achieves strong performance. On IPIArena, it attains a 76.2% ASR against Gemini-2.5-Pro, 61.9% ASR against GPT-5.1, and 42.9% ASR against Claude-Sonnet-4.5. On AgentDojo, it achieves an 86.7% ASR against Gemini-2.5-Pro, 53.3% ASR against GPT-5.1, and 40.0% ASR against Claude-Sonnet-4.5.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.05108 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.05108 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.05108 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.