Large Language Model (LLM) safety has often been evaluated at the behavior level, which provides limited evidence of internal robustness, as these evaluations target outputs rather than representation-level vulnerability under intervention. We formalize this discrepancy as the audit gap: the difference between behavioral safety and robustness under intervention. To study this gap, we construct dissociated models that preserve safe outward behavior while remaining vulnerable in the latent space. We introduce an intervention-based evaluation framework to test model robustness through soft interventions in parameter and latent spaces, including harmful fine-tuning and layer-wise latent perturbations. To formalize the evaluation, we propose the Latent Vulnerability Score (LVS) to measure how easily harmful behavior can be elicited by bounded latent perturbations. Using this evaluation framework, we show that behavioral safety metrics are insufficient measures of representation-level robustness across multiple safely and unsafely aligned state-of-the-art models. Notably, dissociated models show substantially elevated LVSs despite comparable refusal behavior under harmful intervention, with intermediate representations being the most sensitive to intervention. Our results suggest that behavioral safety evaluation alone provides an incomplete picture of model robustness, motivating representation-aware audits of latent vulnerability and observable behavior.</p>\n","updatedAt":"2026-06-10T18:01:41.434Z","author":{"_id":"6058351db2c84d5386b3afe5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6058351db2c84d5386b3afe5/mS1Nu9i5VYgAnHSKXz-b4.jpeg","fullname":"Enyi (Olivia) Jiang","name":"EnyiJiang","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8979530930519104},"editors":["EnyiJiang"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6058351db2c84d5386b3afe5/mS1Nu9i5VYgAnHSKXz-b4.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.08044","authors":[{"_id":"6a29a6446ae15f2243580989","name":"Enyi Jiang","hidden":false},{"_id":"6a29a6446ae15f224358098a","name":"Anders Gjølbye","hidden":false},{"_id":"6a29a6446ae15f224358098b","name":"Yibo Jacky Zhang","hidden":false},{"_id":"6a29a6446ae15f224358098c","name":"Sanmi Koyejo","hidden":false}],"publishedAt":"2026-06-06T00:00:00.000Z","submittedOnDailyAt":"2026-06-10T00:00:00.000Z","title":"When Behavioral Safety Evaluation Fails: A Representation-Level Perspective","submittedOnDailyBy":{"_id":"6058351db2c84d5386b3afe5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6058351db2c84d5386b3afe5/mS1Nu9i5VYgAnHSKXz-b4.jpeg","isPro":false,"fullname":"Enyi (Olivia) Jiang","user":"EnyiJiang","type":"user","name":"EnyiJiang"},"summary":"Large Language Model (LLM) safety has often been evaluated at the behavior level, which provides limited evidence of internal robustness, as these evaluations target outputs rather than representation-level vulnerability under intervention. We formalize this discrepancy as the audit gap: the difference between behavioral safety and robustness under intervention. To study this gap, we construct dissociated models that preserve safe outward behavior while remaining vulnerable in the latent space. We introduce an intervention-based evaluation framework to test model robustness through soft interventions in parameter and latent spaces, including harmful fine-tuning and layer-wise latent perturbations. To formalize the evaluation, we propose the Latent Vulnerability Score (LVS) to measure how easily harmful behavior can be elicited by bounded latent perturbations. Using this evaluation framework, we show that behavioral safety metrics are insufficient measures of representation-level robustness across multiple safely and unsafely aligned state-of-the-art models. Notably, dissociated models show substantially elevated LVSs despite comparable refusal behavior under harmful intervention, with intermediate representations being the most sensitive to intervention. Our results suggest that behavioral safety evaluation alone provides an incomplete picture of model robustness, motivating representation-aware audits of latent vulnerability and observable behavior.","upvotes":1,"discussionId":"6a29a6456ae15f224358098d","ai_summary":"Behavioral safety evaluations of large language models provide incomplete insights into internal robustness, as demonstrated by the audit gap between observable outputs and latent space vulnerabilities revealed through intervention-based testing.","ai_keywords":["large language model","behavioral safety","representation-level robustness","audit gap","dissociated models","soft interventions","parameter space","latent space","harmful fine-tuning","layer-wise latent perturbations","Latent Vulnerability Score","LVS"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"6900c65ccd6f5a08e9683db2","name":"StanfordUniversityy","fullname":"Stanford University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6900c5742cc80701f360da45/6RB8XN4KUNvDsQlYhg0gl.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6058351db2c84d5386b3afe5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6058351db2c84d5386b3afe5/mS1Nu9i5VYgAnHSKXz-b4.jpeg","isPro":false,"fullname":"Enyi (Olivia) Jiang","user":"EnyiJiang","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6900c65ccd6f5a08e9683db2","name":"StanfordUniversityy","fullname":"Stanford University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6900c5742cc80701f360da45/6RB8XN4KUNvDsQlYhg0gl.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.08044.md"}">
When Behavioral Safety Evaluation Fails: A Representation-Level Perspective
Abstract
Behavioral safety evaluations of large language models provide incomplete insights into internal robustness, as demonstrated by the audit gap between observable outputs and latent space vulnerabilities revealed through intervention-based testing.
Large Language Model (LLM) safety has often been evaluated at the behavior level, which provides limited evidence of internal robustness, as these evaluations target outputs rather than representation-level vulnerability under intervention. We formalize this discrepancy as the audit gap: the difference between behavioral safety and robustness under intervention. To study this gap, we construct dissociated models that preserve safe outward behavior while remaining vulnerable in the latent space. We introduce an intervention-based evaluation framework to test model robustness through soft interventions in parameter and latent spaces, including harmful fine-tuning and layer-wise latent perturbations. To formalize the evaluation, we propose the Latent Vulnerability Score (LVS) to measure how easily harmful behavior can be elicited by bounded latent perturbations. Using this evaluation framework, we show that behavioral safety metrics are insufficient measures of representation-level robustness across multiple safely and unsafely aligned state-of-the-art models. Notably, dissociated models show substantially elevated LVSs despite comparable refusal behavior under harmful intervention, with intermediate representations being the most sensitive to intervention. Our results suggest that behavioral safety evaluation alone provides an incomplete picture of model robustness, motivating representation-aware audits of latent vulnerability and observable behavior.
Community
Large Language Model (LLM) safety has often been evaluated at the behavior level, which provides limited evidence of internal robustness, as these evaluations target outputs rather than representation-level vulnerability under intervention. We formalize this discrepancy as the audit gap: the difference between behavioral safety and robustness under intervention. To study this gap, we construct dissociated models that preserve safe outward behavior while remaining vulnerable in the latent space. We introduce an intervention-based evaluation framework to test model robustness through soft interventions in parameter and latent spaces, including harmful fine-tuning and layer-wise latent perturbations. To formalize the evaluation, we propose the Latent Vulnerability Score (LVS) to measure how easily harmful behavior can be elicited by bounded latent perturbations. Using this evaluation framework, we show that behavioral safety metrics are insufficient measures of representation-level robustness across multiple safely and unsafely aligned state-of-the-art models. Notably, dissociated models show substantially elevated LVSs despite comparable refusal behavior under harmful intervention, with intermediate representations being the most sensitive to intervention. Our results suggest that behavioral safety evaluation alone provides an incomplete picture of model robustness, motivating representation-aware audits of latent vulnerability and observable behavior.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.08044 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.08044 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.08044 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.