Large language model evaluations typically focus on performance under nominal conditions, creating an illusion of capability where models comfortably walk a narrow, highly optimized generation corridor. In real-world deployments, however, complex system prompts, safety guardrails, and structural constraints continuously force models off this nominal path, driving a divergence between benchmark scores and deployment performance. To address this issue, we introduce Decoding-Level Taboo, a zero-prompt diagnostic stress test that intervenes directly in logit space at runtime, forcing models out of their nominal paths. By dynamically masking primary candidate tokens at word boundaries, Taboo forces machine circumlocution.<br>Evaluating Taboo across several open-weight model families reveals that off-path robustness is heavily influenced by both parameter scale and post-training instruction alignment, with robustness generally improving with model size and alignment. Beyond the results presented in this paper, Taboo provides a novel primitive for generating diverse synthetic datasets, stress-testing runtime safety guardrails, and auditing model reliability prior to real-world deployment.</p>\n","updatedAt":"2026-08-12T07:04:58.432Z","author":{"_id":"64f31365ed48e3bb9c487d5d","avatarUrl":"/avatars/979c1979eadbd4529c95b925bbb58d78.svg","fullname":"Gonzalo","name":"gonzmart","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.911615788936615},"editors":["gonzmart"],"editorAvatarUrls":["/avatars/979c1979eadbd4529c95b925bbb58d78.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.09900","authors":[{"_id":"6a7c1a951653ef87c6af1da7","name":"Tadanobu Chuyo Kamijo","hidden":false},{"_id":"6a7c1a951653ef87c6af1da8","name":"Ori Rottenstreich","hidden":false},{"_id":"6a7c1a951653ef87c6af1da9","name":"Javier Conde","hidden":false},{"_id":"6a7c1a951653ef87c6af1daa","name":"Gonzalo Martínez","hidden":false},{"_id":"6a7c1a951653ef87c6af1dab","name":"Pedro Reviriego","hidden":false}],"publishedAt":"2026-08-10T00:00:00.000Z","submittedOnDailyAt":"2026-08-12T00:00:00.000Z","title":"Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness","submittedOnDailyBy":{"_id":"64f31365ed48e3bb9c487d5d","avatarUrl":"/avatars/979c1979eadbd4529c95b925bbb58d78.svg","isPro":false,"fullname":"Gonzalo","user":"gonzmart","type":"user","name":"gonzmart"},"summary":"Large language model evaluations typically focus on performance under nominal conditions, creating an illusion of capability where models comfortably walk a narrow, highly optimized generation corridor. In real-world deployments, however, complex system prompts, safety guardrails, and structural constraints continuously force models off this nominal path, driving a divergence between benchmark scores and deployment performance. To address this issue, we introduce Decoding-Level Taboo, a zero-prompt diagnostic stress test that intervenes directly in logit space at runtime, forcing models out of their nominal paths. By dynamically masking primary candidate tokens at word boundaries, Taboo forces machine circumlocution.\n Evaluating Taboo across several open-weight model families reveals that off-path robustness is heavily influenced by both parameter scale and post-training instruction alignment, with robustness generally improving with model size and alignment. Beyond the results presented in this paper, Taboo provides a novel primitive for generating diverse synthetic datasets, stress-testing runtime safety guardrails, and auditing model reliability prior to real-world deployment.","upvotes":7,"discussionId":"6a7c1a961653ef87c6af1dac","ai_summary":"Decoding-Level Taboo is a runtime logit-space stress test that reveals how large language models handle off-nominal generation paths, showing that robustness depends on scale and instruction alignment.","ai_keywords":["Decoding-Level Taboo","logit space","zero-prompt diagnostic","masking candidate tokens","machine circumlocution","off-path robustness","parameter scale","post-training instruction alignment","runtime safety guardrails"],"ai_summary_model":"thinkingmachines/Inkling-Small"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64f31365ed48e3bb9c487d5d","avatarUrl":"/avatars/979c1979eadbd4529c95b925bbb58d78.svg","isPro":false,"fullname":"Gonzalo","user":"gonzmart","type":"user"},{"_id":"6574f66b06fdcd4ca9491299","avatarUrl":"/avatars/e31821949d75efb750ab2d9ebe12b9a8.svg","isPro":false,"fullname":"pedro reviriego","user":"reviriego","type":"user"},{"_id":"69799be76c39c1575660d12b","avatarUrl":"/avatars/d304718d5bea13333a11bd4dd242a7ea.svg","isPro":false,"fullname":"sda","user":"nonneeded","type":"user"},{"_id":"69bcd87e5cb8f0d62bf19480","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_PvNdGUrW3sJFN7xR1dsB.png","isPro":false,"fullname":"小川花","user":"scarlettbaker","type":"user"},{"_id":"69bb851efe142225ca822bb3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/-Y-b_2ifh4Jo_r1m5JNSH.jpeg","isPro":false,"fullname":"Jaxon Young","user":"jackf5","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"66852115f8c3f32152346733","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66852115f8c3f32152346733/EuApIooUvMesCSr27Auze.jpeg","isPro":false,"fullname":"Javier Conde","user":"javicond3","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.09900.md","query":{}}">
Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness
Abstract
Decoding-Level Taboo is a runtime logit-space stress test that reveals how large language models handle off-nominal generation paths, showing that robustness depends on scale and instruction alignment.
Large language model evaluations typically focus on performance under nominal conditions, creating an illusion of capability where models comfortably walk a narrow, highly optimized generation corridor. In real-world deployments, however, complex system prompts, safety guardrails, and structural constraints continuously force models off this nominal path, driving a divergence between benchmark scores and deployment performance. To address this issue, we introduce Decoding-Level Taboo, a zero-prompt diagnostic stress test that intervenes directly in logit space at runtime, forcing models out of their nominal paths. By dynamically masking primary candidate tokens at word boundaries, Taboo forces machine circumlocution.
Evaluating Taboo across several open-weight model families reveals that off-path robustness is heavily influenced by both parameter scale and post-training instruction alignment, with robustness generally improving with model size and alignment. Beyond the results presented in this paper, Taboo provides a novel primitive for generating diverse synthetic datasets, stress-testing runtime safety guardrails, and auditing model reliability prior to real-world deployment.
Community
Large language model evaluations typically focus on performance under nominal conditions, creating an illusion of capability where models comfortably walk a narrow, highly optimized generation corridor. In real-world deployments, however, complex system prompts, safety guardrails, and structural constraints continuously force models off this nominal path, driving a divergence between benchmark scores and deployment performance. To address this issue, we introduce Decoding-Level Taboo, a zero-prompt diagnostic stress test that intervenes directly in logit space at runtime, forcing models out of their nominal paths. By dynamically masking primary candidate tokens at word boundaries, Taboo forces machine circumlocution.
Evaluating Taboo across several open-weight model families reveals that off-path robustness is heavily influenced by both parameter scale and post-training instruction alignment, with robustness generally improving with model size and alignment. Beyond the results presented in this paper, Taboo provides a novel primitive for generating diverse synthetic datasets, stress-testing runtime safety guardrails, and auditing model reliability prior to real-world deployment.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.09900 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.09900 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.09900 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.