We propose the <strong>Zero-CoT Probe (ZCP)</strong>, a black-box method for detecting data contamination in LLMs — including <em>evasive</em> contamination via paraphrasing. ZCP truncates the chain-of-thought (CoT) and forces the model to emit the final answer directly, exposing memorized shortcut mappings that surface-level detectors miss. Contamination strength is quantified by <strong>Contamination Confidence</strong>, a Bayesian posterior in [0.5, 1) calibrated from a frequentist significance test: a value near 0.5 indicates no statistical evidence of contamination, while values approaching 1 indicate increasingly definitive contamination.<br><a href=\"https://cdn-uploads.huggingface.co/production/uploads/677cc30a38114a126d10b6d2/VDKKJc2xDWWoGy3tqIrYb.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/677cc30a38114a126d10b6d2/VDKKJc2xDWWoGy3tqIrYb.png\" alt=\"CoT_truncation_illustration\"></a></p>\n","updatedAt":"2026-05-25T16:37:10.707Z","author":{"_id":"677cc30a38114a126d10b6d2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Izaxu-ihgWADuVfA1ZAH3.png","fullname":"Yifan Lan","name":"yflantmy","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7732519507408142},"editors":["yflantmy"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Izaxu-ihgWADuVfA1ZAH3.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.21856","authors":[{"_id":"6a13fb9a4d9e8d8602d2037d","name":"Yifan Lan","hidden":false},{"_id":"6a13fb9a4d9e8d8602d2037e","name":"Yuanpu Cao","hidden":false},{"_id":"6a13fb9a4d9e8d8602d2037f","name":"Hanyu Wang","hidden":false},{"_id":"6a13fb9a4d9e8d8602d20380","name":"Lu Lin","hidden":false},{"_id":"6a13fb9a4d9e8d8602d20381","name":"Jinghui Chen","hidden":false}],"publishedAt":"2026-05-21T00:00:00.000Z","submittedOnDailyAt":"2026-05-25T00:00:00.000Z","title":"The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation","submittedOnDailyBy":{"_id":"677cc30a38114a126d10b6d2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Izaxu-ihgWADuVfA1ZAH3.png","isPro":false,"fullname":"Yifan Lan","user":"yflantmy","type":"user","name":"yflantmy"},"summary":"Large language models (LLMs) have demonstrated impressive reasoning abilities across a wide range of tasks, but data contamination undermines the objective evaluation of these capabilities. This problem is further exacerbated by malicious model publishers who use evasive, or indirect, contamination strategies, such as paraphrasing benchmark data to evade existing detection methods and artificially boost leaderboard performance. Current approaches struggle to reliably detect such stealthy contamination. In this work, we uncover a critical phenomenon: a model's generated reasoning steps actively mask its underlying memorization. Inspired by this, we propose the Zero-CoT Probe (ZCP), a novel black-box detection method that deliberately truncates the entire Chain-of-Thought (CoT) process to expose latent shortcut mappings. To further isolate memorization from the model's intrinsic problem-solving capabilities, ZCP compares the model's zero-CoT performance on the original benchmark against an isomorphically perturbed reference dataset. Furthermore, we introduce Contamination Confidence, a metric that quantifies both the likelihood and severity of contamination, moving beyond simple binary classifications. Extensive experiments on both previously identified contaminated models and specially fine-tuned contaminated models demonstrate that ZCP robustly detects both direct and evasive data contamination. The code for ZCP is accessible at https://github.com/Yifan-Lan/zero-cot-probe.","upvotes":3,"discussionId":"6a13fb9a4d9e8d8602d20382","githubRepo":"https://github.com/Yifan-Lan/zero-cot-probe","githubRepoAddedBy":"user","ai_summary":"A black-box detection method called Zero-CoT Probe is introduced to identify data contamination in large language models by truncating reasoning processes and comparing performance on original and perturbed datasets.","ai_keywords":["Chain-of-Thought","black-box detection","data contamination","memorization","zero-CoT Probe","Contamination Confidence","isomorphically perturbed reference dataset"],"githubStars":1},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6a148145486a5aab39d4a024","avatarUrl":"/avatars/a4705bbe60d2ece519183889a7dbe28b.svg","isPro":false,"fullname":"user4765","user":"user4765","type":"user"},{"_id":"677cc30a38114a126d10b6d2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Izaxu-ihgWADuVfA1ZAH3.png","isPro":false,"fullname":"Yifan Lan","user":"yflantmy","type":"user"},{"_id":"661ab1f1fa3b144a381fa454","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png","isPro":true,"fullname":"Urro","user":"urroxyz","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.21856.md"}">
The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation
Abstract
A black-box detection method called Zero-CoT Probe is introduced to identify data contamination in large language models by truncating reasoning processes and comparing performance on original and perturbed datasets.
AI-generated summary
Large language models (LLMs) have demonstrated impressive reasoning abilities across a wide range of tasks, but data contamination undermines the objective evaluation of these capabilities. This problem is further exacerbated by malicious model publishers who use evasive, or indirect, contamination strategies, such as paraphrasing benchmark data to evade existing detection methods and artificially boost leaderboard performance. Current approaches struggle to reliably detect such stealthy contamination. In this work, we uncover a critical phenomenon: a model's generated reasoning steps actively mask its underlying memorization. Inspired by this, we propose the Zero-CoT Probe (ZCP), a novel black-box detection method that deliberately truncates the entire Chain-of-Thought (CoT) process to expose latent shortcut mappings. To further isolate memorization from the model's intrinsic problem-solving capabilities, ZCP compares the model's zero-CoT performance on the original benchmark against an isomorphically perturbed reference dataset. Furthermore, we introduce Contamination Confidence, a metric that quantifies both the likelihood and severity of contamination, moving beyond simple binary classifications. Extensive experiments on both previously identified contaminated models and specially fine-tuned contaminated models demonstrate that ZCP robustly detects both direct and evasive data contamination. The code for ZCP is accessible at https://github.com/Yifan-Lan/zero-cot-probe.
Community
We propose the Zero-CoT Probe (ZCP), a black-box method for detecting data contamination in LLMs — including evasive contamination via paraphrasing. ZCP truncates the chain-of-thought (CoT) and forces the model to emit the final answer directly, exposing memorized shortcut mappings that surface-level detectors miss. Contamination strength is quantified by Contamination Confidence, a Bayesian posterior in [0.5, 1) calibrated from a frequentist significance test: a value near 0.5 indicates no statistical evidence of contamination, while values approaching 1 indicate increasingly definitive contamination.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.21856 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.21856 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.21856 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.