When do LLM self-reports actually predict behavior?</p>\n<p>Our paper shows that the answer depends strongly on evaluation design. Broad <strong>personality traits such as Big Five are weak predictors of specific LLM behaviors</strong>, while <strong>behavior-specific instruments such as the Theory of Planned Behavior can reveal stronger coherence under favorable conditions</strong>. But this coherence is selective: it varies by task, model, and session structure, and same-session effects may reflect context-window priming rather than stable behavioral tendencies.</p>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/62d4cfcd4b0933c48f45ae33/jidsv3Xcn_EjZaJLB7o95.jpeg\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/62d4cfcd4b0933c48f45ae33/jidsv3Xcn_EjZaJLB7o95.jpeg\" alt=\"psycohere_framework_v4_better\"></a></p>\n<p>We argue that psychometric probes should not be treated as direct evidence of stable model traits, but as evaluation tools whose validity depends on how they are designed and linked to behavior.</p>\n","updatedAt":"2026-06-12T21:04:58.424Z","author":{"_id":"62d4cfcd4b0933c48f45ae33","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62d4cfcd4b0933c48f45ae33/tCiLvQMywA0UKUkDl5Srl.png","fullname":"Rafal Kocielnik","name":"RKocielnik","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8852326273918152},"editors":["RKocielnik"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/62d4cfcd4b0933c48f45ae33/tCiLvQMywA0UKUkDl5Srl.png"],"reactions":[{"reaction":"🔥","users":["barryhpr","p-song1"],"count":2}],"isReport":false}},{"id":"6a2cb6ecc5212beb935e5ebb","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":364,"isUserFollowing":false},"createdAt":"2026-06-13T01:48:28.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [An LLM-Native Psychometric Instrument Does Not Predict LLM Behavior: Evidence Across 25 Models](https://huggingface.co/papers/2606.09843) (2026)\n* [ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation](https://huggingface.co/papers/2605.29791) (2026)\n* [Personality Anchoring for Social Simulation: Linking Personality, Social Behavior, and Interaction Success with LLM Agents](https://huggingface.co/papers/2606.06936) (2026)\n* [LLM-Based Educational Simulation: Evaluating Temporal Student Persona Stability Across ADHD Profiles](https://huggingface.co/papers/2605.06307) (2026)\n* [Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?](https://huggingface.co/papers/2605.16996) (2026)\n* [Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models](https://huggingface.co/papers/2606.11074) (2026)\n* [GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing](https://huggingface.co/papers/2606.00860) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"<p>This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. </p>\n<p>The following papers were recommended by the Semantic Scholar API </p>\n<ul>\n<li><a href=\"https://huggingface.co/papers/2606.09843\">An LLM-Native Psychometric Instrument Does Not Predict LLM Behavior: Evidence Across 25 Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.29791\">ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.06936\">Personality Anchoring for Social Simulation: Linking Personality, Social Behavior, and Interaction Success with LLM Agents</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.06307\">LLM-Based Educational Simulation: Evaluating Temporal Student Persona Stability Across ADHD Profiles</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.16996\">Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.11074\">Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.00860\">GenPT: Beyond Self-Report for Reliable LLM Psychometrics via Generative Projective Testing</a> (2026)</li>\n</ul>\n<p> Please give a thumbs up to this comment if you found it helpful!</p>\n<p> If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space</p>\n<p> You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code><span class=\"SVELTE_PARTIAL_HYDRATER contents\" data-target=\"UserMention\" data-props=\"{&quot;user&quot;:&quot;librarian-bot&quot;}\"><span class=\"inline-block\"><span class=\"contents\"><a href=\"/librarian-bot\">@<span class=\"underline\">librarian-bot</span></a></span> </span></span> recommend</code></p>\n","updatedAt":"2026-06-13T01:48:28.094Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":364,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7270586490631104},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.12730","authors":[{"_id":"6a2c729da0d4daae4285f05a","name":"Rafal Kocielnik","hidden":false},{"_id":"6a2c729da0d4daae4285f05b","name":"Pengrui Han","hidden":false},{"_id":"6a2c729da0d4daae4285f05c","name":"Peiyang Song","hidden":false},{"_id":"6a2c729da0d4daae4285f05d","name":"Myrl G. Marmarelis","hidden":false},{"_id":"6a2c729da0d4daae4285f05e","name":"Ramit Debnath","hidden":false},{"_id":"6a2c729da0d4daae4285f05f","name":"Dean Mobbs","hidden":false},{"_id":"6a2c729da0d4daae4285f060","name":"Anima Anandkumar","hidden":false},{"_id":"6a2c729da0d4daae4285f061","name":"R. Michael Alvarez","hidden":false}],"publishedAt":"2026-06-10T00:00:00.000Z","submittedOnDailyAt":"2026-06-12T00:00:00.000Z","title":"Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior","submittedOnDailyBy":{"_id":"62d4cfcd4b0933c48f45ae33","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62d4cfcd4b0933c48f45ae33/tCiLvQMywA0UKUkDl5Srl.png","isPro":true,"fullname":"Rafal Kocielnik","user":"RKocielnik","type":"user","name":"RKocielnik"},"summary":"Anticipating LLM behavioral tendencies from low-cost psychometric probes is critical for safe deployment, but only if self-reports (SR) reliably predict behavior. Recent work documented substantial SR-behavior dissociation in LLMs, but relied on broad personality traits (Big 5) that predict specific behaviors weakly, even in humans. Furthermore, the isolation of conversational sessions combined with weak context matching left open whether LLMs truly lack coherence or whether the conditions needed to detect such coherence were not met. We contrast Big 5 with the Theory of Planned Behavior (TPB), which measures intention targeted to a specific behavior and predicts human behavior substantially better than broad traits. We run experiments across four behavioral tasks and 11 frontier LLMs, while also varying session context and identity induction. We find that SR-behavior coherence exists but is selective. 1) Within a shared conversation, the Theory of Planned Behavior reaches human-level coherence; Big 5 does not. 2) Across separate conversations, coherence survives only for behaviors anchored outside the immediate prompt, such as implicit bias shaped by training, and collapses when behavior is strongly primed by context, as with sycophancy. 3) Persona prompting makes self-reports more consistent across conversations, but does not bring behavior into alignment. These findings suggest that coarse personality frameworks, such as Big 5 may not be the best tools for testing deployment behavior. More task- and behavior-specific instruments are needed, and even these must be evaluated across tasks and contexts.","upvotes":4,"discussionId":"6a2c729da0d4daae4285f062","githubRepo":"https://github.com/psychology-of-AI/Rethinking-Pyschometric-Eval-LLMs","githubRepoAddedBy":"user","ai_summary":"Psychometric assessments of LLM behavior reveal that specific behavioral frameworks like Theory of Planned Behavior show better coherence with actual responses than broad personality traits, particularly within shared conversations.","ai_keywords":["LLMs","psychometric probes","self-reports","Theory of Planned Behavior","Big 5","behavioral coherence","conversational sessions","context matching","persona prompting","implicit bias","sycophancy"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":1},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"664263ea8b2d38e53f04079c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/664263ea8b2d38e53f04079c/_7og0ggbPpEHcH6g__DF0.jpeg","isPro":false,"fullname":"Pengrui Han","user":"barryhpr","type":"user"},{"_id":"687ebb87416fcff56959d817","avatarUrl":"/avatars/8f13090dd6179bb18e8c8d205fd20131.svg","isPro":false,"fullname":"Keyang Xuan","user":"keyangx3","type":"user"},{"_id":"631e14ac473a6825f285e89d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/631e14ac473a6825f285e89d/K-6QnoeGLg8XFvbTMMdqA.jpeg","isPro":false,"fullname":"Yury Panikov","user":"panikov","type":"user"},{"_id":"649c5cf5c1ae48cf4d7dda34","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/649c5cf5c1ae48cf4d7dda34/bSJXATqkqBn8ypUy0OezY.jpeg","isPro":false,"fullname":"Peiyang Song","user":"p-song1","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.12730.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2606.12730 in a model README.md to link it from this page.

Cite arxiv.org/abs/2606.12730 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2606.12730 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.