We test whether LLM estimates obey the law of total probability, revealing widespread self-consistency failures and a macro fallacy in which aggregating fine-grained subpopulation estimates outperforms direct population-level prompting.</p>\n","updatedAt":"2026-07-17T06:14:38.742Z","author":{"_id":"680b7786134bf458c86b590e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/680b7786134bf458c86b590e/RhtoUvkknOkI8bhBfdoO5.png","fullname":"Patrik Wolf","name":"patrikwolf","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8559630513191223},"editors":["patrikwolf"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/680b7786134bf458c86b590e/RhtoUvkknOkI8bhBfdoO5.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.15277","authors":[{"_id":"6a59c72a6c2e371e6ca3832d","name":"Patrik Wolf","hidden":false},{"_id":"6a59c72a6c2e371e6ca3832e","name":"Thomas Kleine Buening","hidden":false},{"_id":"6a59c72a6c2e371e6ca3832f","name":"Andreas Krause","hidden":false},{"_id":"6a59c72a6c2e371e6ca38330","name":"Celestine Mendler-Dünner","hidden":false}],"publishedAt":"2026-07-16T00:00:00.000Z","submittedOnDailyAt":"2026-07-17T00:00:00.000Z","title":"Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models","submittedOnDailyBy":{"_id":"680b7786134bf458c86b590e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/680b7786134bf458c86b590e/RhtoUvkknOkI8bhBfdoO5.png","isPro":false,"fullname":"Patrik Wolf","user":"patrikwolf","type":"user","name":"patrikwolf"},"summary":"In-context learning is commonly interpreted as a form of conditional inference, in which the prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities. In particular, the law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population. In this work, we investigate to what extent LLM estimates adhere to this self-consistency principle. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine-grained subpopulations. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity. Applying this protocol across problem domains and state-of-the-art frontier models, we show widespread violations of basic consistency properties. An in-depth study of persona prompting reveals a pattern we call the macro fallacy: estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. This effect persists across variations in tree structure and estimation task, and can be partially recovered through implicit prompting. Together, these findings suggest that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates. This gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.","upvotes":4,"discussionId":"6a59c72b6c2e371e6ca38331","githubRepo":"https://github.com/patrikwolf/statistical_self_consistency","githubRepoAddedBy":"user","githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"680b7786134bf458c86b590e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/680b7786134bf458c86b590e/RhtoUvkknOkI8bhBfdoO5.png","isPro":false,"fullname":"Patrik Wolf","user":"patrikwolf","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"69bf5f36f8fd809097b8a6f8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/L4AAEAVMpbWY9TDYpjStZ.jpeg","isPro":false,"fullname":"Mateo Lee","user":"levmartinez","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"query":{}}">
Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models
Abstract
In-context learning is commonly interpreted as a form of conditional inference, in which the prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities. In particular, the law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population. In this work, we investigate to what extent LLM estimates adhere to this self-consistency principle. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine-grained subpopulations. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity. Applying this protocol across problem domains and state-of-the-art frontier models, we show widespread violations of basic consistency properties. An in-depth study of persona prompting reveals a pattern we call the macro fallacy: estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. This effect persists across variations in tree structure and estimation task, and can be partially recovered through implicit prompting. Together, these findings suggest that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates. This gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.
Community
We test whether LLM estimates obey the law of total probability, revealing widespread self-consistency failures and a macro fallacy in which aggregating fine-grained subpopulation estimates outperforms direct population-level prompting.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.15277 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.15277 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.15277 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.