Hugging Face Daily Papers · June 8, 2026 · 4 min read

UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

UnpredictaBench is a 448-instance benchmark across 40 distributions to test whether LLMs generate true randomness. Using repeated sampling and distributional comparison, we find strong biases and poor stochastic fidelity in current models.</p>\n","updatedAt":"2026-06-08T17:31:16.783Z","author":{"_id":"60e32baedc56466240084155","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60e32baedc56466240084155/Q-cJh3Q3-vvMbe749Gt5B.jpeg","fullname":"Amirhossein Abaskohi","name":"AmirhosseinAbaskohi","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7857680320739746},"editors":["AmirhosseinAbaskohi"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/60e32baedc56466240084155/Q-cJh3Q3-vvMbe749Gt5B.jpeg"],"reactions":[{"reaction":"👍","users":["AmirHossein1378"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.06622","authors":[{"_id":"6a26200fe4c258a029491f93","name":"Amirhossein Abaskohi","hidden":false},{"_id":"6a26200fe4c258a029491f94","name":"Amirhossein Dabiriaghdam","hidden":false},{"_id":"6a26200fe4c258a029491f95","name":"Liang Luo","hidden":false},{"_id":"6a26200fe4c258a029491f96","name":"Ellie Dingqiao Wen","hidden":false},{"_id":"6a26200fe4c258a029491f97","name":"Lele Wang","hidden":false},{"_id":"6a26200fe4c258a029491f98","name":"Giuseppe Carenini","hidden":false},{"_id":"6a26200fe4c258a029491f99","name":"Peter West","hidden":false}],"publishedAt":"2026-06-04T00:00:00.000Z","submittedOnDailyAt":"2026-06-08T00:00:00.000Z","title":"UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs","submittedOnDailyBy":{"_id":"60e32baedc56466240084155","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60e32baedc56466240084155/Q-cJh3Q3-vvMbe749Gt5B.jpeg","isPro":false,"fullname":"Amirhossein Abaskohi","user":"AmirhosseinAbaskohi","type":"user","name":"AmirhosseinAbaskohi"},"summary":"We introduce UnpredictaBench, an evaluation that tests the ability of large language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by stochastic programs, and natural-language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems.","upvotes":12,"discussionId":"6a26200fe4c258a029491f9a","projectPage":"https://huggingface.co/datasets/UnpredictaBench/UnpredictaBench","githubRepo":"https://github.com/UnpredictaBench/UnpredictaBenchCode","githubRepoAddedBy":"user","ai_summary":"UnpredictaBench evaluates large language models' capacity to sample from target distributions, revealing significant gaps in their ability to simulate unpredictable systems despite recent advances in output diversity.","ai_keywords":["large language models","distributional capabilities","Kolmogorov-Smirnov test","stochastic programs","statistical distributions","output diversity","distributional sampling","simulation"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0,"organization":{"_id":"6723d7c5f0e98abd466cd7df","name":"UBC-VisionLanguage","fullname":"Vision and NLP Group at University of British Columbia","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6712caf88a321497c3a15673/0mNkKMu8h74rP2Hzo1O5U.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"60e32baedc56466240084155","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60e32baedc56466240084155/Q-cJh3Q3-vvMbe749Gt5B.jpeg","isPro":false,"fullname":"Amirhossein Abaskohi","user":"AmirhosseinAbaskohi","type":"user"},{"_id":"69f90109191f57df815c6c2b","avatarUrl":"/avatars/e5df02573db877739e0aee8101451dcb.svg","isPro":false,"fullname":"UnpredictaBench","user":"UnpredictaBench","type":"user"},{"_id":"64510a21f800611f94f0d9f8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64510a21f800611f94f0d9f8/uB0uM3zxoiytcgAuJgfQR.jpeg","isPro":false,"fullname":"Yuwei Yin","user":"yuweiyin","type":"user"},{"_id":"64558d926f4ae996562a1046","avatarUrl":"/avatars/b8a15da280e0e61d9e93eebbcef8ce6e.svg","isPro":false,"fullname":"Benjamin Movassagh","user":"Benjam1n","type":"user"},{"_id":"5fa12627a13e063b8b2b5d6a","avatarUrl":"/avatars/5b23d18c25a420dcc949e97061c8a2a7.svg","isPro":false,"fullname":"Alireza Salemi","user":"alireza7","type":"user"},{"_id":"689a846b56a7319615a9088f","avatarUrl":"/avatars/b208a31c47756d3badb334cb8a5c5526.svg","isPro":false,"fullname":"Peter West","user":"peterawest","type":"user"},{"_id":"65a5ea0fd6b35af3100a71b7","avatarUrl":"/avatars/4db77bad4d84f0655cda68206d4b4197.svg","isPro":false,"fullname":"hcc c","user":"Hacel","type":"user"},{"_id":"6228ede94323cef93a956b24","avatarUrl":"/avatars/2f01099f102889f2a621a68dcd61b6b6.svg","isPro":false,"fullname":"AmirHossein Dabiri Aghdam","user":"AmirHossein1378","type":"user"},{"_id":"69abbab8ffd071299e705e74","avatarUrl":"/avatars/5e6de76afbd4424ba90b08ebacbe539c.svg","isPro":false,"fullname":"Alexandrine Fortier","user":"aforti1","type":"user"},{"_id":"68dd763644d8aa330f1dc991","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/68dd763644d8aa330f1dc991/Sv6s_jsXG46WeTbrWTnpR.png","isPro":false,"fullname":"Irmak Bayir","user":"irmakbayir","type":"user"},{"_id":"6a2718c4ee8c9826408a267d","avatarUrl":"/avatars/75b672abbc4fba71f784503e86753088.svg","isPro":false,"fullname":"SARA VAKILI","user":"saravakili123","type":"user"},{"_id":"6a271c4928e4d750aa0e88cd","avatarUrl":"/avatars/1fd882587298d5a30080b54d1b6c96e5.svg","isPro":false,"fullname":"Asami tanaka","user":"Asami105","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6723d7c5f0e98abd466cd7df","name":"UBC-VisionLanguage","fullname":"Vision and NLP Group at University of British Columbia","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6712caf88a321497c3a15673/0mNkKMu8h74rP2Hzo1O5U.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.06622.md"}">

Papers

arxiv:2606.06622

UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs

Published on Jun 4

· Submitted by

Amirhossein Abaskohi on Jun 8

Vision and NLP Group at University of British Columbia

Upvote

Authors:

Abstract

UnpredictaBench evaluates large language models' capacity to sample from target distributions, revealing significant gaps in their ability to simulate unpredictable systems despite recent advances in output diversity.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

We introduce UnpredictaBench, an evaluation that tests the ability of large language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by stochastic programs, and natural-language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems.