Artificial Societies Benchmark: A Validation Framework for Synthetic Research
Mirrored from arXiv — NLP / Computation & Language for archival readability. Support the source by reading on the original site.
Computer Science > Computation and Language
Title:Artificial Societies Benchmark: A Validation Framework for Synthetic Research
Abstract:A synthetic survey can reproduce the average answer while misrepresenting how people differ, how their answers relate to one another, or how they respond to changes in conditions. We introduce the Artificial Societies Benchmark to help researchers assess whether synthetic populations support their intended analyses. The framework combines eleven tests across internal, construct, and external validity, drawing on twenty human sources and comparing nine language models. It connects each research use to the evidence it requires and tests how results change with the information we supply about respondents. Importantly, strong performance in one domain does not establish fidelity in the others. Models often answer too consistently, compress response scales, and alter relationships between traits whilst richer profiles improve prediction for some models and worsen it for others. The resulting scorecard helps researchers identify which aspects of a synthetic population can support their analysis and where researchers need further human evidence.
| Comments: | 36 pages, 9 figures, 9 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.30030 [cs.CL] |
| (or arXiv:2609.30030v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.30030
arXiv-issued DOI via DataCite (pending registration)
|
Submission history
From: Edoardo Chidichimo [view email][v1] Thu, 24 Sep 2026 16:03:28 UTC (4,554 KB)
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
References & Citations
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
More from arXiv — NLP / Computation & Language
-
A Mechanistic Study of AI-Text Detection Neurons in Frozen BERT: Sparse Probing and Activation Patching on RAID
Sep 28
-
Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling
Sep 28
-
Not All Memories Are Equal: Hierarchical Collaborative Memory for Validity-Aware Retrieval in LLM Agents
Sep 28
-
Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline
Sep 28
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.