Decomposing LLM-Judge Uncertainty to Target Expert Labels
Mirrored from arXiv — NLP / Computation & Language for archival readability. Support the source by reading on the original site.
Computer Science > Computation and Language
Title:Decomposing LLM-Judge Uncertainty to Target Expert Labels
Abstract:An LLM judge evaluates outputs at scale. Experts should label only where it is least sure. Its natural escalation signal conflates two uncertainties: aleatoric, real disagreement in the expert pool, which labels cannot reduce, and epistemic, the judge's ignorance, which labels do reduce. A small Bayesian model separates them: a regression on labels already collected learns how far to trust a black-box judge's prediction. Both components follow as simple formulas, with no sampling or further judge calls. The components isolate on a real LLM judge against exactly known truth, and stated confidence is no guide to its actual error. On real human disagreement (ChaosNLI) the epistemic ranking removes 83% more error than total uncertainty for the same expert labels, though simply escalating the least-labelled items does as well there. We demonstrate we can estimate where a judge is ignorant rather than where experts genuinely disagree, and propose using this to direct expert labelling. Code and data are available at this https URL.
| Comments: | 9 pages (4 pages of content plus references and appendices), 3 figures |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.06444 [cs.CL] |
| (or arXiv:2609.06444v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.06444
arXiv-issued DOI via DataCite
|
Submission history
From: Ryan Lail [view email][v1] Sun, 6 Sep 2026 07:36:15 UTC (277 KB)
[v2] Wed, 9 Sep 2026 11:31:37 UTC (278 KB)
[v3] Fri, 11 Sep 2026 15:04:08 UTC (278 KB)
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
References & Citations
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
More from arXiv — NLP / Computation & Language
-
A Mechanistic Study of AI-Text Detection Neurons in Frozen BERT: Sparse Probing and Activation Patching on RAID
Sep 28
-
Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling
Sep 28
-
Not All Memories Are Equal: Hierarchical Collaborative Memory for Validity-Aware Retrieval in LLM Agents
Sep 28
-
Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline
Sep 28
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.