A new perspective looking at rubric-based evaluation for open-ended generation tasks.</p>\n","updatedAt":"2026-07-22T04:13:44.951Z","author":{"_id":"62ffccd18813dd8628172c6c","avatarUrl":"/avatars/62741be5967a74758a2b7914aa842ec8.svg","fullname":"Xilun Chen","name":"ccsasuke","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":11,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9050906896591187},"editors":["ccsasuke"],"editorAvatarUrls":["/avatars/62741be5967a74758a2b7914aa842ec8.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.19322","authors":[{"_id":"6a6043027e7f152167e471e6","user":{"_id":"62ffccd18813dd8628172c6c","avatarUrl":"/avatars/62741be5967a74758a2b7914aa842ec8.svg","isPro":false,"fullname":"Xilun Chen","user":"ccsasuke","type":"user","name":"ccsasuke"},"name":"Xilun Chen","status":"claimed_verified","statusLastChangedAt":"2026-07-22T07:39:29.364Z","hidden":false},{"_id":"6a6043027e7f152167e471e7","name":"Zhaleh Feizollahi","hidden":false},{"_id":"6a6043027e7f152167e471e8","name":"Ross Goodwin","hidden":false},{"_id":"6a6043027e7f152167e471e9","name":"Seungwhan Moon","hidden":false},{"_id":"6a6043027e7f152167e471ea","name":"Scott Yih","hidden":false},{"_id":"6a6043027e7f152167e471eb","name":"Pinar Donmez","hidden":false},{"_id":"6a6043027e7f152167e471ec","name":"Babak Damavandi","hidden":false},{"_id":"6a6043027e7f152167e471ed","name":"Luna Dong","hidden":false}],"publishedAt":"2026-07-21T00:00:00.000Z","submittedOnDailyAt":"2026-07-22T00:00:00.000Z","title":"Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness","submittedOnDailyBy":{"_id":"62ffccd18813dd8628172c6c","avatarUrl":"/avatars/62741be5967a74758a2b7914aa842ec8.svg","isPro":false,"fullname":"Xilun Chen","user":"ccsasuke","type":"user","name":"ccsasuke"},"summary":"Evaluating the factuality of long-form generations has focused predominantly on precision, measuring whether the claims a model makes are correct. The dominant decompose-search-verify pipeline catches incorrect claims well but says little about whether a response contains all the information it should. Measuring factual completeness, the missing half of factuality, is harder: it requires enumerating the full set of facts a complete answer should contain, and these facts rarely form a flat list. They often involve open-ended sets where coverage is what matters, ordered processes, and relationships among facts that a list of independent boolean checks fails to capture. We introduce a two-level meta-rubric framework for evaluating open-ended generation, and instantiate it as Gamut (Grounded Assessment of Multimodal Factuality), a benchmark for factual completeness in long-form generation. The framework rests on a two-level rubric representation: a structured meta-rubric captures the organization and importance of the required content, which is then mechanically compiled into a flat checklist of binary, machine-gradable rubrics that an LLM judge scores reliably. We construct 1,813 questions grounded in real wearable imagery across 10 diverse domains, each paired with an evidence-backed rubric verified by expert human annotators. Because the framework is modality-agnostic, we also release a text-only variant. Evaluating 14 frontier and open-weight models, we find the benchmark genuinely challenging (best score 58.7% from Gemini 3.1 Pro), highly discriminative, and robust to the choice of judge.","upvotes":3,"discussionId":"6a6043027e7f152167e471ee","githubRepo":"https://github.com/facebookresearch/GAMUT","githubRepoAddedBy":"user","githubStars":0,"organization":{"_id":"5e63d8713071d5be688861b8","name":"facebook","fullname":"AI at Meta","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1592839207516-noauth.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"62ffccd18813dd8628172c6c","avatarUrl":"/avatars/62741be5967a74758a2b7914aa842ec8.svg","isPro":false,"fullname":"Xilun Chen","user":"ccsasuke","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"63c1699e40a26dd2db32400d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63c1699e40a26dd2db32400d/3N0-Zp8igv8-52mXAdiiq.jpeg","isPro":false,"fullname":"Chroma","user":"Chroma111","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"5e63d8713071d5be688861b8","name":"facebook","fullname":"AI at Meta","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1592839207516-noauth.png"},"query":{}}">

Cite arxiv.org/abs/2607.19322 in a model README.md to link it from this page.

Cite arxiv.org/abs/2607.19322 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.