As large language models and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework. We design SetwiseEvalKit, a three-level, nine-dimension document set evaluation benchmark covering both short-form and long-form scenarios, comprising approximately 28K high-quality evaluation rubrics. We systematically evaluate 12 rerankers: even the best method achieves no more than 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings. Building on this, we propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.</p>\n","updatedAt":"2026-07-23T02:51:15.930Z","author":{"_id":"65745569839aa08899ea5d27","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg","fullname":"Kailin Jiang","name":"kailinjiang","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":5,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8912325501441956},"editors":["kailinjiang"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg"],"reactions":[],"isReport":false}},{"id":"6a61856f3985769753030657","author":{"_id":"658c0b0574e79b9a8e9de89a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/658c0b0574e79b9a8e9de89a/8HeMEOT5cLEzauXGlrqF_.jpeg","fullname":"Xinping Zhao","name":"Yuki131","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":10,"isUserFollowing":false},"createdAt":"2026-07-23T03:07:27.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Why aren't pointwise rerankers evaluated?","html":"<p>Why aren't pointwise rerankers evaluated?</p>\n","updatedAt":"2026-07-23T03:07:27.294Z","author":{"_id":"658c0b0574e79b9a8e9de89a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/658c0b0574e79b9a8e9de89a/8HeMEOT5cLEzauXGlrqF_.jpeg","fullname":"Xinping Zhao","name":"Yuki131","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":10,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8615462779998779},"editors":["Yuki131"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/658c0b0574e79b9a8e9de89a/8HeMEOT5cLEzauXGlrqF_.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.19747","authors":[{"_id":"6a618176c3792c34f5a040d0","name":"Kailin Jiang","hidden":false},{"_id":"6a618176c3792c34f5a040d1","name":"Lei Liu","hidden":false},{"_id":"6a618176c3792c34f5a040d2","name":"Jian Xi","hidden":false},{"_id":"6a618176c3792c34f5a040d3","name":"Hui Xu","hidden":false},{"_id":"6a618176c3792c34f5a040d4","name":"Junlin Liu","hidden":false},{"_id":"6a618176c3792c34f5a040d5","name":"Baochen Fu","hidden":false},{"_id":"6a618176c3792c34f5a040d6","name":"Shaoqing Ren","hidden":false},{"_id":"6a618176c3792c34f5a040d7","name":"Bin Li","hidden":false},{"_id":"6a618176c3792c34f5a040d8","name":"Vichwang","hidden":false},{"_id":"6a618176c3792c34f5a040d9","name":"Yu Lu","hidden":false},{"_id":"6a618176c3792c34f5a040da","name":"Haibo Shi","hidden":false}],"publishedAt":"2026-07-22T00:00:00.000Z","submittedOnDailyAt":"2026-07-23T00:00:00.000Z","title":"Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking","submittedOnDailyBy":{"_id":"65745569839aa08899ea5d27","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg","isPro":false,"fullname":"Kailin Jiang","user":"kailinjiang","type":"user","name":"kailinjiang"},"summary":"As large language models and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework. We design SetwiseEvalKit, a three-level, nine-dimension document set evaluation benchmark covering both short-form and long-form scenarios, comprising approximately 28K high-quality evaluation rubrics. We systematically evaluate 12 rerankers: even the best method achieves no more than 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings. Building on this, we propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.","upvotes":4,"discussionId":"6a618176c3792c34f5a040db","projectPage":"https://rubric4setwise.github.io/","githubRepo":"https://github.com/Rubric4Setwise/Rubric4Setwise","githubRepoAddedBy":"user","githubStars":1},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"65745569839aa08899ea5d27","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg","isPro":false,"fullname":"Kailin Jiang","user":"kailinjiang","type":"user"},{"_id":"658c0b0574e79b9a8e9de89a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/658c0b0574e79b9a8e9de89a/8HeMEOT5cLEzauXGlrqF_.jpeg","isPro":false,"fullname":"Xinping Zhao","user":"Yuki131","type":"user"},{"_id":"65388514613fe158bd514e4c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65388514613fe158bd514e4c/hpCY_g2oxLd1Ruq8LTA29.jpeg","isPro":false,"fullname":"alterego238","user":"alterego238","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.19747.md","query":{}}">
Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking
Abstract
As large language models and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework. We design SetwiseEvalKit, a three-level, nine-dimension document set evaluation benchmark covering both short-form and long-form scenarios, comprising approximately 28K high-quality evaluation rubrics. We systematically evaluate 12 rerankers: even the best method achieves no more than 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings. Building on this, we propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.
Community
As large language models and AI agents become the primary consumers of search results, document set quality determines the upper bound of downstream generation. Yet existing evaluation systems remain confined to scoring documents independently and aggregating via nDCG, ignoring inter-document interactions (redundancy, conflict, complementarity) and unable to answer what makes one document set better than another. To address these issues, we propose a complete evaluate-diagnose-optimize framework. We design SetwiseEvalKit, a three-level, nine-dimension document set evaluation benchmark covering both short-form and long-form scenarios, comprising approximately 28K high-quality evaluation rubrics. We systematically evaluate 12 rerankers: even the best method achieves no more than 45% coverage, cross-document coordination dimensions are universally weak, and no single method maintains top performance across both settings. Building on this, we propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals, achieving the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.
Why aren't pointwise rerankers evaluated?
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.19747 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.19747 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.19747 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.