Hugging Face Daily Papers · July 31, 2026 · 3 min read

BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

hi! Thanks for supporting!\n","updatedAt":"2026-07-31T03:05:30.357Z","author":{"_id":"68c8ccff2de8edcf8da2fe51","avatarUrl":"/avatars/e55e509d11049691fefe3750a4bed62a.svg","fullname":"wangpengyu","name":"Bstwpy","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9446834921836853},"editors":["Bstwpy"],"editorAvatarUrls":["/avatars/e55e509d11049691fefe3750a4bed62a.svg"],"reactions":[],"isReport":false}},{"id":"6a6c196b81f56c6234104cbc","author":{"_id":"68022e89458239daab4db6bf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/8AsIlNYMIl-GjaBw80Sp0.png","fullname":"Tim","name":"Timgogogo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false},"createdAt":"2026-07-31T03:41:31.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Good!","html":"Good!\n","updatedAt":"2026-07-31T03:41:31.117Z","author":{"_id":"68022e89458239daab4db6bf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/8AsIlNYMIl-GjaBw80Sp0.png","fullname":"Tim","name":"Timgogogo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8619790077209473},"editors":["Timgogogo"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/8AsIlNYMIl-GjaBw80Sp0.png"],"reactions":[],"isReport":false}},{"id":"6a6c4de3d902dc6b00bd43c3","author":{"_id":"66d6819bc8c857729c54ec0d","avatarUrl":"/avatars/c46545865abdf7d73abbd10f1cf516c1.svg","fullname":"Runzhe Teng","name":"Runge","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false},"createdAt":"2026-07-31T07:25:23.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Amazing work！","html":"Amazing work！\n","updatedAt":"2026-07-31T07:25:23.907Z","author":{"_id":"66d6819bc8c857729c54ec0d","avatarUrl":"/avatars/c46545865abdf7d73abbd10f1cf516c1.svg","fullname":"Runzhe Teng","name":"Runge","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.41946351528167725},"editors":["Runge"],"editorAvatarUrls":["/avatars/c46545865abdf7d73abbd10f1cf516c1.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.26497","authors":[{"_id":"6a6c1039202e2d9e3ffdb722","name":"Pengyu Wang","hidden":false},{"_id":"6a6c1039202e2d9e3ffdb723","name":"Benfeng Xu","hidden":false},{"_id":"6a6c1039202e2d9e3ffdb724","name":"Shaohan Wang","hidden":false},{"_id":"6a6c1039202e2d9e3ffdb725","name":"Xin Zeng","hidden":false},{"_id":"6a6c1039202e2d9e3ffdb726","name":"Huarui Wu","hidden":false},{"_id":"6a6c1039202e2d9e3ffdb727","name":"Lei Zhang","hidden":false},{"_id":"6a6c1039202e2d9e3ffdb728","name":"Licheng Zhang","hidden":false}],"publishedAt":"2026-07-30T00:00:00.000Z","submittedOnDailyAt":"2026-07-31T00:00:00.000Z","title":"BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms","submittedOnDailyBy":{"_id":"68c8ccff2de8edcf8da2fe51","avatarUrl":"/avatars/e55e509d11049691fefe3750a4bed62a.svg","isPro":false,"fullname":"wangpengyu","user":"Bstwpy","type":"user","name":"Bstwpy"},"summary":"Retrieval-augmented generation (RAG) spans lexical and dense retrieval, graph-based indexing, and agentic search, but these paradigms are usually evaluated on different benchmarks at one corpus size, leaving their accuracy-cost scaling unclear. To bridge this gap, we present a controlled study that varies corpus size along 28 strictly nested tiers spanning roughly 450-fold, while holding questions and a fixed bedrock of relevant and adversarial documents unchanged. Under one reader model and one judging protocol, we measure official accuracy, construction and query tokens, and latency. The results reveal a scale-dependent crossover rather than an unconditional winner. File-System Agent leads at the smallest shared tiers, but its sequential exploration costs 39 times more query tokens at the bedrock and becomes less effective as the search space grows. Around 10 million corpus tokens, BM25 overtakes it and leads at every larger shared tier, with a margin approaching 20 points at full scale. BM25 also anchors the low-cost end of the Pareto frontier without LLM-based construction. Dense retrieval remains efficient but less accurate, whereas graph-based RAG encounters construction walls before deployment scale and its scalable variants remain below BM25 at shared tiers. Overall, corpus growth increasingly favors global candidate ranking: lexical retrieval is the strongest scalable default, while agentic reasoning works best after ranked discovery rather than in place of it.","upvotes":36,"discussionId":"6a6c103a202e2d9e3ffdb729","organization":{"_id":"6912993f45f02a20f4c50b8a","name":"muset-ai","fullname":"muset.ai","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/C5j0HEiqEmmxshM61WRON.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"68c8ccff2de8edcf8da2fe51","avatarUrl":"/avatars/e55e509d11049691fefe3750a4bed62a.svg","isPro":false,"fullname":"wangpengyu","user":"Bstwpy","type":"user"},{"_id":"6440d49b7663594a126716f2","avatarUrl":"/avatars/bb04af9d1ae9c5bc058ebfbf08f4ebc8.svg","isPro":false,"fullname":"shaohanwang","user":"WShao","type":"user"},{"_id":"646dbba74ad7f907279dd486","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/XGHJMFEIpWeDlh0cn9Slu.png","isPro":false,"fullname":"Mingxuan Du","user":"Ayanami0730","type":"user"},{"_id":"68cbf13fd9c1b531d9564757","avatarUrl":"/avatars/e1e26dc251102c5c18ce5fa172e6a8fa.svg","isPro":false,"fullname":"Zhou","user":"Tuowen","type":"user"},{"_id":"62c926c71eba56b9f6212922","avatarUrl":"/avatars/e78d2bceae9b9f2e5295408f7094c4c8.svg","isPro":false,"fullname":"Hal","user":"farawayxxx","type":"user"},{"_id":"6a0fc58cc8676ad292e8a15d","avatarUrl":"/avatars/2649c5c1d64cd72b77265747b89e0cea.svg","isPro":false,"fullname":"Ruizhe Li","user":"imlrz01","type":"user"},{"_id":"68959bf9950f311270f0e010","avatarUrl":"/avatars/96f5ac05139954579a97a80e0f29b528.svg","isPro":false,"fullname":"Ruizhe Li","user":"imlrz","type":"user"},{"_id":"62d56f063bf5e059f7cac515","avatarUrl":"/avatars/6bd5cfdc21506ace6176d00a2973d8e5.svg","isPro":false,"fullname":"BenfengXu","user":"SpiketheCowboy","type":"user"},{"_id":"64d0e33da2e7f9ff6119dafc","avatarUrl":"/avatars/5bbee68da9f9bc2686f15f686924762b.svg","isPro":false,"fullname":"Zeng","user":"Xin1818","type":"user"},{"_id":"6a6824d5a2bbdde86f7cad76","avatarUrl":"/avatars/6ff679759fc9be5956f1b12799a3b874.svg","isPro":false,"fullname":"Yuhang Zhu","user":"Yuki-Kitayama","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6a2a342b086133dad7d61c92","avatarUrl":"/avatars/db700b4fc2f762f072e4a22b2f48b757.svg","isPro":false,"fullname":"annlin","user":"annl092723","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6912993f45f02a20f4c50b8a","name":"muset-ai","fullname":"muset.ai","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/C5j0HEiqEmmxshM61WRON.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.26497.md","query":{}}">

Papers

arxiv:2607.26497

BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

Published on Jul 30

· Submitted by

wangpengyu on Jul 31

muset.ai

Upvote

Authors:

Abstract

Retrieval-augmented generation (RAG) spans lexical and dense retrieval, graph-based indexing, and agentic search, but these paradigms are usually evaluated on different benchmarks at one corpus size, leaving their accuracy-cost scaling unclear. To bridge this gap, we present a controlled study that varies corpus size along 28 strictly nested tiers spanning roughly 450-fold, while holding questions and a fixed bedrock of relevant and adversarial documents unchanged. Under one reader model and one judging protocol, we measure official accuracy, construction and query tokens, and latency. The results reveal a scale-dependent crossover rather than an unconditional winner. File-System Agent leads at the smallest shared tiers, but its sequential exploration costs 39 times more query tokens at the bedrock and becomes less effective as the search space grows. Around 10 million corpus tokens, BM25 overtakes it and leads at every larger shared tier, with a margin approaching 20 points at full scale. BM25 also anchors the low-cost end of the Pareto frontier without LLM-based construction. Dense retrieval remains efficient but less accurate, whereas graph-based RAG encounters construction walls before deployment scale and its scalable variants remain below BM25 at shared tiers. Overall, corpus growth increasingly favors global candidate ranking: lexical retrieval is the strongest scalable default, while agentic reasoning works best after ranked discovery rather than in place of it.

View arXiv page View PDF Add to collection