Combining LLMs Rarely Beats the Single Best Model: A Provable Co-Failure Ceiling Across 67 Frontier Models</p>\n","updatedAt":"2026-06-26T11:04:33.215Z","author":{"_id":"64442f46af034cdfd69d5bc4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/S1CIatxL_H5QWDA6fgOop.png","fullname":"Josef Chen","name":"josefchen","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5974400639533997},"editors":["josefchen"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/S1CIatxL_H5QWDA6fgOop.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.27288","authors":[{"_id":"6a3e45b63b43e283349ec6fa","user":{"_id":"64442f46af034cdfd69d5bc4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/S1CIatxL_H5QWDA6fgOop.png","isPro":true,"fullname":"Josef Chen","user":"josefchen","type":"user","name":"josefchen"},"name":"Josef Chen","status":"claimed_verified","statusLastChangedAt":"2026-06-27T15:23:05.366Z","hidden":false}],"publishedAt":"2026-06-25T00:00:00.000Z","submittedOnDailyAt":"2026-06-26T00:00:00.000Z","title":"When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models","submittedOnDailyBy":{"_id":"64442f46af034cdfd69d5bc4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/S1CIatxL_H5QWDA6fgOop.png","isPro":true,"fullname":"Josef Chen","user":"josefchen","type":"user","name":"josefchen"},"summary":"Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router.\n Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.","upvotes":3,"discussionId":"6a3e45b63b43e283349ec6fb","projectPage":"https://huggingface.co/spaces/josefchen/orchestration-is-allocation","ai_summary":"Multi-model systems face fundamental accuracy limits determined by the rate at which all models fail simultaneously, regardless of their individual correlations or ensemble strategies.","ai_keywords":["multi-model LLM systems","routing","voting","cascades","fusion","mixture-of-agents","accuracy","beta","error correlation","Clopper-Pearson bound","tetrachoric-calibrated model","Gaussian copula","GPQA-Diamond","Self-MoA","heterogeneous ensembles"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"64442f5f79e7797ce71c3942","name":"Kaikaku","fullname":"Kaikaku","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/64442f46af034cdfd69d5bc4/SRJTr1BaXthGq1fKLBPm3.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64442f46af034cdfd69d5bc4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/S1CIatxL_H5QWDA6fgOop.png","isPro":true,"fullname":"Josef Chen","user":"josefchen","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"},{"_id":"696da0962b3e2d9587d0b35d","avatarUrl":"/avatars/4f6c177ad51fb687ca1be75d18f6f5d6.svg","isPro":false,"fullname":"mini","user":"mini0999","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"64442f5f79e7797ce71c3942","name":"Kaikaku","fullname":"Kaikaku","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/64442f46af034cdfd69d5bc4/SRJTr1BaXthGq1fKLBPm3.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.27288.md","query":{}}">
When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models
Abstract
Multi-model systems face fundamental accuracy limits determined by the rate at which all models fail simultaneously, regardless of their individual correlations or ensemble strategies.
Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router.
Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.
Community
Combining LLMs Rarely Beats the Single Best Model: A Provable Co-Failure Ceiling Across 67 Frontier Models
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.27288 in a model README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.