We introduce <strong>coherent overlap</strong> as a way to understand sparse MoE routing: co-selected experts can occupy substantially overlapping representation subspaces while still providing useful multi-expert computation. Across six MoE architectures, we find that actual routes are more token-relevant than matched alternatives, and that geometric similarity alone does not imply functional redundancy or pruning value.</p>\n<p>We welcome feedback and discussion, especially on the implications for MoE routing, expert redundancy, and pruning.</p>\n","updatedAt":"2026-07-31T13:18:24.116Z","author":{"_id":"6739e34e04ba77f721b95b71","avatarUrl":"/avatars/18cc1c967df708cc8c9681932eb6837f.svg","fullname":"Huiyuan Tian","name":"Huiyuancs","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8847088813781738},"editors":["Huiyuancs"],"editorAvatarUrls":["/avatars/18cc1c967df708cc8c9681932eb6837f.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.28308","authors":[{"_id":"6a6c90ba26c3ed41815c87a1","user":{"_id":"6739e34e04ba77f721b95b71","avatarUrl":"/avatars/18cc1c967df708cc8c9681932eb6837f.svg","isPro":false,"fullname":"Huiyuan Tian","user":"Huiyuancs","type":"user","name":"Huiyuancs"},"name":"Huiyuan Tian","status":"claimed_verified","statusLastChangedAt":"2026-07-31T16:45:27.482Z","hidden":false},{"_id":"6a6c90ba26c3ed41815c87a2","name":"Bonan Xu","hidden":false},{"_id":"6a6c90ba26c3ed41815c87a3","name":"Shijian Li","hidden":false}],"publishedAt":"2026-07-30T00:00:00.000Z","submittedOnDailyAt":"2026-07-31T00:00:00.000Z","title":"Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing","submittedOnDailyBy":{"_id":"6739e34e04ba77f721b95b71","avatarUrl":"/avatars/18cc1c967df708cc8c9681932eb6837f.svg","isPro":false,"fullname":"Huiyuan Tian","user":"Huiyuancs","type":"user","name":"Huiyuancs"},"summary":"Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often conflates route coherence, candidate quality, and candidate-by-context interaction. We distinguish these quantities using an Expert Subspace Separation Index (ESSI), matched-route residuals, and a prefix-controlled 2times2 factorial; frozen-route interventions and a controlled Top-k study assess functional value. Three paired contrasts organize the findings. First, across six MoE architectures, expert subspaces overlap substantially, yet actual routes explain token representations better than matched alternatives. Second, across the 39 factorial cells in OLMoE, Mixtral, and DeepSeek, the selected candidate explains more of the residual representation than the strongest unselected rival in every cell, yet the actual prefix narrows this advantage throughout: all interactions are negative, and every 95% confidence interval lies below zero. Third, this geometric narrowing does not imply functional redundancy: adding later experts improves next-token prediction in 24 of 39 frozen-route comparisons, while the other 15 estimates are inconclusive; a controlled training study also favors Top-2 over Top-1 in all three seeds. We call this joint pattern coherent overlap: routing selects token-relevant experts from a shared geometric neighborhood, while useful multi-expert computation persists without disjoint linear coverage. Separating these quantities clarifies why geometric similarity alone cannot determine redundancy or pruning value.","upvotes":1,"discussionId":"6a6c90ba26c3ed41815c87a4"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.28308.md","query":{}}">
Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
Abstract
Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often conflates route coherence, candidate quality, and candidate-by-context interaction. We distinguish these quantities using an Expert Subspace Separation Index (ESSI), matched-route residuals, and a prefix-controlled 2times2 factorial; frozen-route interventions and a controlled Top-k study assess functional value. Three paired contrasts organize the findings. First, across six MoE architectures, expert subspaces overlap substantially, yet actual routes explain token representations better than matched alternatives. Second, across the 39 factorial cells in OLMoE, Mixtral, and DeepSeek, the selected candidate explains more of the residual representation than the strongest unselected rival in every cell, yet the actual prefix narrows this advantage throughout: all interactions are negative, and every 95% confidence interval lies below zero. Third, this geometric narrowing does not imply functional redundancy: adding later experts improves next-token prediction in 24 of 39 frozen-route comparisons, while the other 15 estimates are inconclusive; a controlled training study also favors Top-2 over Top-1 in all three seeds. We call this joint pattern coherent overlap: routing selects token-relevant experts from a shared geometric neighborhood, while useful multi-expert computation persists without disjoint linear coverage. Separating these quantities clarifies why geometric similarity alone cannot determine redundancy or pruning value.
Community
We introduce coherent overlap as a way to understand sparse MoE routing: co-selected experts can occupy substantially overlapping representation subspaces while still providing useful multi-expert computation. Across six MoE architectures, we find that actual routes are more token-relevant than matched alternatives, and that geometric similarity alone does not imply functional redundancy or pruning value.
We welcome feedback and discussion, especially on the implications for MoE routing, expert redundancy, and pruning.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.28308 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.28308 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.28308 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.