🚀 <strong>UniMoMo</strong>, a practical approach for compressing MoE-based recommendation models.</p>\n<p>The main idea is simple but effective: instead of merging experts based on parameter similarity, UniMoMo looks at <strong>how experts actually behave on calibration data</strong> and how much routing traffic each expert receives.</p>\n<p>A few things I really like:</p>\n<p>🔹 <strong>Functional expert similarity</strong> — experts are compared using their outputs on shared hidden states, rather than just weight distance.</p>\n<p>🔹 <strong>Traffic-aware merging</strong> — heavily used experts are protected during clustering, reducing the risk of hurting important routing paths.</p>\n<p>🔹 <strong>Calibration-aware reconstruction</strong> — after merging SwiGLU experts, UniMoMo uses a least-squares correction to better recover the original intermediate activations.</p>\n<p>🔹 <strong>No extra serving complexity</strong> — the final compressed model is still a standard top-(k) MoE, so there is no special inference-time module.</p>\n<p>The results are also strong: compressing <strong>8 experts → 4 experts</strong> can retain nearly all recommendation quality while achieving meaningful inference speedups, and the more aggressive <strong>8 → 2 / top-1</strong> setting pushes acceleration even further. ⚡</p>\n<p>Overall, a nice example of making MoE compression more <strong>function-aware, routing-aware, and deployment-friendly</strong>. Worth checking out if you work on large-scale recommendation or MoE efficiency. 👀</p>\n","updatedAt":"2026-08-12T06:54:29.439Z","author":{"_id":"652fc2605615e57807e3db19","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/652fc2605615e57807e3db19/kbRcpR0YFQnU3IlziqCHf.png","fullname":"Changjiang Jiang","name":"arnodjiang","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8893231153488159},"editors":["arnodjiang"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/652fc2605615e57807e3db19/kbRcpR0YFQnU3IlziqCHf.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.08627","authors":[{"_id":"6a7ad127019ce76dc7b3abab","name":"Lei Xin","hidden":false},{"_id":"6a7ad127019ce76dc7b3abac","name":"Bin Gu","hidden":false},{"_id":"6a7ad127019ce76dc7b3abad","name":"Peize Li","hidden":false},{"_id":"6a7ad127019ce76dc7b3abae","name":"Zitong Wang","hidden":false},{"_id":"6a7ad127019ce76dc7b3abaf","name":"Jianbo Zhao","hidden":false},{"_id":"6a7ad127019ce76dc7b3abb0","user":{"_id":"652fc2605615e57807e3db19","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/652fc2605615e57807e3db19/kbRcpR0YFQnU3IlziqCHf.png","isPro":false,"fullname":"Changjiang Jiang","user":"arnodjiang","type":"user","name":"arnodjiang"},"name":"Changjiang Jiang","status":"claimed_verified","statusLastChangedAt":"2026-08-11T08:45:04.560Z","hidden":false},{"_id":"6a7ad127019ce76dc7b3abb1","name":"Yanyue Xie","hidden":false},{"_id":"6a7ad127019ce76dc7b3abb2","name":"Chao Huang","hidden":false},{"_id":"6a7ad127019ce76dc7b3abb3","name":"Xuyang Zhao","hidden":false},{"_id":"6a7ad127019ce76dc7b3abb4","user":{"_id":"655b4f5cc11dee7f7e882a0c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655b4f5cc11dee7f7e882a0c/4GaAC5Qt55eYlbNRJpVD5.png","isPro":false,"fullname":"zunhaisu","user":"zunhai","type":"user","name":"zunhai"},"name":"Zunhai Su","status":"claimed_verified","statusLastChangedAt":"2026-08-12T16:45:05.152Z","hidden":false},{"_id":"6a7ad127019ce76dc7b3abb5","name":"Fanhu Zeng","hidden":false},{"_id":"6a7ad127019ce76dc7b3abb6","name":"Zhenglun Kong","hidden":false}],"publishedAt":"2026-08-09T00:00:00.000Z","submittedOnDailyAt":"2026-08-12T00:00:00.000Z","title":"UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models","submittedOnDailyBy":{"_id":"652fc2605615e57807e3db19","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/652fc2605615e57807e3db19/kbRcpR0YFQnU3IlziqCHf.png","isPro":false,"fullname":"Changjiang Jiang","user":"arnodjiang","type":"user","name":"arnodjiang"},"summary":"Sparse mixture-of-experts (MoE) layers expand recommendation capacity through conditional computation, yet a trained checkpoint still stores and routes over its full expert bank. We study a deployment problem: convert that checkpoint to a smaller standard MoE under an explicit expert budget, without adding a compression-specific online module. To address this, we introduce UniMoMo, a post-training compression framework formulated as a constrained graph coarsening problem. Rather than relying on parameter distance, UniMoMo groups experts based on their functional similarity, using an unlabeled calibration set to measure how similarly experts respond to shared recommendation states. To prevent performance degradation, we introduce a layer-adaptive protection mechanism that restricts the merging of high-traffic experts based on their routing exposure. Across Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks, the final four-expert checkpoints obtain source-relative five-run mean NDCG@10 ratios of 99.92%--102.30% and measured A100 speedups of 1.28times--1.63times. An aggressive two-expert, top-1 operating point obtains ratios of 98.36%--104.24% and speedups of 1.47times--2.21times. These endpoint results evaluate the complete conversion-and-adaptation workflow and show that a trained recommendation MoE can be exported at multiple serving budgets.","upvotes":5,"discussionId":"6a7ad127019ce76dc7b3abb7","ai_summary":"UniMoMo compresses trained recommendation mixture-of-experts models into smaller standard MoE checkpoints via functional similarity grouping and layer-adaptive protection, preserving accuracy while accelerating inference.","ai_keywords":["sparse mixture-of-experts","MoE","conditional computation","graph coarsening","functional similarity","calibration set","layer-adaptive protection","routing exposure","NDCG@10"],"ai_summary_model":"thinkingmachines/Inkling-Small"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"652fc2605615e57807e3db19","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/652fc2605615e57807e3db19/kbRcpR0YFQnU3IlziqCHf.png","isPro":false,"fullname":"Changjiang Jiang","user":"arnodjiang","type":"user"},{"_id":"69bcc0f28e79809e9aa8eb9f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GDGAj78M6I055xLUC7w3j.png","isPro":false,"fullname":"Yifan Huang","user":"loganramirez70","type":"user"},{"_id":"69bcc905518f6d1f3d2b86df","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t-ZEqYEkqPI8-saats0nT.jpeg","isPro":false,"fullname":"Liu Jingyi","user":"gao-wenxuan2","type":"user"},{"_id":"655b4f5cc11dee7f7e882a0c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655b4f5cc11dee7f7e882a0c/4GaAC5Qt55eYlbNRJpVD5.png","isPro":false,"fullname":"zunhaisu","user":"zunhai","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.08627.md","query":{}}">
UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models
Abstract
UniMoMo compresses trained recommendation mixture-of-experts models into smaller standard MoE checkpoints via functional similarity grouping and layer-adaptive protection, preserving accuracy while accelerating inference.
Sparse mixture-of-experts (MoE) layers expand recommendation capacity through conditional computation, yet a trained checkpoint still stores and routes over its full expert bank. We study a deployment problem: convert that checkpoint to a smaller standard MoE under an explicit expert budget, without adding a compression-specific online module. To address this, we introduce UniMoMo, a post-training compression framework formulated as a constrained graph coarsening problem. Rather than relying on parameter distance, UniMoMo groups experts based on their functional similarity, using an unlabeled calibration set to measure how similarly experts respond to shared recommendation states. To prevent performance degradation, we introduce a layer-adaptive protection mechanism that restricts the merging of high-traffic experts based on their routing exposure. Across Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks, the final four-expert checkpoints obtain source-relative five-run mean NDCG@10 ratios of 99.92%--102.30% and measured A100 speedups of 1.28times--1.63times. An aggressive two-expert, top-1 operating point obtains ratios of 98.36%--104.24% and speedups of 1.47times--2.21times. These endpoint results evaluate the complete conversion-and-adaptation workflow and show that a trained recommendation MoE can be exported at multiple serving budgets.
Community
🚀 UniMoMo, a practical approach for compressing MoE-based recommendation models.
The main idea is simple but effective: instead of merging experts based on parameter similarity, UniMoMo looks at how experts actually behave on calibration data and how much routing traffic each expert receives.
A few things I really like:
🔹 Functional expert similarity — experts are compared using their outputs on shared hidden states, rather than just weight distance.
🔹 Traffic-aware merging — heavily used experts are protected during clustering, reducing the risk of hurting important routing paths.
🔹 Calibration-aware reconstruction — after merging SwiGLU experts, UniMoMo uses a least-squares correction to better recover the original intermediate activations.
🔹 No extra serving complexity — the final compressed model is still a standard top-(k) MoE, so there is no special inference-time module.
The results are also strong: compressing 8 experts → 4 experts can retain nearly all recommendation quality while achieving meaningful inference speedups, and the more aggressive 8 → 2 / top-1 setting pushes acceleration even further. ⚡
Overall, a nice example of making MoE compression more function-aware, routing-aware, and deployment-friendly. Worth checking out if you work on large-scale recommendation or MoE efficiency. 👀
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.08627 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.08627 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.08627 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.