Welcome discussion!</p>\n","updatedAt":"2026-06-01T02:46:26.727Z","author":{"_id":"67a4a26d5e65aa63c6d30e68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg","fullname":"Sicheng Feng","name":"FSCCS","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":13,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.361380398273468},"editors":["FSCCS"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg"],"reactions":[],"isReport":false}},{"id":"6a1cf8d7621490134dae38b9","author":{"_id":"67a4a26d5e65aa63c6d30e68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg","fullname":"Sicheng Feng","name":"FSCCS","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":13,"isUserFollowing":false},"createdAt":"2026-06-01T03:13:27.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"https://cdn-uploads.huggingface.co/production/uploads/67a4a26d5e65aa63c6d30e68/L3X1Dg_kLU3UahPckqYwP.mp4\n","html":"<p><video src=\"https://cdn-uploads.huggingface.co/production/uploads/67a4a26d5e65aa63c6d30e68/L3X1Dg_kLU3UahPckqYwP.mp4\" controls=\"\" class=\"max-w-full!\"></video></p>\n","updatedAt":"2026-06-01T03:13:27.204Z","author":{"_id":"67a4a26d5e65aa63c6d30e68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg","fullname":"Sicheng Feng","name":"FSCCS","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":13,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.2734924256801605},"editors":["FSCCS"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg"],"reactions":[{"reaction":"🚀","users":["FSCCS"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.30876","authors":[{"_id":"6a1cf241808ddbc3c7d43489","user":{"_id":"67a4a26d5e65aa63c6d30e68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg","isPro":false,"fullname":"Sicheng Feng","user":"FSCCS","type":"user","name":"FSCCS"},"name":"Sicheng Feng","status":"claimed_verified","statusLastChangedAt":"2026-06-01T09:33:21.635Z","hidden":false},{"_id":"6a1cf241808ddbc3c7d4348a","name":"Zigeng Chen","hidden":false},{"_id":"6a1cf241808ddbc3c7d4348b","name":"Gongfan Fang","hidden":false},{"_id":"6a1cf241808ddbc3c7d4348c","name":"Xinyin Ma","hidden":false},{"_id":"6a1cf241808ddbc3c7d4348d","name":"Xinchao Wang","hidden":false}],"publishedAt":"2026-05-29T00:00:00.000Z","submittedOnDailyAt":"2026-06-01T00:00:00.000Z","title":"dMoE: dLLMs with Learnable Block Experts","submittedOnDailyBy":{"_id":"67a4a26d5e65aa63c6d30e68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg","isPro":false,"fullname":"Sicheng Feng","user":"FSCCS","type":"user","name":"FSCCS"},"summary":"Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive models, offering competitive performance while naturally supporting parallel decoding. However, as dLLMs are increasingly integrated with Mixture-of-Experts (MoE) architectures to scale model capacity, a fundamental mismatch arises between block parallel decoding and token-level expert selection. Specifically, each dLLM forward pass processes multiple tokens with bidirectional dependencies, whereas conventional MoE layers route each token independently. This mismatch substantially increases the number of uniquely activated experts, making inference increasingly memory-bound. To address this, we propose dMoE, a simple yet effective block-level MoE framework. The central idea of dMoE is to aggregate token-level expert distributions within each block into a unified block-level expert distribution, which is then used to guide expert routing in a more coherent manner. In this way, dMoE substantially reduces the number of uniquely activated experts during inference without sacrificing performance, thereby mitigating the memory-bound bottleneck. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of dMoE. On average, dMoE reduces the number of uniquely activated experts from 69.5 to 14.6 while retaining 99.11% of the original performance. Meanwhile, it reduces memory usage by 76.64% to 79.84% and achieves 1.14times to 1.66times end-to-end latency speedup. Code is available at: https://github.com/fscdc/dMoE","upvotes":14,"discussionId":"6a1cf242808ddbc3c7d4348e","projectPage":"https://fscdc.github.io/dMoE/","githubRepo":"https://github.com/fscdc/dMoE","githubRepoAddedBy":"user","ai_summary":"Diffusion large language models combined with mixture-of-experts architectures face a mismatch between block parallel decoding and token-level expert selection, which dMoE addresses by aggregating token-level distributions into block-level routing to reduce activated experts and improve efficiency.","ai_keywords":["Diffusion Large Language Models","autoregressive models","parallel decoding","Mixture-of-Experts","block parallel decoding","token-level expert selection","expert routing","block-level expert distribution","memory-bound bottleneck","end-to-end latency"],"githubStars":16},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67a4a26d5e65aa63c6d30e68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg","isPro":false,"fullname":"Sicheng Feng","user":"FSCCS","type":"user"},{"_id":"634cfebc350bcee9bed20a4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png","isPro":false,"fullname":"Xingyi Yang","user":"adamdad","type":"user"},{"_id":"6860fe55a1ab4d5c885c3edf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/HNSwVGoTd33mgbyjxA-fc.jpeg","isPro":false,"fullname":"QIN ZHIBIN","user":"tuantuan0321","type":"user"},{"_id":"6a0c4875512e8cf10c427be1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a0c4875512e8cf10c427be1/eMm3o7aXACXkRf2XQ_oEy.jpeg","isPro":false,"fullname":"Siao Tang","user":"ttu1818","type":"user"},{"_id":"6627cccfded9b7936d5d1d21","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6627cccfded9b7936d5d1d21/LKGr7EP7AirjmkbFZSn4o.jpeg","isPro":true,"fullname":"Guangnian Wan","user":"bigglesworthnotcat","type":"user"},{"_id":"6694aa3f286045211d4b86dd","avatarUrl":"/avatars/b6fd2f95264f710039d1a7a497be6ed2.svg","isPro":false,"fullname":"graenys","user":"graenys","type":"user"},{"_id":"66def1e3ba8b9dac859dbd64","avatarUrl":"/avatars/84797ac61013046db3a495d5033f9d32.svg","isPro":false,"fullname":"Zhenxin Ai","user":"kunkk","type":"user"},{"_id":"66d739266776d114595a70f2","avatarUrl":"/avatars/69a648bbeedccf782f9c997e2666757e.svg","isPro":false,"fullname":"th","user":"CHEN1594","type":"user"},{"_id":"65811eeaa2284a018e51f1ba","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/dH8UZj6Kk5HJkI1DItCNm.jpeg","isPro":false,"fullname":"Zigeng Chen","user":"Zigeng","type":"user"},{"_id":"6624f53748e016b5ea587d40","avatarUrl":"/avatars/f8c16f45de0c3e32437f6e960a5b0959.svg","isPro":false,"fullname":"Shihua Zhang","user":"SuhZhang","type":"user"},{"_id":"6698a027615c05521cccaf5c","avatarUrl":"/avatars/efdbba4bda5428f33314eadf191416b4.svg","isPro":false,"fullname":"Zhenyu Sun","user":"Austin-Sun","type":"user"},{"_id":"67d5848f179ad2756600eca3","avatarUrl":"/avatars/158168a753271b6e024e1fbdf52c9e73.svg","isPro":false,"fullname":"Junhan ZHU","user":"Alrightlone","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.30876.md"}">
dMoE: dLLMs with Learnable Block Experts
Abstract
Diffusion large language models combined with mixture-of-experts architectures face a mismatch between block parallel decoding and token-level expert selection, which dMoE addresses by aggregating token-level distributions into block-level routing to reduce activated experts and improve efficiency.
AI-generated summary
Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive models, offering competitive performance while naturally supporting parallel decoding. However, as dLLMs are increasingly integrated with Mixture-of-Experts (MoE) architectures to scale model capacity, a fundamental mismatch arises between block parallel decoding and token-level expert selection. Specifically, each dLLM forward pass processes multiple tokens with bidirectional dependencies, whereas conventional MoE layers route each token independently. This mismatch substantially increases the number of uniquely activated experts, making inference increasingly memory-bound. To address this, we propose dMoE, a simple yet effective block-level MoE framework. The central idea of dMoE is to aggregate token-level expert distributions within each block into a unified block-level expert distribution, which is then used to guide expert routing in a more coherent manner. In this way, dMoE substantially reduces the number of uniquely activated experts during inference without sacrificing performance, thereby mitigating the memory-bound bottleneck. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of dMoE. On average, dMoE reduces the number of uniquely activated experts from 69.5 to 14.6 while retaining 99.11% of the original performance. Meanwhile, it reduces memory usage by 76.64% to 79.84% and achieves 1.14times to 1.66times end-to-end latency speedup. Code is available at: https://github.com/fscdc/dMoE
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.30876 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.30876 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.30876 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.