Hugging Face Daily Papers · August 24, 2026 · 4 min read

Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

Training large Mixture-of-Experts (MoE) models on trillions of tokens makes conventional hyperparameter sweeps prohibitively expensive. In this work, we introduce a compute-efficient two-step framework that combines μP-based learning-rate transfer across model scales with token-horizon extrapolation from short-budget proxy runs. Using the predicted learning rate, we successfully pretrained a 155B-total / 17B-active-parameter foundation model from scratch for 10T tokens.</p>\n","updatedAt":"2026-08-24T04:47:18.752Z","author":{"_id":"6605028007a154c768e1c4c7","avatarUrl":"/avatars/88678edb83fdb466067e38acd22d07de.svg","fullname":"Nayeon Kim","name":"lana-ny","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8420698642730713},"editors":["lana-ny"],"editorAvatarUrls":["/avatars/88678edb83fdb466067e38acd22d07de.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.20061","authors":[{"_id":"6a882e39404cbc83ad9f08a1","user":{"_id":"6605028007a154c768e1c4c7","avatarUrl":"/avatars/88678edb83fdb466067e38acd22d07de.svg","isPro":false,"fullname":"Nayeon Kim","user":"lana-ny","type":"user","name":"lana-ny"},"name":"Nayeon Kim","status":"claimed_verified","statusLastChangedAt":"2026-08-21T12:57:15.919Z","hidden":false},{"_id":"6a882e39404cbc83ad9f08a2","name":"Hojin Lee","hidden":false},{"_id":"6a882e39404cbc83ad9f08a3","name":"Yunju Bak","hidden":false},{"_id":"6a882e39404cbc83ad9f08a4","user":{"_id":"6136f65440e43b8f748a0833","avatarUrl":"/avatars/f72a5ae3d3e94485de8aed8df94abdad.svg","isPro":false,"fullname":"Jaesun Park","user":"jaesun","type":"user","name":"jaesun"},"name":"Jaesun Park","status":"claimed_verified","statusLastChangedAt":"2026-08-21T12:57:20.117Z","hidden":false},{"_id":"6a882e39404cbc83ad9f08a5","name":"Boseop Kim","hidden":false}],"publishedAt":"2026-08-20T13:57:43.000Z","submittedOnDailyAt":"2026-08-24T00:00:00.000Z","title":"Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts","submittedOnDailyBy":{"_id":"6605028007a154c768e1c4c7","avatarUrl":"/avatars/88678edb83fdb466067e38acd22d07de.svg","isPro":false,"fullname":"Nayeon Kim","user":"lana-ny","type":"user","name":"lana-ny"},"summary":"Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost. However, optimizing their hyperparameters---particularly the learning rate---at extreme scales of both model size and token budget via sweeping remains computationally prohibitive. In this paper, we propose a compute-efficient, two-step hyperparameter transfer framework that estimates optimal learning rates for training large MoE models by transferring them across scaling model widths, and subsequently extrapolating to trillion-token horizons. First, we formulate a Maximal Update Parameterization (μP) adaptation for MoE architectures utilizing Multi-head Latent Attention (MLA) and the Muon optimizer, demonstrating that optimal learning rates transfer consistently across width-scaled models. Second, we extend this transferability along the token dimension by establishing a predictive scaling law. By applying linear regression to the optimal values derived from small proxy models on limited budgets, we successfully extrapolate the ideal learning rate to massive training horizons (e.g., 10 trillion tokens) with high fidelity (R^2=0.95). Consequently, this indicates that proxy training on small models is sufficient to determine the optimal learning rate for the extensive training of large-scale MoEs. We apply the proposed methodology to pretrain our foundation model (155B total, 17B active parameters) from scratch, and the stable training and evaluation results validate that optimal configurations for full-scale target models can be accurately predicted with minimal ablation costs.","upvotes":21,"discussionId":"6a882e39404cbc83ad9f08a6","ai_summary":"A two-step hyperparameter transfer framework predicts optimal learning rates for large Mixture-of-Experts models by scaling across widths and token budgets, enabling efficient pretraining without costly sweeps.","ai_keywords":["Mixture-of-Experts","Maximal Update Parameterization","Multi-head Latent Attention","Muon optimizer","hyperparameter transfer","scaling law","learning rate extrapolation"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"66c2c83e37e5578c16d19500","name":"kakaocorp","fullname":"Kakao Corp.","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c2a902b83a7e94d517c1ea/6Bo5zjJnRCtIi28zFUPmO.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66c2ea881ea0a61c6cc0142e","avatarUrl":"/avatars/0f7f0bf1217be5be50198c33b3729db7.svg","isPro":false,"fullname":"wavy-jung","user":"wavy-jung","type":"user"},{"_id":"6136f65440e43b8f748a0833","avatarUrl":"/avatars/f72a5ae3d3e94485de8aed8df94abdad.svg","isPro":false,"fullname":"Jaesun Park","user":"jaesun","type":"user"},{"_id":"639f676f2e13e54dcbbdf6d5","avatarUrl":"/avatars/9c609d197f876a1a9703f9e7ab4d3af9.svg","isPro":false,"fullname":"Cheon Jaewon","user":"snob","type":"user"},{"_id":"66c8264759227bf53ddced74","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c8264759227bf53ddced74/tJXlxUAfl7kdCkJhcM513.jpeg","isPro":false,"fullname":"ryan-u","user":"ryan-u","type":"user"},{"_id":"640eff5af2d7c41a1e9ba9d4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/640eff5af2d7c41a1e9ba9d4/NcZga_7TbMOW4qhUrE7q1.jpeg","isPro":false,"fullname":"YOUNG SOOK SONG","user":"songys","type":"user"},{"_id":"68de11af9f7c3aa8f47dbee3","avatarUrl":"/avatars/bcf952d3b7545abfa8643d7ed2700ff1.svg","isPro":false,"fullname":"kaya-butter","user":"kaya-butter","type":"user"},{"_id":"62364e49cd8e1f47c8660c67","avatarUrl":"/avatars/94db216e1546e79671cea2dc7fd5469b.svg","isPro":false,"fullname":"Jaegeun Han","user":"haanjack","type":"user"},{"_id":"68369a89aa9ea2b61b632488","avatarUrl":"/avatars/685b6c434928cfe274c7ec545dc08a7c.svg","isPro":false,"fullname":"Sungmin Lee","user":"SungminLee","type":"user"},{"_id":"6a85373fc4c097c4a7747c3c","avatarUrl":"/avatars/937ea8c10e9b5148a23d82644ba531ec.svg","isPro":false,"fullname":"Sungmin Lee","user":"simon-smlee","type":"user"},{"_id":"61bf1b9ac88f3fd22f65405d","avatarUrl":"/avatars/39110562779f51957a8a200e34ab0071.svg","isPro":false,"fullname":"Taekyoon Ted Choi","user":"Taekyoon","type":"user"},{"_id":"65001f10a5c11e3a9af42ae4","avatarUrl":"/avatars/7ec4ab56017a65f93d1b8185c0314206.svg","isPro":false,"fullname":"Sean Jung","user":"sean-ai","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"66c2c83e37e5578c16d19500","name":"kakaocorp","fullname":"Kakao Corp.","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66c2a902b83a7e94d517c1ea/6Bo5zjJnRCtIi28zFUPmO.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.20061.md","query":{}}">

Papers

arxiv:2608.20061

Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts

Published on Aug 20

· Submitted by

Nayeon Kim on Aug 24

Kakao Corp.

Upvote

Authors:

Nayeon Kim ,

Jaesun Park ,

Abstract

A two-step hyperparameter transfer framework predicts optimal learning rates for large Mixture-of-Experts models by scaling across widths and token budgets, enabling efficient pretraining without costly sweeps.

Generated by thinkingmachines/Inkling-Small

Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost. However, optimizing their hyperparameters---particularly the learning rate---at extreme scales of both model size and token budget via sweeping remains computationally prohibitive. In this paper, we propose a compute-efficient, two-step hyperparameter transfer framework that estimates optimal learning rates for training large MoE models by transferring them across scaling model widths, and subsequently extrapolating to trillion-token horizons. First, we formulate a Maximal Update Parameterization (μP) adaptation for MoE architectures utilizing Multi-head Latent Attention (MLA) and the Muon optimizer, demonstrating that optimal learning rates transfer consistently across width-scaled models. Second, we extend this transferability along the token dimension by establishing a predictive scaling law. By applying linear regression to the optimal values derived from small proxy models on limited budgets, we successfully extrapolate the ideal learning rate to massive training horizons (e.g., 10 trillion tokens) with high fidelity (R^2=0.95). Consequently, this indicates that proxy training on small models is sufficient to determine the optimal learning rate for the extensive training of large-scale MoEs. We apply the proposed methodology to pretrain our foundation model (155B total, 17B active parameters) from scratch, and the stable training and evaluation results validate that optimal configurations for full-scale target models can be accurately predicted with minimal ablation costs.

View arXiv page View PDF Add to collection