Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning methods. While the Maximal Update Parametrization (μP) has enabled zero-shot hyperparameter transfer for standard Transformers, its extension to linear models, particularly those with structured state transitions and complicated architectures, remains largely unexplored. By rigorously propagating coordinate-size estimates through the forward pass, gating mechanisms, and recurrent state dynamics, we derive the scaling rules for Gated Delta Network. Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer across model widths under both AdamW and SGD, whereas standard parametrization fails to transfer, validating the correctness and practical utility of our analysis.</p>\n","updatedAt":"2026-06-04T03:49:10.542Z","author":{"_id":"653d276681f52ceb4d12bd85","avatarUrl":"/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg","fullname":"Yifeng Liu","name":"Lewis-Lau","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8349835276603699},"editors":["Lewis-Lau"],"editorAvatarUrls":["/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.04048","authors":[{"_id":"6a20f58915100c5272a84725","name":"Yifeng Liu","hidden":false},{"_id":"6a20f58915100c5272a84726","name":"Quanquan Gu","hidden":false}],"publishedAt":"2026-06-02T00:00:00.000Z","submittedOnDailyAt":"2026-06-04T00:00:00.000Z","title":"Unlocking Feature Learning in Gated Delta Networks at Scale","submittedOnDailyBy":{"_id":"653d276681f52ceb4d12bd85","avatarUrl":"/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg","isPro":false,"fullname":"Yifeng Liu","user":"Lewis-Lau","type":"user","name":"Lewis-Lau"},"summary":"Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning methods. While the Maximal Update Parametrization (μP) has enabled zero-shot hyperparameter transfer for standard Transformers, its extension to linear models, particularly those with structured state transitions and complicated architectures, remains largely unexplored. By rigorously propagating coordinate-size estimates through the forward pass, gating mechanisms, and recurrent state dynamics, we derive the scaling rules for Gated Delta Network. Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer across model widths under both AdamW and SGD, whereas standard parametrization fails to transfer, validating the correctness and practical utility of our analysis.","upvotes":2,"discussionId":"6a20f58a15100c5272a84727","githubRepo":"https://github.com/lauyikfung/gated_delta_net_mup","githubRepoAddedBy":"user","ai_summary":"Scaling rules for Gated Delta Networks are derived through coordinate-size estimation propagation, enabling stable learning-rate transfer across model widths with both AdamW and SGD optimizers.","ai_keywords":["Large Language Models","sub-quadratic architectures","hyperparameter tuning","Maximal Update Parametrization","Transformers","linear models","structured state transitions","gated delta network","coordinate-size estimates","forward pass","gating mechanisms","recurrent state dynamics","language-model pre-training","AdamW","SGD"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0,"organization":{"_id":"67784c39dac147922d8d09f0","name":"UCLA","fullname":"University of California, Los Angeles","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67784bd637dfa531fbce95a2/Nf0seEMEn66sPL3QsJXj4.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"653d276681f52ceb4d12bd85","avatarUrl":"/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg","isPro":false,"fullname":"Yifeng Liu","user":"Lewis-Lau","type":"user"},{"_id":"6984dd7d40e2c84073af8286","avatarUrl":"/avatars/6548aa808ee225af20ea91b9fc890937.svg","isPro":false,"fullname":"Ashley Miller","user":"yoomxyag","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"67784c39dac147922d8d09f0","name":"UCLA","fullname":"University of California, Los Angeles","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67784bd637dfa531fbce95a2/Nf0seEMEn66sPL3QsJXj4.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.04048.md"}">
Unlocking Feature Learning in Gated Delta Networks at Scale
Abstract
Scaling rules for Gated Delta Networks are derived through coordinate-size estimation propagation, enabling stable learning-rate transfer across model widths with both AdamW and SGD optimizers.
Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning methods. While the Maximal Update Parametrization (μP) has enabled zero-shot hyperparameter transfer for standard Transformers, its extension to linear models, particularly those with structured state transitions and complicated architectures, remains largely unexplored. By rigorously propagating coordinate-size estimates through the forward pass, gating mechanisms, and recurrent state dynamics, we derive the scaling rules for Gated Delta Network. Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer across model widths under both AdamW and SGD, whereas standard parametrization fails to transfer, validating the correctness and practical utility of our analysis.
Community
Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning methods. While the Maximal Update Parametrization (μP) has enabled zero-shot hyperparameter transfer for standard Transformers, its extension to linear models, particularly those with structured state transitions and complicated architectures, remains largely unexplored. By rigorously propagating coordinate-size estimates through the forward pass, gating mechanisms, and recurrent state dynamics, we derive the scaling rules for Gated Delta Network. Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer across model widths under both AdamW and SGD, whereas standard parametrization fails to transfer, validating the correctness and practical utility of our analysis.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.04048 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.04048 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.04048 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.