Muon isn’t just for pretraining—this paper shows it can dramatically boost long-horizon agentic RL when paired with the right credit assignment strategy.</p>\n","updatedAt":"2026-07-20T08:10:09.674Z","author":{"_id":"6205fefd3f1dc8a642d70b10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1673948194938-6205fefd3f1dc8a642d70b10.jpeg","fullname":"Kai Ruan","name":"6cf","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":13,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8260976076126099},"editors":["6cf"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1673948194938-6205fefd3f1dc8a642d70b10.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.16169","authors":[{"_id":"6a5dd6486a69ce099f4d6f27","name":"Kai Ruan","hidden":false},{"_id":"6a5dd6486a69ce099f4d6f28","name":"Jinghao Lin","hidden":false},{"_id":"6a5dd6486a69ce099f4d6f29","name":"Zihe Huang","hidden":false},{"_id":"6a5dd6486a69ce099f4d6f2a","name":"Ziqi Zhou","hidden":false},{"_id":"6a5dd6486a69ce099f4d6f2b","name":"Qianshan Wei","hidden":false},{"_id":"6a5dd6486a69ce099f4d6f2c","name":"Xuan Wang","hidden":false},{"_id":"6a5dd6486a69ce099f4d6f2d","name":"Hao Sun","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/6205fefd3f1dc8a642d70b10/T1widoHJalmK6WHjAiRpQ.png"],"publishedAt":"2026-07-17T00:00:00.000Z","submittedOnDailyAt":"2026-07-20T00:00:00.000Z","title":"When Does Muon Help Agentic Reinforcement Learning?","submittedOnDailyBy":{"_id":"6205fefd3f1dc8a642d70b10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1673948194938-6205fefd3f1dc8a642d70b10.jpeg","isPro":false,"fullname":"Kai Ruan","user":"6cf","type":"user","name":"6cf"},"summary":"Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open.","upvotes":9,"discussionId":"6a5dd6496a69ce099f4d6f2e"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6205fefd3f1dc8a642d70b10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1673948194938-6205fefd3f1dc8a642d70b10.jpeg","isPro":false,"fullname":"Kai Ruan","user":"6cf","type":"user"},{"_id":"691aa7a783247309d5720e1b","avatarUrl":"/avatars/1de4741f0d8fc1174e3585fa257c3213.svg","isPro":false,"fullname":"Jackie Lam","user":"Jackie-Lam","type":"user"},{"_id":"668366de0b72be1367e60cd3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/668366de0b72be1367e60cd3/GyFEIpNFV4eOmMsOdBwja.jpeg","isPro":false,"fullname":"Yu Chen","user":"Rocky131","type":"user"},{"_id":"66e3939711e4f40aa75df2ff","avatarUrl":"/avatars/a2d12c3ba93d9e1a650f8283c33e02b3.svg","isPro":false,"fullname":"Cheng","user":"RosyCheng","type":"user"},{"_id":"673ed484635a89280b397356","avatarUrl":"/avatars/3a69ac575cdb14e12bd5b02363190cb2.svg","isPro":false,"fullname":"张佩宇","user":"GlowLED","type":"user"},{"_id":"65388514613fe158bd514e4c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65388514613fe158bd514e4c/hpCY_g2oxLd1Ruq8LTA29.jpeg","isPro":false,"fullname":"alterego238","user":"alterego238","type":"user"},{"_id":"69bcecd7374a079a5d1a722d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/aparkvjVun8TTjZo8t5lk.png","isPro":false,"fullname":"Song Haoran","user":"graysonf","type":"user"},{"_id":"6952244bfbddb08cb2562f3b","avatarUrl":"/avatars/70d67319af29604129378fee3f216757.svg","isPro":false,"fullname":"qianshan wei","user":"b1intern","type":"user"},{"_id":"68256a6c3da72aa4eb9e358d","avatarUrl":"/avatars/6ffb478930237cea395874265da6bb1a.svg","isPro":false,"fullname":"Jinkun Chen","user":"Fyvoid","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.16169.md","query":{}}">
When Does Muon Help Agentic Reinforcement Learning?
Abstract
Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open.
Community
Muon isn’t just for pretraining—this paper shows it can dramatically boost long-horizon agentic RL when paired with the right credit assignment strategy.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.16169 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.16169 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.16169 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.