<video src=\"https://cdn-uploads.huggingface.co/production/uploads/65e8c0e40a099ee0c1302e57/GG0WzCpIQkLdec4Qy5wUq.mp4\" controls=\"\" class=\"max-w-full!\"></video></p>","updatedAt":"2026-07-29T17:41:27.266Z","author":{"_id":"65e8c0e40a099ee0c1302e57","avatarUrl":"/avatars/3905f563be224c0c296dc5c942329d8d.svg","fullname":"Pierre Chambon","name":"pierrechambon","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.901317834854126},"editors":["pierrechambon"],"editorAvatarUrls":["/avatars/3905f563be224c0c296dc5c942329d8d.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.25970","authors":[{"_id":"6a6a39604463a8a84bdc3e65","name":"Pierre Chambon","hidden":false},{"_id":"6a6a39604463a8a84bdc3e66","name":"Kunhao Zheng","hidden":false},{"_id":"6a6a39604463a8a84bdc3e67","name":"Juliette Decugis","hidden":false},{"_id":"6a6a39604463a8a84bdc3e68","name":"Benoit Sagot","hidden":false},{"_id":"6a6a39604463a8a84bdc3e69","name":"Gabriel Synnaeve","hidden":false}],"publishedAt":"2026-07-28T00:00:00.000Z","submittedOnDailyAt":"2026-07-29T00:00:00.000Z","title":"Reinforcement Learning for Code Optimization","submittedOnDailyBy":{"_id":"65e8c0e40a099ee0c1302e57","avatarUrl":"/avatars/3905f563be224c0c296dc5c942329d8d.svg","isPro":false,"fullname":"Pierre Chambon","user":"pierrechambon","type":"user","name":"pierrechambon"},"summary":"RL for code correctness is now established: have the model generate a program, run it against hidden test cases, and reward solutions that pass. Extending this to code optimization seems straightforward: just add execution time to the reward. But in practice, once timing drives the reward, small problems in measurement noise, reward sparsity, or GRPO instability overwhelm the signal and make RL fail: generated solutions are barely faster, and more of them can fail. We make execution time learnable through three stages: (1) how code is tested, by building DMC-Optim with large optimization tests and a calibrated sandbox; (2) how speed is turned into reward, by composing correctness and speed in the RL environment and using an offline simulator to predict the most promising configurations; and (3) how the model learns from that reward, by adapting GRPO and evaluation to the sparser, noisier timed-execution setting. On DMC-Optim, the strongest optimization-aware configurations improve strict top-50% pass@1 from 18.0% to 31.3% on Qwen 2.5 7B and from 30.7% to 50.4% on CWM 32B. These gains further increase at stricter percentiles such as top-30%, with 125% relative improvement for CWM 32B, while preserving pure-correctness scores. When the timing sandbox is degraded, robust optimization RL reaches 100% to 200% improvement over standard RLVR, depending on the evaluation criterion. On LCB, CWM 32B wins up to 83% of median-sample speed comparisons against standard RLVR. Relative to the fastest correct human submissions per problem, it reaches about half the human rate of complexity-class improvements (14% vs. 28%).","upvotes":3,"discussionId":"6a6a39604463a8a84bdc3e6a","organization":{"_id":"5e63d8713071d5be688861b8","name":"facebook","fullname":"AI at Meta","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1592839207516-noauth.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"65e8c0e40a099ee0c1302e57","avatarUrl":"/avatars/3905f563be224c0c296dc5c942329d8d.svg","isPro":false,"fullname":"Pierre Chambon","user":"pierrechambon","type":"user"},{"_id":"6702f8bd419fcb9e5d4e894c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6702f8bd419fcb9e5d4e894c/QgFKs5g8MUm-yIl0IVNDB.png","isPro":false,"fullname":"Ron","user":"ronedgecomb","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"5e63d8713071d5be688861b8","name":"facebook","fullname":"AI at Meta","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1592839207516-noauth.png"},"query":{}}">
Reinforcement Learning for Code Optimization
Abstract
RL for code correctness is now established: have the model generate a program, run it against hidden test cases, and reward solutions that pass. Extending this to code optimization seems straightforward: just add execution time to the reward. But in practice, once timing drives the reward, small problems in measurement noise, reward sparsity, or GRPO instability overwhelm the signal and make RL fail: generated solutions are barely faster, and more of them can fail. We make execution time learnable through three stages: (1) how code is tested, by building DMC-Optim with large optimization tests and a calibrated sandbox; (2) how speed is turned into reward, by composing correctness and speed in the RL environment and using an offline simulator to predict the most promising configurations; and (3) how the model learns from that reward, by adapting GRPO and evaluation to the sparser, noisier timed-execution setting. On DMC-Optim, the strongest optimization-aware configurations improve strict top-50% pass@1 from 18.0% to 31.3% on Qwen 2.5 7B and from 30.7% to 50.4% on CWM 32B. These gains further increase at stricter percentiles such as top-30%, with 125% relative improvement for CWM 32B, while preserving pure-correctness scores. When the timing sandbox is degraded, robust optimization RL reaches 100% to 200% improvement over standard RLVR, depending on the evaluation criterion. On LCB, CWM 32B wins up to 83% of median-sample speed comparisons against standard RLVR. Relative to the fastest correct human submissions per problem, it reaches about half the human rate of complexity-class improvements (14% vs. 28%).
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.25970 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.25970 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.25970 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.