On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss. We therefore propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation, exact full-vocabulary on-policy self-distillation, as well as reference-policy KL regularization. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines.</p>\n","updatedAt":"2026-06-04T01:39:59.212Z","author":{"_id":"653d276681f52ceb4d12bd85","avatarUrl":"/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg","fullname":"Yifeng Liu","name":"Lewis-Lau","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.879159152507782},"editors":["Lewis-Lau"],"editorAvatarUrls":["/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg"],"reactions":[],"isReport":false}},{"id":"6a20ee0e521d74082657f818","author":{"_id":"661ab1f1fa3b144a381fa454","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png","fullname":"Urro","name":"urroxyz","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":11,"isUserFollowing":false},"createdAt":"2026-06-04T03:16:30.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Refreshing paper to read.\n\nI like it. Simple and useful.\n\nAnd thank you for releasing the code.","html":"<p>Refreshing paper to read.</p>\n<p>I like it. Simple and useful.</p>\n<p>And thank you for releasing the code.</p>\n","updatedAt":"2026-06-04T03:16:30.080Z","author":{"_id":"661ab1f1fa3b144a381fa454","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png","fullname":"Urro","name":"urroxyz","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":11,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9406116604804993},"editors":["urroxyz"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.04036","authors":[{"_id":"6a20d73a15100c5272a8463d","name":"Yifeng Liu","hidden":false},{"_id":"6a20d73a15100c5272a8463e","name":"Shiyuan Zhang","hidden":false},{"_id":"6a20d73a15100c5272a8463f","name":"Yifan Zhang","hidden":false},{"_id":"6a20d73a15100c5272a84640","name":"Quanquan Gu","hidden":false}],"publishedAt":"2026-06-02T00:00:00.000Z","submittedOnDailyAt":"2026-06-04T00:00:00.000Z","title":"Self-Distilled Policy Gradient","submittedOnDailyBy":{"_id":"653d276681f52ceb4d12bd85","avatarUrl":"/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg","isPro":false,"fullname":"Yifeng Liu","user":"Lewis-Lau","type":"user","name":"Lewis-Lau"},"summary":"On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss. We therefore propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation, exact full-vocabulary on-policy self-distillation, as well as reference-policy KL regularization. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines. The code is available at https://github.com/lauyikfung/SDPG.","upvotes":9,"discussionId":"6a20d73a15100c5272a84641","projectPage":"https://lauyikfung.github.io/SDPG/","githubRepo":"https://github.com/lauyikfung/SDPG","githubRepoAddedBy":"user","ai_summary":"A self-distilled policy-gradient framework combines on-policy self-distillation with verifier advantages and KL regularization to improve reinforcement learning stability and performance.","ai_keywords":["self-distillation","policy-gradient","reverse Kullback-Leibler divergence","on-policy learning","verifier advantages","KL regularization","reinforcement learning"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0,"organization":{"_id":"67784c39dac147922d8d09f0","name":"UCLA","fullname":"University of California, Los Angeles","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67784bd637dfa531fbce95a2/Nf0seEMEn66sPL3QsJXj4.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"653d276681f52ceb4d12bd85","avatarUrl":"/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg","isPro":false,"fullname":"Yifeng Liu","user":"Lewis-Lau","type":"user"},{"_id":"661ab1f1fa3b144a381fa454","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png","isPro":false,"fullname":"Urro","user":"urroxyz","type":"user"},{"_id":"6323f399462470712720c155","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6323f399462470712720c155/SWsMNa7vETUSrOt9Qf-oe.png","isPro":false,"fullname":"Yinxu Pan","user":"cppowboy","type":"user"},{"_id":"647bf082aba7062fe5c51ca9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/VvKAhQC_LxBcBuy3XROSX.jpeg","isPro":true,"fullname":"Yifan Zhang","user":"yifAI","type":"user"},{"_id":"6a18aa9dfa4e01ee51e5a696","avatarUrl":"/avatars/71424c3cfda1ccca8a2fd8fa56c26130.svg","isPro":false,"fullname":"Yif","user":"yifanzhangnv","type":"user"},{"_id":"64b2f97434a92b848c7e941e","avatarUrl":"/avatars/c699c50f3b43cd1641469521127753bb.svg","isPro":false,"fullname":"Nagori","user":"MohammedNaeem","type":"user"},{"_id":"66935bdc5489e4f73c76bc7b","avatarUrl":"/avatars/129d1e86bbaf764b507501f4feb177db.svg","isPro":false,"fullname":"Abidoye Aanuoluwapo","user":"Aanuoluwapo65","type":"user"},{"_id":"69830152b0a0db5422531de2","avatarUrl":"/avatars/9574d1c3acccfca9494b90cfead96b9f.svg","isPro":false,"fullname":"Fatima El-Haddad","user":"digital-wanderer","type":"user"},{"_id":"629f3b18ee05727ce328ccbe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1669189789447-629f3b18ee05727ce328ccbe.jpeg","isPro":false,"fullname":"Kashif Rasul","user":"kashif","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"67784c39dac147922d8d09f0","name":"UCLA","fullname":"University of California, Los Angeles","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67784bd637dfa531fbce95a2/Nf0seEMEn66sPL3QsJXj4.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.04036.md"}">
Self-Distilled Policy Gradient
Abstract
A self-distilled policy-gradient framework combines on-policy self-distillation with verifier advantages and KL regularization to improve reinforcement learning stability and performance.
On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss. We therefore propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation, exact full-vocabulary on-policy self-distillation, as well as reference-policy KL regularization. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines. The code is available at https://github.com/lauyikfung/SDPG.
Community
On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kullback-Leibler divergence loss. We therefore propose SDPG, a self-distilled policy-gradient framework that combines group-relative verifier advantages with normalized standard deviation, exact full-vocabulary on-policy self-distillation, as well as reference-policy KL regularization. Empirically, SDPG improves stability and performance over RLVR and self-distillation baselines.
Refreshing paper to read.
I like it. Simple and useful.
And thank you for releasing the code.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.04036 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.04036 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.04036 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.