On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine \"self\"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively.</p>\n","updatedAt":"2026-08-11T21:07:48.477Z","author":{"_id":"6419309f22270b3ccf177c77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg","fullname":"William Li","name":"williamium","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":5,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9303058981895447},"editors":["williamium"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg"],"reactions":[],"isReport":false}},{"id":"6a7bce8a31daa4ee29dfb047","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":378,"isUserFollowing":false},"createdAt":"2026-08-12T01:38:18.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Consensus as Privileged Context for Label-Free Self-Distillation](https://huggingface.co/papers/2607.13643) (2026)\n* [dOPSD: On-Policy Self-Distillation for Diffusion Language Models](https://huggingface.co/papers/2607.04428) (2026)\n* [RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models](https://huggingface.co/papers/2607.24447) (2026)\n* [ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation](https://huggingface.co/papers/2606.23104) (2026)\n* [Trace-Based On-Policy Distillation for Masked Diffusion Language Models](https://huggingface.co/papers/2607.16872) (2026)\n* [Learning from the Self-future: On-policy Self-distillation for dLLMs](https://huggingface.co/papers/2606.18195) (2026)\n* [DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models](https://huggingface.co/papers/2608.06243) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"<p>This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. </p>\n<p>The following papers were recommended by the Semantic Scholar API </p>\n<ul>\n<li><a href=\"https://huggingface.co/papers/2607.13643\">Consensus as Privileged Context for Label-Free Self-Distillation</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.04428\">dOPSD: On-Policy Self-Distillation for Diffusion Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.24447\">RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.23104\">ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.16872\">Trace-Based On-Policy Distillation for Masked Diffusion Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.18195\">Learning from the Self-future: On-policy Self-distillation for dLLMs</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.06243\">DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models</a> (2026)</li>\n</ul>\n<p> Please give a thumbs up to this comment if you found it helpful!</p>\n<p> If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space</p>\n<p> You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code>@librarian-bot recommend</code></p>\n","updatedAt":"2026-08-12T01:38:18.986Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":378,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7378172874450684},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.06296","authors":[{"_id":"6a7b8eceb7183653340c1446","name":"Yijiang Li","hidden":false},{"_id":"6a7b8eceb7183653340c1447","name":"Bingyang Wang","hidden":false},{"_id":"6a7b8eceb7183653340c1448","name":"Yijun Liang","hidden":false},{"_id":"6a7b8eceb7183653340c1449","name":"Yunjie Tian","hidden":false},{"_id":"6a7b8eceb7183653340c144a","name":"Di Fu","hidden":false},{"_id":"6a7b8eceb7183653340c144b","name":"Nuno Vasconcelos","hidden":false}],"publishedAt":"2026-08-09T00:00:00.000Z","submittedOnDailyAt":"2026-08-11T00:00:00.000Z","title":"On-Policy Self-Distillation without Any Supervision","submittedOnDailyBy":{"_id":"6419309f22270b3ccf177c77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg","isPro":true,"fullname":"William Li","user":"williamium","type":"user","name":"williamium"},"summary":"On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine \"self\"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at [https://github.com/williamium3000/u-opsd](https://github.com/williamium3000/u-opsd).","upvotes":63,"discussionId":"6a7b8ecfb7183653340c144c","projectPage":"https://williamium3000.github.io/u-opsd/","ai_summary":"Unsupervised on-policy self-distillation improves large language models by using internal consistency and majority-vote pseudo-solutions to correct confident errors without external supervision.","ai_keywords":["on-policy self-distillation","self-consistency","pseudo-solution","majority vote","unsupervised on-policy self-distillation","U-OPSD","GRPO","reasoning benchmarks"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"697e87d12cc19315a8497001","name":"UCSanDiego","fullname":"University of California at San Diego","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/697e8687c00f332cf492d29e/KUQpvngxP4r9oBSDZwIwZ.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6419309f22270b3ccf177c77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg","isPro":true,"fullname":"William Li","user":"williamium","type":"user"},{"_id":"66720ab819bebc69b5b93685","avatarUrl":"/avatars/b2f1314d9a26f6f5eaf6cebdb0d28812.svg","isPro":true,"fullname":"Yijun Liang","user":"joliang17","type":"user"},{"_id":"68f193a1bebd45d3b89c06fa","avatarUrl":"/avatars/1543609163e78699f4e8e87954f5fa27.svg","isPro":false,"fullname":"Yunjie Tian","user":"sunsmarterjie2","type":"user"},{"_id":"60ebb7211a27bb2618b766d1","avatarUrl":"/avatars/47624a0dae4f8ab413eeb025b20504e2.svg","isPro":false,"fullname":"yejin","user":"yejin","type":"user"},{"_id":"643822111ee0e43f14d7ebf5","avatarUrl":"/avatars/0a6b7bfc438b7b331c81ff59f22c7b65.svg","isPro":false,"fullname":"Jiayu Zheng","user":"BubbleJoe","type":"user"},{"_id":"6a6300d4f71a0ec03ce14daf","avatarUrl":"/avatars/20187beface35d8bdc45109a1178e9f8.svg","isPro":false,"fullname":"Shufan Wang","user":"Pcbuilding-wang","type":"user"},{"_id":"66393f5a1231260674ae798e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66393f5a1231260674ae798e/JlWLdLTjMwgh0x1ASx4Mx.jpeg","isPro":false,"fullname":"Haichao Zhang","user":"haichaozhang","type":"user"},{"_id":"654427a0326cb9a32be7a0e9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/654427a0326cb9a32be7a0e9/-yiqFr8v6luvuhfYaqFbA.jpeg","isPro":false,"fullname":"Icy Wang","user":"Icey444","type":"user"},{"_id":"6532c3018bde2fae19578587","avatarUrl":"/avatars/7231538f3d682a1e7b80e15ea91b2a97.svg","isPro":false,"fullname":"X","user":"Hudx111","type":"user"},{"_id":"63ef0af2bfe4ead22ca8f69a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1676610243576-noauth.jpeg","isPro":false,"fullname":"Haozheng Luo","user":"robinzixuan","type":"user"},{"_id":"6a630696e07f75b8f88ebe85","avatarUrl":"/avatars/386bd884058b0e7b1d398b07e97dbd40.svg","isPro":false,"fullname":"Chengfan Li","user":"enlacatedral","type":"user"},{"_id":"6a67987555648d99d3b97257","avatarUrl":"/avatars/c9edc183823a6aba25d9e3e477778fbe.svg","isPro":false,"fullname":"Bozheng LI","user":"bozheng4704","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"697e87d12cc19315a8497001","name":"UCSanDiego","fullname":"University of California at San Diego","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/697e8687c00f332cf492d29e/KUQpvngxP4r9oBSDZwIwZ.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.06296.md","query":{}}">
On-Policy Self-Distillation without Any Supervision
Abstract
Unsupervised on-policy self-distillation improves large language models by using internal consistency and majority-vote pseudo-solutions to correct confident errors without external supervision.
On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at [https://github.com/williamium3000/u-opsd](https://github.com/williamium3000/u-opsd).
Community
On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.06296 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.06296 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.06296 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.