Hugging Face Daily Papers · May 18, 2026 · 7 min read

Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

This paper offers an insightful explanation of the training efficiency of On-Policy Distillation (OPD): the reason OPD is faster is that it can form relatively stable update directions close to the final solution early in training. The paper analyzes this “foresight” mechanism from two perspectives: module allocation and low-rank update directions. It further proposes EffOPD, which accelerates training by extrapolating along early update directions. Experiments show that EffOPD achieves around 3× training speedup while maintaining comparable performance. Overall, the paper advances the understanding of OPD’s efficiency from an empirical observation to a parameter-dynamics perspective, offering a novel analytical angle and a simple yet practical method.\n","updatedAt":"2026-05-18T07:33:13.024Z","author":{"_id":"651e2a19242e10766e61a669","avatarUrl":"/avatars/e5351fb14997269d8d3b9539f6f27d9e.svg","fullname":"caiyuchen","name":"caiyuchen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":2,"identifiedLanguage":{"language":"en","probability":0.8881392478942871},"editors":["caiyuchen"],"editorAvatarUrls":["/avatars/e5351fb14997269d8d3b9539f6f27d9e.svg"],"reactions":[],"isReport":false}},{"id":"6a0bc129c60913e432c34b03","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":357,"isUserFollowing":false},"createdAt":"2026-05-19T01:47:21.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [A Survey of On-Policy Distillation for Large Language Models](https://huggingface.co/papers/2604.00626) (2026)\n* [Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes](https://huggingface.co/papers/2603.25562) (2026)\n* [Curriculum Learning-Guided Progressive Distillation in Large Language Models](https://huggingface.co/papers/2605.11260) (2026)\n* [KL for a KL: On-Policy Distillation with Control Variate Baseline](https://huggingface.co/papers/2605.07865) (2026)\n* [Combining On-Policy Optimization and Distillation for Long-Context Reasoning in Large Language Models](https://huggingface.co/papers/2605.12227) (2026)\n* [Distribution Corrected Offline Data Distillation for Large Language Models](https://huggingface.co/papers/2605.14071) (2026)\n* [Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation](https://huggingface.co/papers/2604.13010) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. \nThe following papers were recommended by the Semantic Scholar API \n<ul>\n<li><a href=\"https://huggingface.co/papers/2604.00626\">A Survey of On-Policy Distillation for Large Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2603.25562\">Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.11260\">Curriculum Learning-Guided Progressive Distillation in Large Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.07865\">KL for a KL: On-Policy Distillation with Control Variate Baseline</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.12227\">Combining On-Policy Optimization and Distillation for Long-Context Reasoning in Large Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.14071\">Distribution Corrected Offline Data Distillation for Large Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2604.13010\">Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation</a> (2026)</li>\n</ul>\n Please give a thumbs up to this comment if you found it helpful!\n If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code><a href=\"/librarian-bot\">@librarian-bot</a> recommend</code>\n","updatedAt":"2026-05-19T01:47:21.112Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":357,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7263587713241577},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.11739","authors":[{"_id":"6a05d416b1a8cbabc9f094eb","user":{"_id":"651e2a19242e10766e61a669","avatarUrl":"/avatars/e5351fb14997269d8d3b9539f6f27d9e.svg","isPro":false,"fullname":"caiyuchen","user":"caiyuchen","type":"user","name":"caiyuchen"},"name":"Yuchen Cai","status":"claimed_verified","statusLastChangedAt":"2026-05-18T09:46:49.111Z","hidden":false},{"_id":"6a05d416b1a8cbabc9f094ec","name":"Ding Cao","hidden":false},{"_id":"6a05d416b1a8cbabc9f094ed","name":"Liang Lin","hidden":false},{"_id":"6a05d416b1a8cbabc9f094ee","name":"Chunxi Luo","hidden":false},{"_id":"6a05d416b1a8cbabc9f094ef","user":{"_id":"64e2d169d2af12910d682130","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64e2d169d2af12910d682130/VG8UdqJCJGc0K4G0P0XQP.jpeg","isPro":false,"fullname":"xuxin","user":"xx18","type":"user","name":"xx18"},"name":"Xin Xu","status":"claimed_verified","statusLastChangedAt":"2026-05-18T09:46:46.835Z","hidden":false},{"_id":"6a05d416b1a8cbabc9f094f0","name":"Kai Yang","hidden":false},{"_id":"6a05d416b1a8cbabc9f094f1","name":"Weijie Liu","hidden":false},{"_id":"6a05d416b1a8cbabc9f094f2","name":"Saiyong Yang","hidden":false},{"_id":"6a05d416b1a8cbabc9f094f3","name":"Tianxiang Zhao","hidden":false},{"_id":"6a05d416b1a8cbabc9f094f4","name":"Guangzhong Sun","hidden":false},{"_id":"6a05d416b1a8cbabc9f094f5","name":"Guiquan Liu","hidden":false},{"_id":"6a05d416b1a8cbabc9f094f6","name":"Junfeng Fang","hidden":false}],"publishedAt":"2026-05-13T00:00:00.000Z","submittedOnDailyAt":"2026-05-18T00:00:00.000Z","title":"Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation","submittedOnDailyBy":{"_id":"651e2a19242e10766e61a669","avatarUrl":"/avatars/e5351fb14997269d8d3b9539f6f27d9e.svg","isPro":false,"fullname":"caiyuchen","user":"caiyuchen","type":"user","name":"caiyuchen"},"summary":"On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the Module-Allocation Level, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the Update-Direction Level, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose EffOPD, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of 3times while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for large language models.","upvotes":49,"discussionId":"6a05d416b1a8cbabc9f094ff","projectPage":"https://arxiv.org/abs/2605.11739","githubRepo":"https://github.com/caiyuchen-ustc/EffOPD","githubRepoAddedBy":"user","ai_summary":"On-policy distillation efficiency arises from early establishment of stable update trajectories, with findings leading to a plug-and-play acceleration method achieving 3x training speedup.","ai_keywords":["on-policy distillation","post-training paradigm","parameter-level mechanisms","module-allocation level","update-direction level","low-rank concentration","extrapolation step size","update trajectory","training acceleration"],"githubStars":13,"organization":{"_id":"6645f953c39288df638dbdd5","name":"Tencent-Hunyuan","fullname":"Tencent Hunyuan","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/62d22496c58f969c152bcefd/woKSjt2wXvBNKussyYPsa.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"651e2a19242e10766e61a669","avatarUrl":"/avatars/e5351fb14997269d8d3b9539f6f27d9e.svg","isPro":false,"fullname":"caiyuchen","user":"caiyuchen","type":"user"},{"_id":"64e2d169d2af12910d682130","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64e2d169d2af12910d682130/VG8UdqJCJGc0K4G0P0XQP.jpeg","isPro":false,"fullname":"xuxin","user":"xx18","type":"user"},{"_id":"69bcef54696bd8657c82efd4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/gzjJ77rITv25AUD5Dhq-8.png","isPro":false,"fullname":"山下颯太","user":"thomas-taylor","type":"user"},{"_id":"69cd37430da32eaa24652979","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Fk37TBCPqhH-CBYZFbYdD.jpeg","isPro":false,"fullname":"朱沐宇","user":"lyoung2023","type":"user"},{"_id":"699db0d31fba232a34c15f07","avatarUrl":"/avatars/a5994d36eb614f9cee5514c27e034b20.svg","isPro":false,"fullname":"Ma Yichen","user":"ma-yichen","type":"user"},{"_id":"69ccf7749da28b72a80edd75","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4xGoPAP6OZS1ZWPyAikhA.jpeg","isPro":false,"fullname":"Joseph Sanchez","user":"wu-wenhao19","type":"user"},{"_id":"674572a99543fbaf3c63f35b","avatarUrl":"/avatars/6c891450c2ceeb7b034556548afc772d.svg","isPro":false,"fullname":"蔡正舟","user":"conctsai","type":"user"},{"_id":"69bcfd27745b859b4b20c7fc","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/TA-T4r9HNMjQdhMBkxL0-.png","isPro":false,"fullname":"宇轩马","user":"lhall33","type":"user"},{"_id":"69bba1740b329303783193c6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/y4mD3hEDw7dM550zJIXtU.png","isPro":false,"fullname":"王林曦","user":"matthewpvpc55","type":"user"},{"_id":"698f8ee4801eed96d9d3555a","avatarUrl":"/avatars/f98f9b01a6cceb0f725185fde6187be6.svg","isPro":false,"fullname":"Q6v1uxi6","user":"q6v1uxi6","type":"user"},{"_id":"69a3c3eb2927b6e8152e9d65","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QvgoAncC4ErE1GZUMLJMK.png","isPro":false,"fullname":"Полина Козлов","user":"johnlopez61","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6645f953c39288df638dbdd5","name":"Tencent-Hunyuan","fullname":"Tencent Hunyuan","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/62d22496c58f969c152bcefd/woKSjt2wXvBNKussyYPsa.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.11739.md"}">

Papers

arxiv:2605.11739

Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

Published on May 13

· Submitted by

caiyuchen on May 18

Tencent Hunyuan

Upvote

Authors:

Yuchen Cai ,

Xin Xu ,

Abstract

On-policy distillation efficiency arises from early establishment of stable update trajectories, with findings leading to a plug-and-play acceleration method achieving 3x training speedup.

AI-generated summary

On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the Module-Allocation Level, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the Update-Direction Level, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose EffOPD, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of 3times while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for large language models.

View arXiv page View PDF Project page GitHub 13 Add to collection

Community

caiyuchen

Paper author Paper submitter about 19 hours ago

•

edited about 18 hours ago

This paper offers an insightful explanation of the training efficiency of On-Policy Distillation (OPD): the reason OPD is faster is that it can form relatively stable update directions close to the final solution early in training. The paper analyzes this “foresight” mechanism from two perspectives: module allocation and low-rank update directions. It further proposes EffOPD, which accelerates training by extrapolating along early update directions. Experiments show that EffOPD achieves around 3× training speedup while maintaining comparable performance. Overall, the paper advances the understanding of OPD’s efficiency from an empirical observation to a parameter-dynamics perspective, offering a novel analytical angle and a simple yet practical method.