On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.</p>\n","updatedAt":"2026-06-09T02:23:34.372Z","author":{"_id":"64bfa52094c0e3be4a33863e","avatarUrl":"/avatars/bc0737bf4481433b669433facf6ce6f4.svg","fullname":"Shenzhennan","name":"5456es","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9018331170082092},"editors":["5456es"],"editorAvatarUrls":["/avatars/bc0737bf4481433b669433facf6ce6f4.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.07082","authors":[{"_id":"6a2778236dde1c5ef75bceee","name":"Zhennan Shen","hidden":false},{"_id":"6a2778236dde1c5ef75bceef","name":"Yanshu Li","hidden":false},{"_id":"6a2778236dde1c5ef75bcef0","name":"Qingyu Yin","hidden":false},{"_id":"6a2778236dde1c5ef75bcef1","name":"Chak Tou Leong","hidden":false},{"_id":"6a2778236dde1c5ef75bcef2","name":"Zhilin Wang","hidden":false},{"_id":"6a2778236dde1c5ef75bcef3","name":"Yanxu Chen","hidden":false},{"_id":"6a2778236dde1c5ef75bcef4","name":"Rongduo Han","hidden":false},{"_id":"6a2778236dde1c5ef75bcef5","name":"Sunbowen Lee","hidden":false},{"_id":"6a2778236dde1c5ef75bcef6","name":"Yi R. Fung","hidden":false}],"publishedAt":"2026-06-05T00:00:00.000Z","submittedOnDailyAt":"2026-06-09T00:00:00.000Z","title":"On the Geometry of On-Policy Distillation","submittedOnDailyBy":{"_id":"64bfa52094c0e3be4a33863e","avatarUrl":"/avatars/bc0737bf4481433b669433facf6ce6f4.svg","isPro":false,"fullname":"Shenzhennan","user":"5456es","type":"user","name":"5456es"},"summary":"On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.","upvotes":48,"discussionId":"6a2778236dde1c5ef75bcef7","ai_summary":"On-policy distillation exhibits distinct parameter space dynamics characterized by relaxed off-principal updates and subspace locking, forming a unique geometric pattern separate from supervised fine-tuning and reinforcement learning with verifiable rewards.","ai_keywords":["on-policy distillation","parameter space","supervised fine-tuning","reinforcement learning","verifiable rewards","parameter-space diagnostics","subspace locking","update subspace","rank dynamics"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"672d6e79bde669ec7e59ecc6","name":"hkustitd","fullname":"香港科技大学","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a757020246fad8d9c95265/PJZHQfGWiA_HNHI0S8vCz.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64bfa52094c0e3be4a33863e","avatarUrl":"/avatars/bc0737bf4481433b669433facf6ce6f4.svg","isPro":false,"fullname":"Shenzhennan","user":"5456es","type":"user"},{"_id":"65372bae0d973d3fee4131c7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/OeBjr1hlUINmjUJ8iu42I.png","isPro":false,"fullname":"BadCat","user":"Foresta","type":"user"},{"_id":"69cbf010de9f33f48b52800c","avatarUrl":"/avatars/5e9717ab44ef8a570f47b03a2a5cbdc9.svg","isPro":false,"fullname":"AndyXu","user":"Rupert-Griffin12","type":"user"},{"_id":"65745569839aa08899ea5d27","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg","isPro":false,"fullname":"Kailin Jiang","user":"kailinjiang","type":"user"},{"_id":"629311a945f405d06678224b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1661878264498-629311a945f405d06678224b.png","isPro":false,"fullname":"Chengguang Gan","user":"ganchengguang","type":"user"},{"_id":"647354de7afa69c3c7a5acc1","avatarUrl":"/avatars/0d54c598adc4bef4be4f55d024eadb29.svg","isPro":false,"fullname":"lizou","user":"lizouss","type":"user"},{"_id":"68986aab81115b07e63e875d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0fY9XZmKW2H6AdiX-G3Zb.png","isPro":false,"fullname":"Tiancheng Xing","user":"nayameow7","type":"user"},{"_id":"63218319ad7399662c790148","avatarUrl":"/avatars/7c5db4086b8ed80afc9a22bb8be968c4.svg","isPro":false,"fullname":"yht","user":"yht0511","type":"user"},{"_id":"675eebb86205e9e0d9a0feab","avatarUrl":"/avatars/8204df1b850e556094f51c4ad019fc7c.svg","isPro":false,"fullname":"ccrrkk","user":"ccrrkk","type":"user"},{"_id":"68989015e9e1c6671e60b305","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/4RgiFj8uAMrFPjG-roJDc.png","isPro":false,"fullname":"d3do","user":"d3do","type":"user"},{"_id":"64bd46d1cf4f379eeb9f8f3d","avatarUrl":"/avatars/65a841a7d3d272fab3799240941b995a.svg","isPro":false,"fullname":"jingkuan wang","user":"jkKing","type":"user"},{"_id":"65db54e5ab2f64915c0b9cf0","avatarUrl":"/avatars/d451523cad8c17cda603eea2961c50ad.svg","isPro":false,"fullname":"Shaoqiu Zhang","user":"qiushao","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":2,"organization":{"_id":"672d6e79bde669ec7e59ecc6","name":"hkustitd","fullname":"香港科技大学","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a757020246fad8d9c95265/PJZHQfGWiA_HNHI0S8vCz.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.07082.md"}">
On the Geometry of On-Policy Distillation
Abstract
On-policy distillation exhibits distinct parameter space dynamics characterized by relaxed off-principal updates and subspace locking, forming a unique geometric pattern separate from supervised fine-tuning and reinforcement learning with verifiable rewards.
On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.
Community
On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.07082 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.07082 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.07082 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.