Hugging Face Daily Papers · July 21, 2026 · 30 min read

Distilled Reinforcement Learning for LLM Post-training

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

\n <a href=\"https://arxiv.org/abs/2607.17247\">\n <img src=\"https://img.shields.io/badge/arXiv-2607.17247-b31b1b.svg\" alt=\"arXiv\">\n </a>\n <a href=\"https://github.com/597358816/Distilled-RL\">\n <img src=\"https://img.shields.io/badge/Code-GitHub-black.svg\" alt=\"GitHub\">\n </a>\n\n\n\n\n\n\nStandard reinforcement learning relies on coarse-grained outcome rewards, while on-policy distillation usually encourages the student to imitate the teacher distribution unconditionally. Distilled RL instead uses the teacher to redistribute the policy-gradient signal at the token level, providing selective and fine-grained guidance while preserving reward-driven optimization.\n\n## Overview\n\n![distilled-RL-main](https://cdn-uploads.huggingface.co/production/uploads/67d139a2c37dc49c9b4427e1/oakRP-GudVK-lDj4z9YiI.png)\n\nDistilled RL consists of three components:\n\n1. **Reverse importance sampling**, which measures the teacher's relative preference for each student-generated token.\n2. **Negative sample reset**, which disables teacher reweighting on negative-advantage trajectories.\n3. **Sequence-level geometric normalization**, which removes sequence-level scale bias while preserving relative token preferences.\n\n## Method\n\nGiven a prompt $q$ and a response $o_i$ sampled from the old student policy, the standard policy ratio is\n\n```math\n$$\nr_{i,t}(\\theta)\n=\n\\frac{\n\\pi_{\\theta}(o_{i,t} \\mid q, o_{i,1:t-1})\n}{\n\\pi_{\\mathrm{old}}(o_{i,t} \\mid q, o_{i,1:t-1})\n}\n$$\n```\n\nThe response-level advantage is estimated using group-normalized rewards:\n\n```math\n$$\nA_i\n=\n\\frac{\nR_i - \\mathrm{mean}(\\{R_j\\}_{j=1}^{G})\n}{\n\\mathrm{std}(\\{R_j\\}_{j=1}^{G})\n}.\n$$\n```\n\n### Reverse Importance Sampling\n\nWe measure the teacher's relative preference for each student-generated token using\n\n```math\n$$\n\\rho_{i,t}\n=\n\\frac{\n\\pi_{\\mathrm{teacher}}(o_{i,t} \\mid q, o_{i,1:t-1})\n}{\n\\pi_{\\theta_{\\mathrm{old}}}(o_{i,t} \\mid q, o_{i,1:t-1})\n}.\n$$\n```\n\nTo prevent extreme teacher–student likelihood ratios, we apply symmetric clipping:\n\n```math\n$$\n\\bar{\\rho}_{i,t}\n=\n\\mathrm{clip}\n\\left(\n\\rho_{i,t},\n\\epsilon_{\\rho}^{-1},\n\\epsilon_{\\rho}\n\\right).\n$$\n```\n\n### Sequence-Level Geometric Normalization\n\nThe clipped ratios are normalized within each response:\n\n```math\n$$\n\\widetilde{\\rho}_{i,t}\n=\n\\frac{\n\\bar{\\rho}_{i,t}\n}{\n\\exp\n\\left(\n\\frac{1}{|o_i|}\n\\sum_{s=1}^{|o_i|}\n\\log \\bar{\\rho}_{i,s}\n\\right)\n}.\n$$\n```\n\nThe normalized ratios satisfy\n\n```math\n$$\n\\left(\n\\prod_{t=1}^{|o_i|}\n\\widetilde{\\rho}_{i,t}\n\\right)^{1/|o_i|}\n=\n1.\n$$\n```\n\nThis normalization removes the sequence-level mean shift in log importance ratios while preserving the teacher's relative preferences across tokens.\n\n### Negative Sample Reset\n\nTeacher guidance is applied only to positive-advantage responses:\n\n```math\n$$\nw_{i,t}\n=\n\\begin{cases}\n\\widetilde{\\rho}_{i,t}, & A_i > 0, \\\\\n1, & A_i \\leq 0.\n\\end{cases}\n$$\n```\n\nFor negative-advantage responses, the update reduces to the original RL objective.\n\n### Distilled RL Objective\n\nFor responses sampled from the old student policy, the final policy optimization objective is\n\n```math\n$$\n\\mathcal{J}_{\\mathrm{DistilledRL}}(\\theta)\n=\n\\mathbb{E}\n\\left[\n\\frac{1}{G}\n\\sum_{i=1}^{G}\n\\frac{1}{|o_i|}\n\\sum_{t=1}^{|o_i|}\n\\min\n\\left(\nr_{i,t}(\\theta) w_{i,t} A_i,\n\\hat{r}_{i,t}(\\theta) w_{i,t} A_i\n\\right)\n\\right],\n$$\n```\n\nwhere the clipped policy ratio is\n\n```math\n$$\n\\hat{r}_{i,t}(\\theta)\n=\n\\mathrm{clip}\n\\left(\nr_{i,t}(\\theta),\n1-\\epsilon_{\\mathrm{low}},\n1+\\epsilon_{\\mathrm{high}}\n\\right).\n$$\n```\n\nUnlike KL-based on-policy distillation, Distilled RL does not treat the teacher as an unconditional imitation target. Instead, the teacher selectively redistributes the reward-driven policy-gradient signal at the token level.\n\n## Main Results\n\nWe evaluate Distilled RL on three student models using Qwen3-8B-GRPO as the teacher. The table below reports the average Pass@1 over ten mathematical reasoning benchmarks.\n\n| Student Model | Base | OPD | RL | OPD+RL | Distilled RL |\n|---|---:|---:|---:|---:|---:|\n| DeepSeek-R1-Distill-Qwen-1.5B | 31.70 | 35.27 | 36.86 | 36.54 | **40.00** |\n| Qwen3-1.7B | 39.86 | 45.21 | 44.76 | 44.89 | **46.37** |\n| Qwen3-4B | 46.33 | 55.97 | 57.40 | 56.38 | **58.96** |\n\nDistilled RL consistently improves over standard RL, OPD, and their direct combination across different student scales and teacher–student settings.\n\n## Requirements\n\n### Software\n\nClone the repository:\n\n```bash\ngit clone https://github.com/597358816/Distilled-RL.git\ncd Distilled-RL\n```\n\nInstall the required dependencies:\n\n```bash\npip install torch==2.6.0 torchaudio==2.6.0 torchvision==0.21.0 vllm==0.8.3 transformers==4.51.2\npip install ray==2.48.0 tensordict==0.9.1 pydantic==2.11.7\npip install flash-attn\npip install -e .\npip install tensorboard\ncd examples\nbash XX.sh\n```\n\n\n","html":"<h1 class=\"relative group flex items-baseline\">\n\t<a id=\"distilled-reinforcement-learning-for-llm-post-training\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#distilled-reinforcement-learning-for-llm-post-training\" rel=\"nofollow\">\n\t\t<svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg>\n\t</a>\n\t\n\t\tDistilled Reinforcement Learning for LLM Post-Training\n\t\n</h1>\n\n <a href=\"https://arxiv.org/abs/2607.17247\" rel=\"nofollow\">\n <img src=\"https://img.shields.io/badge/arXiv-2607.17247-b31b1b.svg\" alt=\"arXiv\">\n </a>\n <a href=\"https://github.com/597358816/Distilled-RL\" rel=\"nofollow\">\n <img src=\"https://img.shields.io/badge/Code-GitHub-black.svg\" alt=\"GitHub\">\n </a>\n\n\n\n\n\n\nStandard reinforcement learning relies on coarse-grained outcome rewards, while on-policy distillation usually encourages the student to imitate the teacher distribution unconditionally. Distilled RL instead uses the teacher to redistribute the policy-gradient signal at the token level, providing selective and fine-grained guidance while preserving reward-driven optimization.\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"overview\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#overview\" rel=\"nofollow\">\n\t\t<svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg>\n\t</a>\n\t\n\t\tOverview\n\t\n</h2>\n<a href=\"https://cdn-uploads.huggingface.co/production/uploads/67d139a2c37dc49c9b4427e1/oakRP-GudVK-lDj4z9YiI.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/67d139a2c37dc49c9b4427e1/oakRP-GudVK-lDj4z9YiI.png\" alt=\"distilled-RL-main\"></a>\nDistilled RL consists of three components:\n<ol>\n<li>Reverse importance sampling, which measures the teacher's relative preference for each student-generated token.</li>\n<li>Negative sample reset, which disables teacher reweighting on negative-advantage trajectories.</li>\n<li>Sequence-level geometric normalization, which removes sequence-level scale bias while preserving relative token preferences.</li>\n</ol>\n<h2 class=\"relative group flex items-baseline\">\n\t<a class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#method\" rel=\"nofollow\">\n\t\t<svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg>\n\t</a>\n\t\n\t\tMethod\n\t\n</h2>\nGiven a prompt $q$ and a response $o_i$ sampled from the old student policy, the standard policy ratio is\n<pre><code class=\"language-math\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><mrow><msub><mi>r</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo stretchy=\"false\">(</mo><mi>θ</mi><mo stretchy=\"false\">)</mo><mo>=</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy=\"false\">(</mo><msub><mi>o</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo>∣</mo><mi>q</mi><mo separator=\"true\">,</mo><msub><mi>o</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mn>1</mn><mo>:</mo><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mo stretchy=\"false\">)</mo></mrow><mrow><msub><mi>π</mi><mrow><mi mathvariant=\"normal\">o</mi><mi mathvariant=\"normal\">l</mi><mi mathvariant=\"normal\">d</mi></mrow></msub><mo stretchy=\"false\">(</mo><msub><mi>o</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo>∣</mo><mi>q</mi><mo separator=\"true\">,</mo><msub><mi>o</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mn>1</mn><mo>:</mo><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mo stretchy=\"false\">)</mo></mrow></mfrac></mrow>\nr_{i,t}(\\theta)\n=\n\\frac{\n\\pi_{\\theta}(o_{i,t} \\mid q, o_{i,1:t-1})\n}{\n\\pi_{\\mathrm{old}}(o_{i,t} \\mid q, o_{i,1:t-1})\n}\n</math>ri,t(θ)=πold(oi,t∣q,oi,1:t−1)πθ(oi,t∣q,oi,1:t−1)\n</code></pre>\nThe response-level advantage is estimated using group-normalized rewards:\n<pre><code class=\"language-math\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><mrow><msub><mi>A</mi><mi>i</mi></msub><mo>=</mo><mfrac><mrow><msub><mi>R</mi><mi>i</mi></msub><mo>−</mo><mrow><mi mathvariant=\"normal\">m</mi><mi mathvariant=\"normal\">e</mi><mi mathvariant=\"normal\">a</mi><mi mathvariant=\"normal\">n</mi></mrow><mo stretchy=\"false\">(</mo><mo stretchy=\"false\">{</mo><msub><mi>R</mi><mi>j</mi></msub><msubsup><mo stretchy=\"false\">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo stretchy=\"false\">)</mo></mrow><mrow><mrow><mi mathvariant=\"normal\">s</mi><mi mathvariant=\"normal\">t</mi><mi mathvariant=\"normal\">d</mi></mrow><mo stretchy=\"false\">(</mo><mo stretchy=\"false\">{</mo><msub><mi>R</mi><mi>j</mi></msub><msubsup><mo stretchy=\"false\">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo stretchy=\"false\">)</mo></mrow></mfrac><mi mathvariant=\"normal\">.</mi></mrow>\nA_i\n=\n\\frac{\nR_i - \\mathrm{mean}(\\{R_j\\}_{j=1}^{G})\n}{\n\\mathrm{std}(\\{R_j\\}_{j=1}^{G})\n}.\n</math>Ai=std({Rj}j=1G)Ri−mean({Rj}j=1G).\n</code></pre>\n<h3 class=\"relative group flex items-baseline\">\n\t<a id=\"reverse-importance-sampling\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#reverse-importance-sampling\" rel=\"nofollow\">\n\t\t<svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg>\n\t</a>\n\t\n\t\tReverse Importance Sampling\n\t\n</h3>\nWe measure the teacher's relative preference for each student-generated token using\n<pre><code class=\"language-math\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><mrow><msub><mi>ρ</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo>=</mo><mfrac><mrow><msub><mi>π</mi><mrow><mi mathvariant=\"normal\">t</mi><mi mathvariant=\"normal\">e</mi><mi mathvariant=\"normal\">a</mi><mi mathvariant=\"normal\">c</mi><mi mathvariant=\"normal\">h</mi><mi mathvariant=\"normal\">e</mi><mi mathvariant=\"normal\">r</mi></mrow></msub><mo stretchy=\"false\">(</mo><msub><mi>o</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo>∣</mo><mi>q</mi><mo separator=\"true\">,</mo><msub><mi>o</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mn>1</mn><mo>:</mo><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mo stretchy=\"false\">)</mo></mrow><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi mathvariant=\"normal\">o</mi><mi mathvariant=\"normal\">l</mi><mi mathvariant=\"normal\">d</mi></mrow></msub></msub><mo stretchy=\"false\">(</mo><msub><mi>o</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo>∣</mo><mi>q</mi><mo separator=\"true\">,</mo><msub><mi>o</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mn>1</mn><mo>:</mo><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mo stretchy=\"false\">)</mo></mrow></mfrac><mi mathvariant=\"normal\">.</mi></mrow>\n\\rho_{i,t}\n=\n\\frac{\n\\pi_{\\mathrm{teacher}}(o_{i,t} \\mid q, o_{i,1:t-1})\n}{\n\\pi_{\\theta_{\\mathrm{old}}}(o_{i,t} \\mid q, o_{i,1:t-1})\n}.\n</math>ρi,t=πθold(oi,t∣q,oi,1:t−1)πteacher(oi,t∣q,oi,1:t−1).\n</code></pre>\nTo prevent extreme teacher–student likelihood ratios, we apply symmetric clipping:\n<pre><code class=\"language-math\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><mrow><msub><mover accent=\"true\"><mi>ρ</mi><mo>ˉ</mo></mover><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo>=</mo><mrow><mi mathvariant=\"normal\">c</mi><mi mathvariant=\"normal\">l</mi><mi mathvariant=\"normal\">i</mi><mi mathvariant=\"normal\">p</mi></mrow><mrow><mo fence=\"true\">(</mo><msub><mi>ρ</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo separator=\"true\">,</mo><msubsup><mi>ϵ</mi><mi>ρ</mi><mrow><mo>−</mo><mn>1</mn></mrow></msubsup><mo separator=\"true\">,</mo><msub><mi>ϵ</mi><mi>ρ</mi></msub><mo fence=\"true\">)</mo></mrow><mi mathvariant=\"normal\">.</mi></mrow>\n\\bar{\\rho}_{i,t}\n=\n\\mathrm{clip}\n\\left(\n\\rho_{i,t},\n\\epsilon_{\\rho}^{-1},\n\\epsilon_{\\rho}\n\\right).\n</math>ρˉi,t=clip(ρi,t,ϵρ−1,ϵρ).\n</code></pre>\n<h3 class=\"relative group flex items-baseline\">\n\t<a id=\"sequence-level-geometric-normalization\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#sequence-level-geometric-normalization\" rel=\"nofollow\">\n\t\t<svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg>\n\t</a>\n\t\n\t\tSequence-Level Geometric Normalization\n\t\n</h3>\nThe clipped ratios are normalized within each response:\n<pre><code class=\"language-math\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><mrow><msub><mover accent=\"true\"><mi>ρ</mi><mo stretchy=\"true\">~</mo></mover><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo>=</mo><mfrac><msub><mover accent=\"true\"><mi>ρ</mi><mo>ˉ</mo></mover><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mrow><mi>exp</mi><mo>⁡</mo><mrow><mo fence=\"true\">(</mo><mfrac><mn>1</mn><mrow><mi mathvariant=\"normal\">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant=\"normal\">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>s</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant=\"normal\">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant=\"normal\">∣</mi></mrow></munderover><mi>log</mi><mo>⁡</mo><msub><mover accent=\"true\"><mi>ρ</mi><mo>ˉ</mo></mover><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>s</mi></mrow></msub><mo fence=\"true\">)</mo></mrow></mrow></mfrac><mi mathvariant=\"normal\">.</mi></mrow>\n\\widetilde{\\rho}_{i,t}\n=\n\\frac{\n\\bar{\\rho}_{i,t}\n}{\n\\exp\n\\left(\n\\frac{1}{|o_i|}\n\\sum_{s=1}^{|o_i|}\n\\log \\bar{\\rho}_{i,s}\n\\right)\n}.\n</math>ρ<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100%\" height=\"0.26em\" viewBox=\"0 0 600 260\" preserveAspectRatio=\"none\"><path d=\"M200 55.538c-77 0-168 73.953-177 73.953-3 0-7\n-2.175-9-5.437L2 97c-1-2-2-4-2-6 0-4 2-7 5-9l20-12C116 12 171 0 207 0c86 0\n 114 68 191 68 78 0 168-68 177-68 4 0 7 2 9 5l12 19c1 2.175 2 4.35 2 6.525 0\n 4.35-2 7.613-5 9.788l-19 13.05c-92 63.077-116.937 75.308-183 76.128\n-68.267.847-113-73.952-191-73.952z\"></path></svg>i,t=exp(∣oi∣1∑s=1∣oi∣logρˉi,s)ρˉi,t.\n</code></pre>\nThe normalized ratios satisfy\n<pre><code class=\"language-math\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><mrow><msup><mrow><mo fence=\"true\">(</mo><munderover><mo>∏</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant=\"normal\">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant=\"normal\">∣</mi></mrow></munderover><msub><mover accent=\"true\"><mi>ρ</mi><mo stretchy=\"true\">~</mo></mover><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo fence=\"true\">)</mo></mrow><mrow><mn>1</mn><mi mathvariant=\"normal\">/</mi><mi mathvariant=\"normal\">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant=\"normal\">∣</mi></mrow></msup><mo>=</mo><mn>1.</mn></mrow>\n\\left(\n\\prod_{t=1}^{|o_i|}\n\\widetilde{\\rho}_{i,t}\n\\right)^{1/|o_i|}\n=\n1.\n</math><svg xmlns=\"http://www.w3.org/2000/svg\" width=\"0.875em\" height=\"3.600em\" viewBox=\"0 0 875 3600\"><path d=\"M863,9c0,-2,-2,-5,-6,-9c0,0,-17,0,-17,0c-12.7,0,-19.3,0.3,-20,1\nc-5.3,5.3,-10.3,11,-15,17c-242.7,294.7,-395.3,682,-458,1162c-21.3,163.3,-33.3,349,\n-36,557 l0,84c0.2,6,0,26,0,60c2,159.3,10,310.7,24,454c53.3,528,210,\n949.7,470,1265c4.7,6,9.7,11.7,15,17c0.7,0.7,7,1,19,1c0,0,18,0,18,0c4,-4,6,-7,6,-9\nc0,-2.7,-3.3,-8.7,-10,-18c-135.3,-192.7,-235.5,-414.3,-300.5,-665c-65,-250.7,-102.5,\n-544.7,-112.5,-882c-2,-104,-3,-167,-3,-189\nl0,-92c0,-162.7,5.7,-314,17,-454c20.7,-272,63.7,-513,129,-723c65.3,\n-210,155.3,-396.3,270,-559c6.7,-9.3,10,-15.3,10,-18z\"></path></svg>t=1∏∣oi∣ρ<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100%\" height=\"0.26em\" viewBox=\"0 0 600 260\" preserveAspectRatio=\"none\"><path d=\"M200 55.538c-77 0-168 73.953-177 73.953-3 0-7\n-2.175-9-5.437L2 97c-1-2-2-4-2-6 0-4 2-7 5-9l20-12C116 12 171 0 207 0c86 0\n 114 68 191 68 78 0 168-68 177-68 4 0 7 2 9 5l12 19c1 2.175 2 4.35 2 6.525 0\n 4.35-2 7.613-5 9.788l-19 13.05c-92 63.077-116.937 75.308-183 76.128\n-68.267.847-113-73.952-191-73.952z\"></path></svg>i,t<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"0.875em\" height=\"3.600em\" viewBox=\"0 0 875 3600\"><path d=\"M76,0c-16.7,0,-25,3,-25,9c0,2,2,6.3,6,13c21.3,28.7,42.3,60.3,\n63,95c96.7,156.7,172.8,332.5,228.5,527.5c55.7,195,92.8,416.5,111.5,664.5\nc11.3,139.3,17,290.7,17,454c0,28,1.7,43,3.3,45l0,9\nc-3,4,-3.3,16.7,-3.3,38c0,162,-5.7,313.7,-17,455c-18.7,248,-55.8,469.3,-111.5,664\nc-55.7,194.7,-131.8,370.3,-228.5,527c-20.7,34.7,-41.7,66.3,-63,95c-2,3.3,-4,7,-6,11\nc0,7.3,5.7,11,17,11c0,0,11,0,11,0c9.3,0,14.3,-0.3,15,-1c5.3,-5.3,10.3,-11,15,-17\nc242.7,-294.7,395.3,-681.7,458,-1161c21.3,-164.7,33.3,-350.7,36,-558\nl0,-144c-2,-159.3,-10,-310.7,-24,-454c-53.3,-528,-210,-949.7,\n-470,-1265c-4.7,-6,-9.7,-11.7,-15,-17c-0.7,-0.7,-6.7,-1,-18,-1z\"></path></svg>1/∣oi∣=1.\n</code></pre>\nThis normalization removes the sequence-level mean shift in log importance ratios while preserving the teacher's relative preferences across tokens.\n<h3 class=\"relative group flex items-baseline\">\n\t<a id=\"negative-sample-reset\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#negative-sample-reset\" rel=\"nofollow\">\n\t\t<svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg>\n\t</a>\n\t\n\t\tNegative Sample Reset\n\t\n</h3>\nTeacher guidance is applied only to positive-advantage responses:\n<pre><code class=\"language-math\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><mrow><msub><mi>w</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo>=</mo><mrow><mo fence=\"true\">{</mo><mtable rowspacing=\"0.36em\" columnalign=\"left left\" columnspacing=\"1em\"><mtr><mtd><mstyle scriptlevel=\"0\" displaystyle=\"false\"><mrow><msub><mover accent=\"true\"><mi>ρ</mi><mo stretchy=\"true\">~</mo></mover><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo separator=\"true\">,</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel=\"0\" displaystyle=\"false\"><mrow><msub><mi>A</mi><mi>i</mi></msub><mo>></mo><mn>0</mn><mo separator=\"true\">,</mo></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel=\"0\" displaystyle=\"false\"><mrow><mn>1</mn><mo separator=\"true\">,</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel=\"0\" displaystyle=\"false\"><mrow><msub><mi>A</mi><mi>i</mi></msub><mo>≤</mo><mn>0.</mn></mrow></mstyle></mtd></mtr></mtable></mrow></mrow>\nw_{i,t}\n=\n\\begin{cases}\n\\widetilde{\\rho}_{i,t}, & A_i > 0, \\\\\n1, & A_i \\leq 0.\n\\end{cases}\n</math>wi,t={ρ<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100%\" height=\"0.26em\" viewBox=\"0 0 600 260\" preserveAspectRatio=\"none\"><path d=\"M200 55.538c-77 0-168 73.953-177 73.953-3 0-7\n-2.175-9-5.437L2 97c-1-2-2-4-2-6 0-4 2-7 5-9l20-12C116 12 171 0 207 0c86 0\n 114 68 191 68 78 0 168-68 177-68 4 0 7 2 9 5l12 19c1 2.175 2 4.35 2 6.525 0\n 4.35-2 7.613-5 9.788l-19 13.05c-92 63.077-116.937 75.308-183 76.128\n-68.267.847-113-73.952-191-73.952z\"></path></svg>i,t,1,Ai>0,Ai≤0.\n</code></pre>\nFor negative-advantage responses, the update reduces to the original RL objective.\n<h3 class=\"relative group flex items-baseline\">\n\t<a id=\"distilled-rl-objective\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#distilled-rl-objective\" rel=\"nofollow\">\n\t\t<svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg>\n\t</a>\n\t\n\t\tDistilled RL Objective\n\t\n</h3>\nFor responses sampled from the old student policy, the final policy optimization objective is\n<pre><code class=\"language-math\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><mrow><msub><mi mathvariant=\"script\">J</mi><mrow><mi mathvariant=\"normal\">D</mi><mi mathvariant=\"normal\">i</mi><mi mathvariant=\"normal\">s</mi><mi mathvariant=\"normal\">t</mi><mi mathvariant=\"normal\">i</mi><mi mathvariant=\"normal\">l</mi><mi mathvariant=\"normal\">l</mi><mi mathvariant=\"normal\">e</mi><mi mathvariant=\"normal\">d</mi><mi mathvariant=\"normal\">R</mi><mi mathvariant=\"normal\">L</mi></mrow></msub><mo stretchy=\"false\">(</mo><mi>θ</mi><mo stretchy=\"false\">)</mo><mo>=</mo><mi mathvariant=\"double-struck\">E</mi><mrow><mo fence=\"true\">[</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><mfrac><mn>1</mn><mrow><mi mathvariant=\"normal\">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant=\"normal\">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant=\"normal\">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant=\"normal\">∣</mi></mrow></munderover><mi>min</mi><mo>⁡</mo><mrow><mo fence=\"true\">(</mo><msub><mi>r</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo stretchy=\"false\">(</mo><mi>θ</mi><mo stretchy=\"false\">)</mo><msub><mi>w</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><msub><mi>A</mi><mi>i</mi></msub><mo separator=\"true\">,</mo><msub><mover accent=\"true\"><mi>r</mi><mo>^</mo></mover><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo stretchy=\"false\">(</mo><mi>θ</mi><mo stretchy=\"false\">)</mo><msub><mi>w</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><msub><mi>A</mi><mi>i</mi></msub><mo fence=\"true\">)</mo></mrow><mo fence=\"true\">]</mo></mrow><mo separator=\"true\">,</mo></mrow>\n\\mathcal{J}_{\\mathrm{DistilledRL}}(\\theta)\n=\n\\mathbb{E}\n\\left[\n\\frac{1}{G}\n\\sum_{i=1}^{G}\n\\frac{1}{|o_i|}\n\\sum_{t=1}^{|o_i|}\n\\min\n\\left(\nr_{i,t}(\\theta) w_{i,t} A_i,\n\\hat{r}_{i,t}(\\theta) w_{i,t} A_i\n\\right)\n\\right],\n</math>JDistilledRL(θ)=E<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"0.667em\" height=\"3.600em\" viewBox=\"0 0 667 3600\"><path d=\"M403 1759 V84 H666 V0 H319 V1759 v0 v1759 h347 v-84\nH403z M403 1759 V0 H319 V1759 v0 v1759 h84z\"></path></svg>G1i=1∑G∣oi∣1t=1∑∣oi∣min(ri,t(θ)wi,tAi,r^i,t(θ)wi,tAi)<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"0.667em\" height=\"3.600em\" viewBox=\"0 0 667 3600\"><path d=\"M347 1759 V0 H0 V84 H263 V1759 v0 v1759 H0 v84 H347z\nM347 1759 V0 H263 V1759 v0 v1759 h84z\"></path></svg>,\n</code></pre>\nwhere the clipped policy ratio is\n<pre><code class=\"language-math\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><mrow><msub><mover accent=\"true\"><mi>r</mi><mo>^</mo></mover><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo stretchy=\"false\">(</mo><mi>θ</mi><mo stretchy=\"false\">)</mo><mo>=</mo><mrow><mi mathvariant=\"normal\">c</mi><mi mathvariant=\"normal\">l</mi><mi mathvariant=\"normal\">i</mi><mi mathvariant=\"normal\">p</mi></mrow><mrow><mo fence=\"true\">(</mo><msub><mi>r</mi><mrow><mi>i</mi><mo separator=\"true\">,</mo><mi>t</mi></mrow></msub><mo stretchy=\"false\">(</mo><mi>θ</mi><mo stretchy=\"false\">)</mo><mo separator=\"true\">,</mo><mn>1</mn><mo>−</mo><msub><mi>ϵ</mi><mrow><mi mathvariant=\"normal\">l</mi><mi mathvariant=\"normal\">o</mi><mi mathvariant=\"normal\">w</mi></mrow></msub><mo separator=\"true\">,</mo><mn>1</mn><mo>+</mo><msub><mi>ϵ</mi><mrow><mi mathvariant=\"normal\">h</mi><mi mathvariant=\"normal\">i</mi><mi mathvariant=\"normal\">g</mi><mi mathvariant=\"normal\">h</mi></mrow></msub><mo fence=\"true\">)</mo></mrow><mi mathvariant=\"normal\">.</mi></mrow>\n\\hat{r}_{i,t}(\\theta)\n=\n\\mathrm{clip}\n\\left(\nr_{i,t}(\\theta),\n1-\\epsilon_{\\mathrm{low}},\n1+\\epsilon_{\\mathrm{high}}\n\\right).\n</math>r^i,t(θ)=clip(ri,t(θ),1−ϵlow,1+ϵhigh).\n</code></pre>\nUnlike KL-based on-policy distillation, Distilled RL does not treat the teacher as an unconditional imitation target. Instead, the teacher selectively redistributes the reward-driven policy-gradient signal at the token level.\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"main-results\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#main-results\" rel=\"nofollow\">\n\t\t<svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg>\n\t</a>\n\t\n\t\tMain Results\n\t\n</h2>\nWe evaluate Distilled RL on three student models using Qwen3-8B-GRPO as the teacher. The table below reports the average Pass@1 over ten mathematical reasoning benchmarks.\n<div class=\"max-w-full overflow-auto\">\n\t<table>\n\t\t<thead><tr>\n<th>Student Model</th>\n<th align=\"right\">Base</th>\n<th align=\"right\">OPD</th>\n<th align=\"right\">RL</th>\n<th align=\"right\">OPD+RL</th>\n<th align=\"right\">Distilled RL</th>\n</tr>\n\n\t\t</thead><tbody><tr>\n<td>DeepSeek-R1-Distill-Qwen-1.5B</td>\n<td align=\"right\">31.70</td>\n<td align=\"right\">35.27</td>\n<td align=\"right\">36.86</td>\n<td align=\"right\">36.54</td>\n<td align=\"right\">40.00</td>\n</tr>\n<tr>\n<td>Qwen3-1.7B</td>\n<td align=\"right\">39.86</td>\n<td align=\"right\">45.21</td>\n<td align=\"right\">44.76</td>\n<td align=\"right\">44.89</td>\n<td align=\"right\">46.37</td>\n</tr>\n<tr>\n<td>Qwen3-4B</td>\n<td align=\"right\">46.33</td>\n<td align=\"right\">55.97</td>\n<td align=\"right\">57.40</td>\n<td align=\"right\">56.38</td>\n<td align=\"right\">58.96</td>\n</tr>\n</tbody>\n\t</table>\n</div>\nDistilled RL consistently improves over standard RL, OPD, and their direct combination across different student scales and teacher–student settings.\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"requirements\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#requirements\" rel=\"nofollow\">\n\t\t<svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg>\n\t</a>\n\t\n\t\tRequirements\n\t\n</h2>\n<h3 class=\"relative group flex items-baseline\">\n\t<a id=\"software\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#software\" rel=\"nofollow\">\n\t\t<svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg>\n\t</a>\n\t\n\t\tSoftware\n\t\n</h3>\nClone the repository:\n<pre><code class=\"language-bash\">git clone https://github.com/597358816/Distilled-RL.git\ncd Distilled-RL\n</code></pre>\nInstall the required dependencies:\n<pre><code class=\"language-bash\">pip install torch==2.6.0 torchaudio==2.6.0 torchvision==0.21.0 vllm==0.8.3 transformers==4.51.2\npip install ray==2.48.0 tensordict==0.9.1 pydantic==2.11.7\npip install flash-attn\npip install -e .\npip install tensorboard\ncd examples\nbash XX.sh\n</code></pre>\n","updatedAt":"2026-07-21T03:58:05.870Z","author":{"_id":"67d139a2c37dc49c9b4427e1","avatarUrl":"/avatars/bc1279ddb7b1e4cd32d1803dbd130f98.svg","fullname":"Chen Wang","name":"wc597358816","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":3,"identifiedLanguage":{"language":"en","probability":0.65157550573349},"editors":["wc597358816"],"editorAvatarUrls":["/avatars/bc1279ddb7b1e4cd32d1803dbd130f98.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.17247","authors":[{"_id":"6a5eed504fe5d1d13e84abb2","name":"Chen Wang","hidden":false},{"_id":"6a5eed504fe5d1d13e84abb3","name":"Zhaochun Li","hidden":false},{"_id":"6a5eed504fe5d1d13e84abb4","name":"Jionghao Bai","hidden":false},{"_id":"6a5eed504fe5d1d13e84abb5","name":"Yining Zhang","hidden":false},{"_id":"6a5eed504fe5d1d13e84abb6","name":"Hexuan Deng","hidden":false},{"_id":"6a5eed504fe5d1d13e84abb7","name":"Ge Lan","hidden":false},{"_id":"6a5eed504fe5d1d13e84abb8","name":"Yue Wang","hidden":false}],"publishedAt":"2026-07-19T00:00:00.000Z","submittedOnDailyAt":"2026-07-21T00:00:00.000Z","title":"Distilled Reinforcement Learning for LLM Post-training","submittedOnDailyBy":{"_id":"67d139a2c37dc49c9b4427e1","avatarUrl":"/avatars/bc1279ddb7b1e4cd32d1803dbd130f98.svg","isPro":false,"fullname":"Chen Wang","user":"wc597358816","type":"user","name":"wc597358816"},"summary":"Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.","upvotes":4,"discussionId":"6a5eed504fe5d1d13e84abb9","projectPage":"https://arxiv.org/abs/2607.17247","githubRepo":"https://github.com/597358816/Distilled-RL","githubRepoAddedBy":"user","githubStars":0,"organization":{"_id":"68896d3a716ee5bfb1428441","name":"ZGCA","fullname":"Zhongguancun Academy","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6854c3ab09a3ba7d16243875/aZ3tp3lZk1yQoXDwSklye.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67d139a2c37dc49c9b4427e1","avatarUrl":"/avatars/bc1279ddb7b1e4cd32d1803dbd130f98.svg","isPro":false,"fullname":"Chen Wang","user":"wc597358816","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"},{"_id":"65c06c6469429d85dc5d72d8","avatarUrl":"/avatars/0e431ed06c5392715e162777d8070075.svg","isPro":false,"fullname":"daeyunkim","user":"senciva","type":"user"},{"_id":"61af81009f77f7b669578f95","avatarUrl":"/avatars/fb50773ac49948940eb231834ee6f2fd.svg","isPro":false,"fullname":"rotem israeli","user":"irotem98","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"68896d3a716ee5bfb1428441","name":"ZGCA","fullname":"Zhongguancun Academy","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6854c3ab09a3ba7d16243875/aZ3tp3lZk1yQoXDwSklye.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.17247.md","query":{}}">

Papers

arxiv:2607.17247

Distilled Reinforcement Learning for LLM Post-training

Published on Jul 19

· Submitted by

Chen Wang on Jul 21

Zhongguancun Academy

Upvote

Authors:

Abstract

Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.

View arXiv page View PDF Project page GitHub 0 Add to collection

Community

wc597358816

Paper submitter about 4 hours ago edited about 4 hours ago

Distilled Reinforcement Learning for LLM Post-Training

Standard reinforcement learning relies on coarse-grained outcome rewards, while on-policy distillation usually encourages the student to imitate the teacher distribution unconditionally. Distilled RL instead uses the teacher to redistribute the policy-gradient signal at the token level, providing selective and fine-grained guidance while preserving reward-driven optimization.

Overview

Distilled RL consists of three components:

Reverse importance sampling, which measures the teacher's relative preference for each student-generated token.
Negative sample reset, which disables teacher reweighting on negative-advantage trajectories.
Sequence-level geometric normalization, which removes sequence-level scale bias while preserving relative token preferences.

Method

Given a prompt $q$ and a response $o_i$ sampled from the old student policy, the standard policy ratio is

ri,t(θ)=πold(oi,t∣q,oi,1:t−1)πθ(oi,t∣q,oi,1:t−1)

The response-level advantage is estimated using group-normalized rewards:

Ai=std({Rj}j=1G)Ri−mean({Rj}j=1G).

Reverse Importance Sampling

We measure the teacher's relative preference for each student-generated token using

ρi,t=πθold(oi,t∣q,oi,1:t−1)πteacher(oi,t∣q,oi,1:t−1).

To prevent extreme teacher–student likelihood ratios, we apply symmetric clipping:

ρˉi,t=clip(ρi,t,ϵρ−1,ϵρ).

Sequence-Level Geometric Normalization

The clipped ratios are normalized within each response:

ρi,t=exp(∣oi∣1∑s=1∣oi∣logρˉi,s)ρˉi,t.

The normalized ratios satisfy

t=1∏∣oi∣ρi,t1/∣oi∣=1.

This normalization removes the sequence-level mean shift in log importance ratios while preserving the teacher's relative preferences across tokens.

Negative Sample Reset

Teacher guidance is applied only to positive-advantage responses:

wi,t={ρi,t,1,Ai>0,Ai≤0.

For negative-advantage responses, the update reduces to the original RL objective.

Distilled RL Objective

For responses sampled from the old student policy, the final policy optimization objective is

JDistilledRL(θ)=EG1i=1∑G∣oi∣1t=1∑∣oi∣min(ri,t(θ)wi,tAi,r^i,t(θ)wi,tAi),

where the clipped policy ratio is

r^i,t(θ)=clip(ri,t(θ),1−ϵlow,1+ϵhigh).

Unlike KL-based on-policy distillation, Distilled RL does not treat the teacher as an unconditional imitation target. Instead, the teacher selectively redistributes the reward-driven policy-gradient signal at the token level.

Main Results

We evaluate Distilled RL on three student models using Qwen3-8B-GRPO as the teacher. The table below reports the average Pass@1 over ten mathematical reasoning benchmarks.

Student Model	Base	OPD	RL	OPD+RL	Distilled RL
DeepSeek-R1-Distill-Qwen-1.5B	31.70	35.27	36.86	36.54	40.00
Qwen3-1.7B	39.86	45.21	44.76	44.89	46.37
Qwen3-4B	46.33	55.97	57.40	56.38	58.96

Distilled RL consistently improves over standard RL, OPD, and their direct combination across different student scales and teacher–student settings.

Requirements

Software

Clone the repository:

git clone https://github.com/597358816/Distilled-RL.git
cd Distilled-RL

Install the required dependencies:

pip install torch==2.6.0 torchaudio==2.6.0 torchvision==0.21.0 vllm==0.8.3 transformers==4.51.2
pip install ray==2.48.0 tensordict==0.9.1 pydantic==2.11.7
pip install flash-attn
pip install -e .
pip install tensorboard
cd examples
bash XX.sh

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Upvote

Get this paper in your agent:

hf papers read 2607.17247

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper

No model linking this paper

Cite arxiv.org/abs/2607.17247 in a model README.md to link it from this page.

Datasets citing this paper

No dataset linking this paper

Cite arxiv.org/abs/2607.17247 in a dataset README.md to link it from this page.

Spaces citing this paper

No Space linking this paper

Cite arxiv.org/abs/2607.17247 in a Space README.md to link it from this page.

Collections including this paper

No Collection including this paper

Add this paper to a collection to link it from this page.

Discussion (0)

No comments yet. Sign in and be the first to say something.

Distilled Reinforcement Learning for LLM Post-training

Abstract

Community

Distilled Reinforcement Learning for LLM Post-Training

Overview

Method

Reverse Importance Sampling

Sequence-Level Geometric Normalization

Negative Sample Reset

Distilled RL Objective

Main Results

Requirements

Software

Models citing this paper

Datasets citing this paper

Spaces citing this paper

Collections including this paper

Discussion (0)

More from Hugging Face Daily Papers