We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs). Rather than pretraining from scratch, prior work replaces the causal attention in ARLMs with bidirectional attention and then trains the resulting model using a DLM objective. However, these approaches incur two distribution shifts. First, transitioning from a next-token prediction objective to a DLM objective can discard knowledge acquired by the ARLM during training. Second, standard DLMs suffer from a train-inference mismatch, as the training loss is defined on randomly masked sequences rather than the trajectories encountered at inference produced by confidence-based decoding. To address both challenges, we introduce an On-Policy Diffusion Language Model (OPDLM) in which On-Policy Distillation (OPD) is employed for ARLM-to-DLM transformation. Specifically, OPDLM is trained via self-OPD, where the student, an ARLM with bidirectional attention, generates its own trajectories, and the teacher, the original frozen ARLM, distills its knowledge by providing target logits on these trajectories. By training directly in an on-policy manner, OPDLM eliminates the train-inference mismatch in DLMs, while distillation from the original model enhances knowledge retention from the ARLM. Empirical results demonstrate that OPDLM requires 15x to 7,000x fewer training tokens with strong performance across a wide variety of tasks. OPDLM avoids the prohibitive cost of DLM pretraining and positions DLM transformation as a form of ARLM post-training.</p>\n","updatedAt":"2026-06-08T17:12:30.517Z","author":{"_id":"647d3c5c1f878439e202967b","avatarUrl":"/avatars/b2ad4a4fc4f53f7bcb173a5adf6049ff.svg","fullname":"Shubham Parashar","name":"shubhamprshr","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9199231863021851},"editors":["shubhamprshr"],"editorAvatarUrls":["/avatars/b2ad4a4fc4f53f7bcb173a5adf6049ff.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.06712","authors":[{"_id":"6a26f7616dde1c5ef75bcced","name":"Xingyu Su","hidden":false},{"_id":"6a26f7616dde1c5ef75bccee","name":"Jacob Helwig","hidden":false},{"_id":"6a26f7616dde1c5ef75bccef","name":"Shubham Parashar","hidden":false},{"_id":"6a26f7616dde1c5ef75bccf0","name":"Atharv Chagi","hidden":false},{"_id":"6a26f7616dde1c5ef75bccf1","name":"Lakshmi Jotsna","hidden":false},{"_id":"6a26f7616dde1c5ef75bccf2","name":"Degui Zhi","hidden":false},{"_id":"6a26f7616dde1c5ef75bccf3","name":"James Caverlee","hidden":false},{"_id":"6a26f7616dde1c5ef75bccf4","name":"Dileep Kalathil","hidden":false},{"_id":"6a26f7616dde1c5ef75bccf5","name":"Shuiwang Ji","hidden":false}],"publishedAt":"2026-06-04T00:00:00.000Z","submittedOnDailyAt":"2026-06-08T00:00:00.000Z","title":"Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation","submittedOnDailyBy":{"_id":"647d3c5c1f878439e202967b","avatarUrl":"/avatars/b2ad4a4fc4f53f7bcb173a5adf6049ff.svg","isPro":false,"fullname":"Shubham Parashar","user":"shubhamprshr","type":"user","name":"shubhamprshr"},"summary":"We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs). Rather than pretraining from scratch, prior work replaces the causal attention in ARLMs with bidirectional attention and then trains the resulting model using a DLM objective. However, these approaches incur two distribution shifts. First, transitioning from a next-token prediction objective to a DLM objective can discard knowledge acquired by the ARLM during training. Second, standard DLMs suffer from a train-inference mismatch, as the training loss is defined on randomly masked sequences rather than the trajectories encountered at inference produced by confidence-based decoding. To address both challenges, we introduce an On-Policy Diffusion Language Model (OPDLM) in which On-Policy Distillation (OPD) is employed for ARLM-to-DLM transformation. Specifically, OPDLM is trained via self-OPD, where the student, an ARLM with bidirectional attention, generates its own trajectories, and the teacher, the original frozen ARLM, distills its knowledge by providing target logits on these trajectories. By training directly in an on-policy manner, OPDLM eliminates the train-inference mismatch in DLMs, while distillation from the original model enhances knowledge retention from the ARLM. Empirical results demonstrate that OPDLM requires 15x to 7,000x fewer training tokens with strong performance across a wide variety of tasks. OPDLM avoids the prohibitive cost of DLM pretraining and positions DLM transformation as a form of ARLM post-training.","upvotes":1,"discussionId":"6a26f7616dde1c5ef75bccf6","projectPage":"https://opdlm.vercel.app/","githubRepo":"https://github.com/divelab/OPDLM","githubRepoAddedBy":"user","ai_summary":"Autoregressive language models are transformed into diffusion language models through on-policy distillation that eliminates train-inference mismatch and reduces training token requirements.","ai_keywords":["autoregressive models","diffusion language models","causal attention","bidirectional attention","train-inference mismatch","confidence-based decoding","on-policy distillation","self-OPD","knowledge distillation","token efficiency"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":2,"organization":{"_id":"67f181921838f78b68baa0b6","name":"divelab","fullname":"DIVE Lab, Texas A&M University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/677ec4bf992798cc2e218eae/tS3s1ErcYx1-wenBz_mf9.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"647d3c5c1f878439e202967b","avatarUrl":"/avatars/b2ad4a4fc4f53f7bcb173a5adf6049ff.svg","isPro":false,"fullname":"Shubham Parashar","user":"shubhamprshr","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"67f181921838f78b68baa0b6","name":"divelab","fullname":"DIVE Lab, Texas A&M University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/677ec4bf992798cc2e218eae/tS3s1ErcYx1-wenBz_mf9.jpeg"}}">
Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
Abstract
Autoregressive language models are transformed into diffusion language models through on-policy distillation that eliminates train-inference mismatch and reduces training token requirements.
We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs). Rather than pretraining from scratch, prior work replaces the causal attention in ARLMs with bidirectional attention and then trains the resulting model using a DLM objective. However, these approaches incur two distribution shifts. First, transitioning from a next-token prediction objective to a DLM objective can discard knowledge acquired by the ARLM during training. Second, standard DLMs suffer from a train-inference mismatch, as the training loss is defined on randomly masked sequences rather than the trajectories encountered at inference produced by confidence-based decoding. To address both challenges, we introduce an On-Policy Diffusion Language Model (OPDLM) in which On-Policy Distillation (OPD) is employed for ARLM-to-DLM transformation. Specifically, OPDLM is trained via self-OPD, where the student, an ARLM with bidirectional attention, generates its own trajectories, and the teacher, the original frozen ARLM, distills its knowledge by providing target logits on these trajectories. By training directly in an on-policy manner, OPDLM eliminates the train-inference mismatch in DLMs, while distillation from the original model enhances knowledge retention from the ARLM. Empirical results demonstrate that OPDLM requires 15x to 7,000x fewer training tokens with strong performance across a wide variety of tasks. OPDLM avoids the prohibitive cost of DLM pretraining and positions DLM transformation as a form of ARLM post-training.
Community
We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs). Rather than pretraining from scratch, prior work replaces the causal attention in ARLMs with bidirectional attention and then trains the resulting model using a DLM objective. However, these approaches incur two distribution shifts. First, transitioning from a next-token prediction objective to a DLM objective can discard knowledge acquired by the ARLM during training. Second, standard DLMs suffer from a train-inference mismatch, as the training loss is defined on randomly masked sequences rather than the trajectories encountered at inference produced by confidence-based decoding. To address both challenges, we introduce an On-Policy Diffusion Language Model (OPDLM) in which On-Policy Distillation (OPD) is employed for ARLM-to-DLM transformation. Specifically, OPDLM is trained via self-OPD, where the student, an ARLM with bidirectional attention, generates its own trajectories, and the teacher, the original frozen ARLM, distills its knowledge by providing target logits on these trajectories. By training directly in an on-policy manner, OPDLM eliminates the train-inference mismatch in DLMs, while distillation from the original model enhances knowledge retention from the ARLM. Empirical results demonstrate that OPDLM requires 15x to 7,000x fewer training tokens with strong performance across a wide variety of tasks. OPDLM avoids the prohibitive cost of DLM pretraining and positions DLM transformation as a form of ARLM post-training.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.06712 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.