We take a distributional view of the multi-stage distillation pipeline and organize the paper around two questions:</p>\n<p>1️⃣ For the intermediate initialization, what matters more — higher quality, or a better-matched distribution?<br>Our experiments give a clear (and slightly surprising) answer: when the initialization and DMD stages aim at the same distribution, even a weaker teacher or lower-quality data beats a stronger-teacher / better-data combo whose distributions are mismatched. Alignment beats raw quality. Motivated by this, we design a distribution-aware evaluation (teacher-normalized precision & coverage) that gives a far richer lens on intermediate distillation checkpoints than visual scores like VBench.</p>\n<p>2️⃣ Once the distributions are matched, what happens inside the distillation stage?<br>We track how the student distribution evolves and find that pure reverse-KL DMD contracts it in late training — so even as visual quality stays high, diversity quietly collapses. To counter this, we propose joint distillation, which balances the mode-covering and mode-seeking objectives: broad coverage preserved, quality still sharpened.</p>\n<p>Notably, with only a 1.3B DMD teacher, DistillAlign already surpasses baselines refined with a 14B teacher. We open-source our training data, models at every stage, and the full measured intermediate results — come give it a try! 🚀</p>\n","updatedAt":"2026-07-30T17:01:22.729Z","author":{"_id":"65464049e70ffa3c07f22e92","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EAYwKThe-BGeza-CmzIdx.jpeg","fullname":" Li Jiaxing","name":"LiJiaxing","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9120877385139465},"editors":["LiJiaxing"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EAYwKThe-BGeza-CmzIdx.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.26811","authors":[{"_id":"6a6b7561b2106777884ab196","name":"Jiaxing Li","hidden":false},{"_id":"6a6b7561b2106777884ab197","name":"Kai Zou","hidden":false},{"_id":"6a6b7561b2106777884ab198","name":"Cindy Zhou","hidden":false},{"_id":"6a6b7561b2106777884ab199","name":"Kaichen Huang","hidden":false},{"_id":"6a6b7561b2106777884ab19a","name":"Junyao Gao","hidden":false},{"_id":"6a6b7561b2106777884ab19b","name":"Zile Wang","hidden":false},{"_id":"6a6b7561b2106777884ab19c","name":"Yang Liu","hidden":false},{"_id":"6a6b7561b2106777884ab19d","name":"Bin Liu","hidden":false},{"_id":"6a6b7561b2106777884ab19e","name":"Bo An","hidden":false},{"_id":"6a6b7561b2106777884ab19f","name":"Yangguang Li","hidden":false}],"publishedAt":"2026-07-29T00:00:00.000Z","submittedOnDailyAt":"2026-07-30T00:00:00.000Z","title":"DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation","submittedOnDailyBy":{"_id":"65464049e70ffa3c07f22e92","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EAYwKThe-BGeza-CmzIdx.jpeg","isPro":true,"fullname":" Li Jiaxing","user":"LiJiaxing","type":"user","name":"LiJiaxing"},"summary":"Existing autoregressive video distillation methods commonly adopt a Distribution Matching Distillation (DMD)-based multi-stage pipeline. However, they typically decouple the initialization and DMD stages -- which then pursue different target distributions -- and judge the intermediate student mainly by visual scores such as VBench. In this paper, we revisit this design from a distributional perspective. Given the mode-seeking nature of the distribution matching loss, a good initialization should match the mode coverage of the target DMD teacher, rather than merely pursuing high quality. To analyze this, we introduce a distributional evaluation protocol that measures precision and coverage between student and teacher distributions in a shared latent space. It exposes differences hidden by visual scores: some initializations reach high precision but low coverage, leading to suboptimal refinement, while mode-covering ones preserve broader support. Furthermore, even when the target distributions are aligned, DMD's reverse-KL objective can still drive the student toward high-probability teacher regions in late training, reducing coverage and diversity. To address this, we propose joint distillation, which combines DMD's mode-seeking objective with a Consistency Distillation-based mode-covering constraint. Experiments show that our method improves generation quality, coverage, and diversity; notably, even with a Wan-1.3B DMD teacher, it outperforms baselines refined with Wan-14B, underscoring the importance of distributional alignment in autoregressive video distillation.","upvotes":0,"discussionId":"6a6b7561b2106777884ab1a0","projectPage":"https://lijiaxing0213.github.io/DistillAlign/","githubRepo":"https://github.com/LiJiaxing0213/DistillAlign","githubRepoAddedBy":"user","githubStars":48},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.26811.md","query":{}}">
DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation
Abstract
Existing autoregressive video distillation methods commonly adopt a Distribution Matching Distillation (DMD)-based multi-stage pipeline. However, they typically decouple the initialization and DMD stages -- which then pursue different target distributions -- and judge the intermediate student mainly by visual scores such as VBench. In this paper, we revisit this design from a distributional perspective. Given the mode-seeking nature of the distribution matching loss, a good initialization should match the mode coverage of the target DMD teacher, rather than merely pursuing high quality. To analyze this, we introduce a distributional evaluation protocol that measures precision and coverage between student and teacher distributions in a shared latent space. It exposes differences hidden by visual scores: some initializations reach high precision but low coverage, leading to suboptimal refinement, while mode-covering ones preserve broader support. Furthermore, even when the target distributions are aligned, DMD's reverse-KL objective can still drive the student toward high-probability teacher regions in late training, reducing coverage and diversity. To address this, we propose joint distillation, which combines DMD's mode-seeking objective with a Consistency Distillation-based mode-covering constraint. Experiments show that our method improves generation quality, coverage, and diversity; notably, even with a Wan-1.3B DMD teacher, it outperforms baselines refined with Wan-14B, underscoring the importance of distributional alignment in autoregressive video distillation.
Community
We take a distributional view of the multi-stage distillation pipeline and organize the paper around two questions:
1️⃣ For the intermediate initialization, what matters more — higher quality, or a better-matched distribution?
Our experiments give a clear (and slightly surprising) answer: when the initialization and DMD stages aim at the same distribution, even a weaker teacher or lower-quality data beats a stronger-teacher / better-data combo whose distributions are mismatched. Alignment beats raw quality. Motivated by this, we design a distribution-aware evaluation (teacher-normalized precision & coverage) that gives a far richer lens on intermediate distillation checkpoints than visual scores like VBench.
2️⃣ Once the distributions are matched, what happens inside the distillation stage?
We track how the student distribution evolves and find that pure reverse-KL DMD contracts it in late training — so even as visual quality stays high, diversity quietly collapses. To counter this, we propose joint distillation, which balances the mode-covering and mode-seeking objectives: broad coverage preserved, quality still sharpened.
Notably, with only a 1.3B DMD teacher, DistillAlign already surpasses baselines refined with a 14B teacher. We open-source our training data, models at every stage, and the full measured intermediate results — come give it a try! 🚀
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.26811 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.26811 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.26811 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.