We introduce Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout. Autoregressive (AR) video diffusion models have shown great potential in real-time video generation. Recent methods distill pretrained bidirectional video diffusion models into causal AR students through Distribution Matching Distillation (DMD), but the generated videos often suffer from over-saturation and over-smoothing issues, resulting in limited visual quality and realism. The key contributing factor is the mode-seeking behavior of the reverse KL objective in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution. To address this, we propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR diffusion distillation. Such perturbations encourage the student rollouts to explore more regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner tokens act as denoising guidance for other noisier tokens, improving the intermediate rollout predictions and reducing error accumulation. Extensive experiments demonstrate that our method improves multiple AR video diffusion distillation methods with higher visual quality efficiently, without incorporating real video data or additional post-training stages.</p>\n","updatedAt":"2026-09-09T04:58:24.584Z","author":{"_id":"67f87bc19d597ac661a75b68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67f87bc19d597ac661a75b68/ARLLbu1CJ5mCQu6ptyfmG.jpeg","fullname":"Zhuoran Zhao","name":"Alicezrzhao","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8695119619369507},"editors":["Alicezrzhao"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/67f87bc19d597ac661a75b68/ARLLbu1CJ5mCQu6ptyfmG.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.09123","authors":[{"_id":"6aa0e617d0174964227bedf4","user":{"_id":"67f87bc19d597ac661a75b68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67f87bc19d597ac661a75b68/ARLLbu1CJ5mCQu6ptyfmG.jpeg","isPro":false,"fullname":"Zhuoran Zhao","user":"Alicezrzhao","type":"user","name":"Alicezrzhao"},"name":"Zhuoran Zhao","status":"claimed_verified","statusLastChangedAt":"2026-09-09T08:45:04.735Z","hidden":false},{"_id":"6aa0e617d0174964227bedf5","name":"Shengju Qian","hidden":false},{"_id":"6aa0e617d0174964227bedf6","name":"Tongtong Liang","hidden":false},{"_id":"6aa0e617d0174964227bedf7","user":{"_id":"6428fd124fe87caede856311","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5OrNPwZkxu3Dm1IInCxML.jpeg","isPro":false,"fullname":"Xianghao Kong","user":"refkxh","type":"user","name":"refkxh"},"name":"Xianghao Kong","status":"claimed_verified","statusLastChangedAt":"2026-09-09T08:45:04.718Z","hidden":false},{"_id":"6aa0e617d0174964227bedf8","name":"Songchun Zhang","hidden":false},{"_id":"6aa0e617d0174964227bedf9","name":"Junchao Huang","hidden":false},{"_id":"6aa0e617d0174964227bedfa","user":{"_id":"623461fccd8a0462e55b3666","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1647600114080-noauth.jpeg","isPro":true,"fullname":"Guian Fang","user":"Enderfga","type":"user","name":"Enderfga"},"name":"Guian Fang","status":"claimed_verified","statusLastChangedAt":"2026-09-09T08:45:04.726Z","hidden":false},{"_id":"6aa0e617d0174964227bedfb","name":"Xin Wang","hidden":false},{"_id":"6aa0e617d0174964227bedfc","name":"Pan Hui","hidden":false},{"_id":"6aa0e617d0174964227bedfd","name":"Anyi Rao","hidden":false}],"publishedAt":"2026-09-08T00:00:00.000Z","submittedOnDailyAt":"2026-09-09T00:00:00.000Z","title":"Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout","submittedOnDailyBy":{"_id":"67f87bc19d597ac661a75b68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67f87bc19d597ac661a75b68/ARLLbu1CJ5mCQu6ptyfmG.jpeg","isPro":false,"fullname":"Zhuoran Zhao","user":"Alicezrzhao","type":"user","name":"Alicezrzhao"},"summary":"Autoregressive (AR) video diffusion models have shown great potential in real-time video generation. Recent methods distill pretrained bidirectional video diffusion models into causal AR students through Distribution Matching Distillation (DMD), but the generated videos often suffer from over-saturation and over-smoothing issues, resulting in limited visual quality and realism. The key contributing factor is the mode-seeking behavior of the reverse KL objective in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution. To address this, we propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR diffusion distillation. Such perturbations encourage the student rollouts to explore more regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner tokens act as denoising guidance for other noisier tokens, improving the intermediate rollout predictions and reducing error accumulation. Extensive experiments demonstrate that our method improves multiple AR video diffusion distillation methods with higher visual quality efficiently, without incorporating real video data or additional post-training stages.","upvotes":42,"discussionId":"6aa0e617d0174964227bedfe","projectPage":"https://alicezrzhao.github.io/mask_forcing/","githubRepo":"https://github.com/delaprada/Mask-Forcing","githubRepoAddedBy":"user","ai_summary":"Mask Forcing mitigates mode collapse in distilled autoregressive video diffusion by injecting masked cleaner signals during self-rollout, improving visual quality without extra training data.","ai_keywords":["autoregressive video diffusion","Distribution Matching Distillation","reverse KL","mode collapse","Mask Forcing","Dual-Noise Masking Rollout","self-rollout","denoising guidance"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":11},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67f87bc19d597ac661a75b68","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67f87bc19d597ac661a75b68/ARLLbu1CJ5mCQu6ptyfmG.jpeg","isPro":false,"fullname":"Zhuoran Zhao","user":"Alicezrzhao","type":"user"},{"_id":"64b6abe17f3490ffe72071d0","avatarUrl":"/avatars/9d1b7ec4dd6d3e07c255e63c314dbed5.svg","isPro":false,"fullname":"Marcus kk","user":"marcuskwan","type":"user"},{"_id":"66877d72bec843c681fb4169","avatarUrl":"/avatars/02165c97f8d97cf04a268d9ea74e6188.svg","isPro":false,"fullname":"chen houyuan","user":"houyuanchen","type":"user"},{"_id":"6428fd124fe87caede856311","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5OrNPwZkxu3Dm1IInCxML.jpeg","isPro":false,"fullname":"Xianghao Kong","user":"refkxh","type":"user"},{"_id":"662d968eac05b4f7c280d6f9","avatarUrl":"/avatars/0fb03561fff425fddfa48e1ebde6325b.svg","isPro":false,"fullname":"YaoYang Liu","user":"LazySheeep","type":"user"},{"_id":"6539f922eeb6c5f65cc5f486","avatarUrl":"/avatars/9882c34c1a9c8dc415cf495c12b29fa3.svg","isPro":false,"fullname":"Zeyu Zhang","user":"JohnnyZ666","type":"user"},{"_id":"65214c46f6ceb915cc790275","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65214c46f6ceb915cc790275/Jb-iNYkDc9g5nODi0stGE.png","isPro":false,"fullname":"Yihua Du","user":"Duyh","type":"user"},{"_id":"6411c801e872ae3fb1e2c96e","avatarUrl":"/avatars/f8898dc13d700e545eedbbfab1c18353.svg","isPro":true,"fullname":"Franklin","user":"Franklinzhang","type":"user"},{"_id":"64d1efef4a204a4d125fd4fc","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64d1efef4a204a4d125fd4fc/mOjMzPUkAR-SmzYwofazX.jpeg","isPro":false,"fullname":"Jiehui Huang","user":"JackAILab","type":"user"},{"_id":"68961bcd83f79bb28f788da1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/oqGDqZ5OMd_9LImvz2BJD.png","isPro":false,"fullname":"Junchao-cs","user":"junchaoh-cs","type":"user"},{"_id":"64049ae20ab5e22719f35103","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1678023295407-noauth.jpeg","isPro":false,"fullname":"Dongyu Yan","user":"StarYDY","type":"user"},{"_id":"674aa9af9494dd9106006c27","avatarUrl":"/avatars/2f04bb009983ec0ade738aa7941cf6dc.svg","isPro":false,"fullname":"Zixin Zhang","user":"zhangzixin02","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.09123.md","query":{}}">
Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout
Abstract
Mask Forcing mitigates mode collapse in distilled autoregressive video diffusion by injecting masked cleaner signals during self-rollout, improving visual quality without extra training data.
Autoregressive (AR) video diffusion models have shown great potential in real-time video generation. Recent methods distill pretrained bidirectional video diffusion models into causal AR students through Distribution Matching Distillation (DMD), but the generated videos often suffer from over-saturation and over-smoothing issues, resulting in limited visual quality and realism. The key contributing factor is the mode-seeking behavior of the reverse KL objective in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution. To address this, we propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR diffusion distillation. Such perturbations encourage the student rollouts to explore more regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner tokens act as denoising guidance for other noisier tokens, improving the intermediate rollout predictions and reducing error accumulation. Extensive experiments demonstrate that our method improves multiple AR video diffusion distillation methods with higher visual quality efficiently, without incorporating real video data or additional post-training stages.
Community
We introduce Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout. Autoregressive (AR) video diffusion models have shown great potential in real-time video generation. Recent methods distill pretrained bidirectional video diffusion models into causal AR students through Distribution Matching Distillation (DMD), but the generated videos often suffer from over-saturation and over-smoothing issues, resulting in limited visual quality and realism. The key contributing factor is the mode-seeking behavior of the reverse KL objective in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution. To address this, we propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR diffusion distillation. Such perturbations encourage the student rollouts to explore more regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner tokens act as denoising guidance for other noisier tokens, improving the intermediate rollout predictions and reducing error accumulation. Extensive experiments demonstrate that our method improves multiple AR video diffusion distillation methods with higher visual quality efficiently, without incorporating real video data or additional post-training stages.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2609.09123 in a model README.md to link it from this page.
Cite arxiv.org/abs/2609.09123 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2609.09123 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.