Hugging Face Daily Papers · September 1, 2026 · 4 min read

DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

github：<a href=\"https://github.com/AMAP-ML/DreamX-Creator\" rel=\"nofollow\">https://github.com/AMAP-ML/DreamX-Creator</a>\n","updatedAt":"2026-09-01T04:01:12.970Z","author":{"_id":"67d116c47be76de1a40873ca","avatarUrl":"/avatars/d33a689b4a95709e50458a7163e0691d.svg","fullname":"AMAP-ML","name":"AMAP-ML","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.3623913824558258},"editors":["AMAP-ML"],"editorAvatarUrls":["/avatars/d33a689b4a95709e50458a7163e0691d.svg"],"reactions":[],"isReport":false}},{"id":"6a964e861d2975bd20caa1e2","author":{"_id":"67d116c47be76de1a40873ca","avatarUrl":"/avatars/d33a689b4a95709e50458a7163e0691d.svg","fullname":"AMAP-ML","name":"AMAP-ML","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false},"createdAt":"2026-09-01T04:03:18.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"\nhttps://cdn-uploads.huggingface.co/production/uploads/67d116c47be76de1a40873ca/eV55oM4Z4c_koWwN88DPb.qt\n","html":"<video src=\"https://cdn-uploads.huggingface.co/production/uploads/67d116c47be76de1a40873ca/eV55oM4Z4c_koWwN88DPb.qt\" controls=\"\" class=\"max-w-full!\"></video>\n","updatedAt":"2026-09-01T04:03:18.697Z","author":{"_id":"67d116c47be76de1a40873ca","avatarUrl":"/avatars/d33a689b4a95709e50458a7163e0691d.svg","fullname":"AMAP-ML","name":"AMAP-ML","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.3560323417186737},"editors":["AMAP-ML"],"editorAvatarUrls":["/avatars/d33a689b4a95709e50458a7163e0691d.svg"],"reactions":[],"isReport":false}},{"id":"6a96571f4e2541b828b299f2","author":{"_id":"648c6537aeff9347218f49f2","avatarUrl":"/avatars/1891855926eec77f91a389755998212f.svg","fullname":"Jiachen Lei","name":"jiachenlei","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false},"createdAt":"2026-09-01T04:39:59.000Z","type":"comment","data":{"edited":true,"hidden":true,"hiddenBy":"","latest":{"raw":"This comment has been hidden","html":"This comment has been hidden","updatedAt":"2026-09-01T04:42:58.299Z","author":{"_id":"648c6537aeff9347218f49f2","avatarUrl":"/avatars/1891855926eec77f91a389755998212f.svg","fullname":"Jiachen Lei","name":"jiachenlei","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"editors":[],"editorAvatarUrls":[],"reactions":[]}},{"id":"6a965a185a0c80051e9f916b","author":{"_id":"648c6537aeff9347218f49f2","avatarUrl":"/avatars/1891855926eec77f91a389755998212f.svg","fullname":"Jiachen Lei","name":"jiachenlei","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false},"createdAt":"2026-09-01T04:52:40.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"@AdinaY Hi, could you please help us add a teaser video to our paper? The video is included in the comments above. Thank you!","html":"<a href=\"/AdinaY\">@AdinaY</a> Hi, could you please help us add a teaser video to our paper? The video is included in the comments above. Thank you!\n","updatedAt":"2026-09-01T04:58:39.980Z","author":{"_id":"648c6537aeff9347218f49f2","avatarUrl":"/avatars/1891855926eec77f91a389755998212f.svg","fullname":"Jiachen Lei","name":"jiachenlei","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.949942946434021},"editors":["jiachenlei"],"editorAvatarUrls":["/avatars/1891855926eec77f91a389755998212f.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.31106","authors":[{"_id":"6a96475acd6ebc484732eced","name":"Jiashu Zhu","hidden":false},{"_id":"6a96475acd6ebc484732ecee","name":"Yanhao Zheng","hidden":false},{"_id":"6a96475acd6ebc484732ecef","name":"Ruitian Tian","hidden":false},{"_id":"6a96475acd6ebc484732ecf0","name":"Rujing Dang","hidden":false},{"_id":"6a96475acd6ebc484732ecf1","name":"Shen Zhang","hidden":false},{"_id":"6a96475acd6ebc484732ecf2","name":"Bingze Song","hidden":false},{"_id":"6a96475acd6ebc484732ecf3","name":"Jiachen Lei","hidden":false},{"_id":"6a96475acd6ebc484732ecf4","name":"Ruimin Lin","hidden":false},{"_id":"6a96475acd6ebc484732ecf5","name":"Jiahong Wu","hidden":false},{"_id":"6a96475acd6ebc484732ecf6","name":"Xiangxiang Chu","hidden":false}],"publishedAt":"2026-08-31T00:00:00.000Z","submittedOnDailyAt":"2026-09-01T00:00:00.000Z","title":"DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution","submittedOnDailyBy":{"_id":"648c6537aeff9347218f49f2","avatarUrl":"/avatars/1891855926eec77f91a389755998212f.svg","isPro":false,"fullname":"Jiachen Lei","user":"jiachenlei","type":"user","name":"jiachenlei"},"summary":"Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator. Conditioned on a first frame and a text prompt, the generator jointly denoises modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning further post-trains the generator with Modality-Aware Multimodal Feedback that routes video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.","upvotes":64,"discussionId":"6a96475bcd6ebc484732ecf7","githubRepo":"https://github.com/AMAP-ML/DreamX-Creator","githubRepoAddedBy":"user","ai_summary":"A compact 7B native joint audio-video generator uses cross-modal attention, progressive joint training, reinforcement learning with multimodal feedback, and an autoregressive 2K refinement pipeline to produce synchronized high-resolution outputs.","ai_keywords":["Gated Cross-Modal Attention","token- and head-wise output gates","Audio-Video Data System","Progressive Joint Training","High-Quality Finetuning","Audio-Video Reinforcement Learning","Modality-Aware Multimodal Feedback","Autoregressive 1-Step 2K Refinement","bidirectional multi-step teacher","autoregressive multi-step refiner"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":37,"organization":{"_id":"67d11771890254196d3174e5","name":"GD-ML","fullname":"AMAP-ML","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67d116c47be76de1a40873ca/s5ukAx9E36ZZIKvbpBRi4.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67d116c47be76de1a40873ca","avatarUrl":"/avatars/d33a689b4a95709e50458a7163e0691d.svg","isPro":false,"fullname":"AMAP-ML","user":"AMAP-ML","type":"user"},{"_id":"689d4a717fb1c6267bb59acc","avatarUrl":"/avatars/13b5117a0108e40cbffa8114c112cfad.svg","isPro":false,"fullname":"peter","user":"peterlrm","type":"user"},{"_id":"66d255e3947594430c723ff6","avatarUrl":"/avatars/c56e4792332a01bf34085a75ee64916e.svg","isPro":false,"fullname":"xiaochonglinghu","user":"xiaochonglinghu","type":"user"},{"_id":"66f3800367e707182cfabc92","avatarUrl":"/avatars/005c9ab9d7c78761c282c85a1258456d.svg","isPro":false,"fullname":"Haopeng Zhang","user":"espen123","type":"user"},{"_id":"650844d1773ceaa8d55cd325","avatarUrl":"/avatars/385923e6feb4fe58a81c8d5ff8d44831.svg","isPro":false,"fullname":"Bingze Song","user":"Flow1113","type":"user"},{"_id":"66ab82416fe3807667acf0a9","avatarUrl":"/avatars/1808f38c425e185a99186e9f60df5e2d.svg","isPro":false,"fullname":"LucasLee","user":"yunyouke","type":"user"},{"_id":"663bbb61ec1aafe3d6c05558","avatarUrl":"/avatars/b2f593d0ae0adbaad9a7a99b490e3a2b.svg","isPro":false,"fullname":"Ziyu Ma","user":"poiuytrewq123","type":"user"},{"_id":"6a6827a027e380e4530eca51","avatarUrl":"/avatars/e1730cca6ea88c1e508a2c9668fdc746.svg","isPro":false,"fullname":"jun20260728","user":"jun20260728","type":"user"},{"_id":"64d1dc5273174cecdffc97d3","avatarUrl":"/avatars/6564e6b68fee9673f75b6366adf39a3b.svg","isPro":false,"fullname":"Wang Yong","user":"seashell11","type":"user"},{"_id":"673c09d251d8d86ed0e4b343","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/9p0jpsSw_HYINDHsKasDW.png","isPro":false,"fullname":"guo","user":"sigma28","type":"user"},{"_id":"6465e8abe03f25ce60f5799f","avatarUrl":"/avatars/21b2b4b2688e59bb1dfca2591cdb2d23.svg","isPro":false,"fullname":"renda Li","user":"SSSAMMMM","type":"user"},{"_id":"645e0282412d9e9cd044c764","avatarUrl":"/avatars/1eebe4e89e151611ea96c38483549bd0.svg","isPro":false,"fullname":"gao","user":"bingjie","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"67d11771890254196d3174e5","name":"GD-ML","fullname":"AMAP-ML","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67d116c47be76de1a40873ca/s5ukAx9E36ZZIKvbpBRi4.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.31106.md","query":{}}">

Papers

arxiv:2608.31106

DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution

Published on Aug 31

· Submitted by

Jiachen Lei on Sep 1

AMAP-ML

Upvote

Authors:

Abstract

A compact 7B native joint audio-video generator uses cross-modal attention, progressive joint training, reinforcement learning with multimodal feedback, and an autoregressive 2K refinement pipeline to produce synchronized high-resolution outputs.

Generated by thinkingmachines/Inkling-Small

Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator. Conditioned on a first frame and a text prompt, the generator jointly denoises modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning further post-trains the generator with Modality-Aware Multimodal Feedback that routes video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.