The community is undergoing a macro-level paradigm shift from early modular assembly, i.e., late-fusion and grafted pipelines blind to raw sensory signals, toward born-native multimodal convergence, where multimodal understanding and generation fluidly coexist within unified transformer spaces.</p>\n<p>If you are trying to navigate the messy, fragmented design space of multimodal models, this paper delivers the community's first definitive, full-lifecycle roadmap.</p>\n","updatedAt":"2026-05-26T02:51:36.023Z","author":{"_id":"647ffa46d037515be116ad1f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/647ffa46d037515be116ad1f/pHlZrNqZcALf4jAvLJ9YB.jpeg","fullname":"HansonDong","name":"HansonDJN","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8526309728622437},"editors":["HansonDJN"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/647ffa46d037515be116ad1f/pHlZrNqZcALf4jAvLJ9YB.jpeg"],"reactions":[{"reaction":"🚀","users":["Junrulu"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.25343","authors":[{"_id":"6a150884b57a1823d5708a52","name":"Siyu An","hidden":false},{"_id":"6a150884b57a1823d5708a53","user":{"_id":"64ddbc3f1f2dad27e1a05ac1","avatarUrl":"/avatars/4fb2753b7998c8536bfd4780d3b10a6d.svg","isPro":false,"fullname":"Junrulu","user":"Junrulu","type":"user","name":"Junrulu"},"name":"Junru Lu","status":"claimed_verified","statusLastChangedAt":"2026-05-26T07:47:52.653Z","hidden":false},{"_id":"6a150884b57a1823d5708a54","name":"Junnan Dong","hidden":false},{"_id":"6a150884b57a1823d5708a55","name":"Qiufeng Wang","hidden":false},{"_id":"6a150884b57a1823d5708a56","name":"Yinghui Li","hidden":false},{"_id":"6a150884b57a1823d5708a57","name":"Weizhi Fei","hidden":false},{"_id":"6a150884b57a1823d5708a58","name":"Zichao Yu","hidden":false},{"_id":"6a150884b57a1823d5708a59","name":"Zheng Yuan","hidden":false},{"_id":"6a150884b57a1823d5708a5a","name":"Biao Liu","hidden":false},{"_id":"6a150884b57a1823d5708a5b","name":"Haopeng Wang","hidden":false},{"_id":"6a150884b57a1823d5708a5c","name":"Renzhao Liang","hidden":false},{"_id":"6a150884b57a1823d5708a5d","name":"Yixuan Yang","hidden":false},{"_id":"6a150884b57a1823d5708a5e","name":"Yunhang Shen","hidden":false},{"_id":"6a150884b57a1823d5708a5f","name":"Bo Ke","hidden":false},{"_id":"6a150884b57a1823d5708a60","name":"Keyu Chen","hidden":false},{"_id":"6a150884b57a1823d5708a61","name":"Linhao Luo","hidden":false},{"_id":"6a150884b57a1823d5708a62","name":"Difan Zou","hidden":false},{"_id":"6a150884b57a1823d5708a63","name":"Xiao Huang","hidden":false},{"_id":"6a150884b57a1823d5708a64","name":"Di Yin","hidden":false},{"_id":"6a150884b57a1823d5708a65","name":"Ruizhi Qiao","hidden":false},{"_id":"6a150884b57a1823d5708a66","user":{"_id":"647401e50da364bd0d002f2a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png","isPro":false,"fullname":"XING SUN","user":"tedsun","type":"user","name":"tedsun"},"name":"Xing Sun","status":"claimed_verified","statusLastChangedAt":"2026-05-26T07:47:49.647Z","hidden":false}],"publishedAt":"2026-05-25T00:00:00.000Z","submittedOnDailyAt":"2026-05-26T00:00:00.000Z","title":"Toward Native Multimodal Modeling: A Roadmap","submittedOnDailyBy":{"_id":"647ffa46d037515be116ad1f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/647ffa46d037515be116ad1f/pHlZrNqZcALf4jAvLJ9YB.jpeg","isPro":false,"fullname":"HansonDong","user":"HansonDJN","type":"user","name":"HansonDJN"},"summary":"Multimodal modeling represents a vital step from modality-agnostic reasoning toward world modeling. While early approaches predominantly rely on late-fusion that assembles encoders and frozen language backbones with output heads, recent efforts have shifted the paradigm toward native multimodal modeling (NMM) with the intrinsic integration of modalities for superior multimodal performance. Despite its potential, the design space of native architectures remains insufficiently defined. In this paper, we present the community with a formalized roadmap for this transition. Specifically, we formally define the architectural nativity, distinguishing mid-fusion and early-fusion from non-native paradigms. We further organize the existing native models through the lens of input-output duality into three categories: (i) Multi-to-Text for cross-modal comprehension with text-only output; (ii) Multi-to-Target for scenario-oriented generation, e.g., image, audio and video generation, and (iii) Multi-to-Multi for unified modeling with symmetric input-output. We deliver a comprehensive and industrial-grade investigation into the transition toward the definitive NMM framework, where understanding and generation seamlessly coexist within a unified transformer paradigm. We systematically unpack the end-to-end pipeline from industrial perspectives from architectural coordination, massive data curation, to full-stack training recipes, inference & deployment, and the comprehensive evaluation for truly native modeling.","upvotes":19,"discussionId":"6a150885b57a1823d5708a67","projectPage":"https://nmm-roadmap.github.io/","githubRepo":"https://github.com/NMM-Roadmap/Awesome-NMM-List","githubRepoAddedBy":"user","ai_summary":"Native multimodal modeling advances beyond traditional fusion approaches by integrating modalities inherently within a unified transformer framework, enabling seamless understanding and generation across diverse input-output configurations.","ai_keywords":["multimodal modeling","native multimodal modeling","late-fusion","mid-fusion","early-fusion","cross-modal comprehension","scenario-oriented generation","unified modeling","transformer paradigm","architectural nativity","input-output duality"],"githubStars":2,"organization":{"_id":"66543b6e420092799d2f625c","name":"tencent","fullname":"Tencent","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/Lp3m-XLpjQGwBItlvn69q.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64ddbc3f1f2dad27e1a05ac1","avatarUrl":"/avatars/4fb2753b7998c8536bfd4780d3b10a6d.svg","isPro":false,"fullname":"Junrulu","user":"Junrulu","type":"user"},{"_id":"647ffa46d037515be116ad1f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/647ffa46d037515be116ad1f/pHlZrNqZcALf4jAvLJ9YB.jpeg","isPro":false,"fullname":"HansonDong","user":"HansonDJN","type":"user"},{"_id":"6363ad21287b5ce02ed187ae","avatarUrl":"/avatars/9903cddc7330852fc6d8cf672cebcb56.svg","isPro":false,"fullname":"SiyuAn","user":"moonlake","type":"user"},{"_id":"63fc75f9b9db84750cea9c5c","avatarUrl":"/avatars/2c5bf9685e0cfc4b5785a4a86c34e0db.svg","isPro":false,"fullname":"DI YIN","user":"DIYIN","type":"user"},{"_id":"692e96fd7bf6e835a968d7c6","avatarUrl":"/avatars/c18a79cb53df74434f2d04ba145fe980.svg","isPro":false,"fullname":"Yinghui Li","user":"liyinghui1998","type":"user"},{"_id":"6562db314e8918182da42706","avatarUrl":"/avatars/b113bbbb496bf4dac254f0e840f08e10.svg","isPro":false,"fullname":"Jiafei Lyu","user":"dmux","type":"user"},{"_id":"6226f23d7551f131b4eba61d","avatarUrl":"/avatars/4b135465799760468999336cef39d768.svg","isPro":false,"fullname":"Yidong Wang","user":"qianlanwyd","type":"user"},{"_id":"640f3984c025ddf618978589","avatarUrl":"/avatars/f8b11defb4a115f1af8d68b6d1228571.svg","isPro":false,"fullname":"Qiufeng","user":"bruceqfw","type":"user"},{"_id":"63b6dbc8ccebeadccc888456","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1673396893898-63b6dbc8ccebeadccc888456.jpeg","isPro":false,"fullname":"Xin Zhang","user":"izhx","type":"user"},{"_id":"654c4f6d848e8c8b118e16e3","avatarUrl":"/avatars/b2323e95624e28c2578a3c5d73020c77.svg","isPro":true,"fullname":"Arnold Yang","user":"B3rrYang","type":"user"},{"_id":"6565e44b9bf6665f10016213","avatarUrl":"/avatars/0d7cf0e1b42cd116960cd030478446c5.svg","isPro":false,"fullname":"Zhuoshi Pan","user":"panzs19","type":"user"},{"_id":"63e9df3746574e63a2cc55c5","avatarUrl":"/avatars/3d27ad1ccfa51387e4b97d02e13deb41.svg","isPro":false,"fullname":"Lingfeng Qiao","user":"leafqiaoqiao","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"66543b6e420092799d2f625c","name":"tencent","fullname":"Tencent","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/Lp3m-XLpjQGwBItlvn69q.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.25343.md"}">
Toward Native Multimodal Modeling: A Roadmap
Authors: ,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Native multimodal modeling advances beyond traditional fusion approaches by integrating modalities inherently within a unified transformer framework, enabling seamless understanding and generation across diverse input-output configurations.
AI-generated summary
Multimodal modeling represents a vital step from modality-agnostic reasoning toward world modeling. While early approaches predominantly rely on late-fusion that assembles encoders and frozen language backbones with output heads, recent efforts have shifted the paradigm toward native multimodal modeling (NMM) with the intrinsic integration of modalities for superior multimodal performance. Despite its potential, the design space of native architectures remains insufficiently defined. In this paper, we present the community with a formalized roadmap for this transition. Specifically, we formally define the architectural nativity, distinguishing mid-fusion and early-fusion from non-native paradigms. We further organize the existing native models through the lens of input-output duality into three categories: (i) Multi-to-Text for cross-modal comprehension with text-only output; (ii) Multi-to-Target for scenario-oriented generation, e.g., image, audio and video generation, and (iii) Multi-to-Multi for unified modeling with symmetric input-output. We deliver a comprehensive and industrial-grade investigation into the transition toward the definitive NMM framework, where understanding and generation seamlessly coexist within a unified transformer paradigm. We systematically unpack the end-to-end pipeline from industrial perspectives from architectural coordination, massive data curation, to full-stack training recipes, inference & deployment, and the comprehensive evaluation for truly native modeling.
Community
The community is undergoing a macro-level paradigm shift from early modular assembly, i.e., late-fusion and grafted pipelines blind to raw sensory signals, toward born-native multimodal convergence, where multimodal understanding and generation fluidly coexist within unified transformer spaces.
If you are trying to navigate the messy, fragmented design space of multimodal models, this paper delivers the community's first definitive, full-lifecycle roadmap.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.25343 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.25343 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.25343 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.