Project page: <a href=\"https://yuecheong.github.io/SmartMage/\" rel=\"nofollow\">https://yuecheong.github.io/SmartMage/</a></p>\n","updatedAt":"2026-08-07T02:57:48.080Z","author":{"_id":"646c77911ee398a4e9404b8b","avatarUrl":"/avatars/05d1ea421dd4f3e2fd47cbe99fc52933.svg","fullname":"Yunqiu Xu","name":"Yunqiu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.27113664150238037},"editors":["Yunqiu"],"editorAvatarUrls":["/avatars/05d1ea421dd4f3e2fd47cbe99fc52933.svg"],"reactions":[{"reaction":"👍","users":["kunli-cs"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.05137","authors":[{"_id":"6a75494ce1228e04b32381cb","name":"Yue Zhang","hidden":false},{"_id":"6a75494ce1228e04b32381cc","name":"Yingzhao Jian","hidden":false},{"_id":"6a75494ce1228e04b32381cd","name":"Yunqiu Xu","hidden":false},{"_id":"6a75494ce1228e04b32381ce","name":"Xiaoxiao Sun","hidden":false},{"_id":"6a75494ce1228e04b32381cf","name":"Hehe Fan","hidden":false}],"publishedAt":"2026-08-05T00:00:00.000Z","submittedOnDailyAt":"2026-08-07T00:00:00.000Z","title":"SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding","submittedOnDailyBy":{"_id":"646c77911ee398a4e9404b8b","avatarUrl":"/avatars/05d1ea421dd4f3e2fd47cbe99fc52933.svg","isPro":false,"fullname":"Yunqiu Xu","user":"Yunqiu","type":"user","name":"Yunqiu"},"summary":"Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous information from multiple modalities, including visual and geometric cues. However, the relevance of these modalities often varies across queries. Existing Multimodal Large Language Models (MLLMs) typically rely on fixed modality combinations, overlooking query-dependent modality needs. Such a rigid design can introduce semantic noise from irrelevant modalities while underutilizing more informative ones, leading to wasted computation and diluted reasoning. To address these challenges, this paper proposes SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. Specifically, SmartMage incorporates: (1) a Semantic-guided Modality Adaptive RouTng (SMART) module that selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality; and (2) a Modality-Aware Gating Expert (MAGE) module that leverages modality priors to guide expert activation, fostering adaptive specialization in multimodal reasoning. Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks, and attains competitive results on RGB-only video understanding benchmarks. In our diagnostic benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness. Project page: https://yuecheong.github.io/SmartMage/.","upvotes":11,"discussionId":"6a75494ce1228e04b32381d0","projectPage":"https://yuecheong.github.io/SmartMage/"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"646c77911ee398a4e9404b8b","avatarUrl":"/avatars/05d1ea421dd4f3e2fd47cbe99fc52933.svg","isPro":false,"fullname":"Yunqiu Xu","user":"Yunqiu","type":"user"},{"_id":"69d4a975e4b32a4662faefdc","avatarUrl":"/avatars/e37c182499e7564f564d7757452cbd98.svg","isPro":false,"fullname":"yue cheong","user":"CheongYue-icloud","type":"user"},{"_id":"69cd0fed27bbc8efb93c7008","avatarUrl":"/avatars/f62b0624e64379ae85f7a1aaa11a9915.svg","isPro":false,"fullname":"John qonse","user":"xieerbi","type":"user"},{"_id":"6640c647acae6bb179eedff5","avatarUrl":"/avatars/bcaafaaa1d4b4c241d72a886401772e3.svg","isPro":false,"fullname":"Yuetong Liu","user":"YuetongLiu","type":"user"},{"_id":"63983d65b85ac406e0bd7b8a","avatarUrl":"/avatars/92a5f2b8b80c919f6948eb67a413488a.svg","isPro":false,"fullname":"Michael Chen","user":"howtoosee","type":"user"},{"_id":"65c4eb7cd1dcbd30d86febec","avatarUrl":"/avatars/001c8f02e8ce794b2c21883628b2da72.svg","isPro":false,"fullname":"free-bit","user":"free-bit","type":"user"},{"_id":"69b116035e5605678c754787","avatarUrl":"/avatars/9e8d7b89fbb74f8ce186d2d52e399753.svg","isPro":false,"fullname":"as","user":"WZLWLM0913","type":"user"},{"_id":"66481769691370727c2f5349","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66481769691370727c2f5349/_y4GUqN2nBiEUY0CuS7ZH.jpeg","isPro":true,"fullname":"Kun Li","user":"kunli-cs","type":"user"},{"_id":"67b82b97701171142716c63c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/3nnH8ryAm8BFwV0F17I2S.png","isPro":false,"fullname":"Yi_Zhou","user":"YiZhou123","type":"user"},{"_id":"66d7cfa4bae0e60d009235f3","avatarUrl":"/avatars/a1be185f0e791909b7b4ee494d8fa423.svg","isPro":false,"fullname":"Junhao Xiao","user":"XiaoJH","type":"user"},{"_id":"6a75a5e23963527d4a2a6dd2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a75a5e23963527d4a2a6dd2/GSsbQzP3MGLOIH08lVHTT.jpeg","isPro":false,"fullname":"Yue Jiang","user":"Jiayyyyy","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.05137.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2608.05137 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.05137 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2608.05137 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.