0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.","html":"<p>Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.</p>\n","updatedAt":"2026-06-01T02:07:28.509Z","author":{"_id":"6419fe794577db917f6d17fc","avatarUrl":"/avatars/ed10371a0082c5e02d32b76217200207.svg","fullname":"zhipeng cai","name":"zcai","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9132455587387085},"editors":["zcai"],"editorAvatarUrls":["/avatars/ed10371a0082c5e02d32b76217200207.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.30561","authors":[{"_id":"6a1ce909808ddbc3c7d433f6","name":"Zhipeng Cai","hidden":false},{"_id":"6a1ce909808ddbc3c7d433f7","name":"Zhuang Liu","hidden":false},{"_id":"6a1ce909808ddbc3c7d433f8","name":"Yunyang Xiong","hidden":false},{"_id":"6a1ce909808ddbc3c7d433f9","name":"Zechun Liu","hidden":false},{"_id":"6a1ce909808ddbc3c7d433fa","name":"Vikas Chandra","hidden":false},{"_id":"6a1ce909808ddbc3c7d433fb","name":"Yangyang Shi","hidden":false}],"publishedAt":"2026-05-28T00:00:00.000Z","submittedOnDailyAt":"2026-06-01T00:00:00.000Z","title":"VLM3: Vision Language Models Are Native 3D Learners","submittedOnDailyBy":{"_id":"6419fe794577db917f6d17fc","avatarUrl":"/avatars/ed10371a0082c5e02d32b76217200207.svg","isPro":false,"fullname":"zhipeng cai","user":"zcai","type":"user","name":"zcai"},"summary":"Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.","upvotes":9,"discussionId":"6a1ce90a808ddbc3c7d433fc","githubRepo":"https://github.com/facebookresearch/VLM3","githubRepoAddedBy":"user","ai_summary":"Vision Language Models can be adapted for 3D understanding tasks through simple architectural modifications and text-based training, achieving performance comparable to specialized vision models without requiring complex designs or extensive data augmentation.","ai_keywords":["Vision Language Models","3D understanding","focal length unification","text-based pixel reference","data mixture","scaling","depth estimation","pixel correspondence","camera pose estimation","object-level 3D understanding"],"githubStars":1,"organization":{"_id":"664b1a7ccf5fd472d7d5fa1a","name":"facebook-research","fullname":"AI at Meta","avatar":"https://www.gravatar.com/avatar/7700baf1456a0ce14ca5fe9c5f27c32a?d=retro&size=100"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6419fe794577db917f6d17fc","avatarUrl":"/avatars/ed10371a0082c5e02d32b76217200207.svg","isPro":false,"fullname":"zhipeng cai","user":"zcai","type":"user"},{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"692fd35e1dbd99f126854b8d","avatarUrl":"/avatars/8e1c10584a347c88cd3109f46736a25a.svg","isPro":false,"fullname":"Yangyang Shi","user":"shiyang1983","type":"user"},{"_id":"65304b62e7535baecd85d080","avatarUrl":"/avatars/6e546c7d1414bd92c5a7c8d8c404de92.svg","isPro":false,"fullname":"Yunyang Xiong","user":"yunyangx","type":"user"},{"_id":"6407e5294edf9f5c4fd32228","avatarUrl":"/avatars/8e2d55460e9fe9c426eb552baf4b2cb0.svg","isPro":false,"fullname":"Stoney Kang","user":"sikang99","type":"user"},{"_id":"6747ede3a9c72aebe1322382","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/inILqQ05sESbYLdsEldJ_.png","isPro":false,"fullname":"Tong He","user":"tonghe90","type":"user"},{"_id":"672f5f86359d27c87963553d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-hTmvLgCD22VOVWP7Wq3L.png","isPro":false,"fullname":"Timex Peachtree","user":"TimexPeachtree","type":"user"},{"_id":"68b6973cdd7f21b758972704","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/JpeCJCu3gnRci-FLZVDTG.png","isPro":false,"fullname":"gibsonlee","user":"gibsonlee","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"664b1a7ccf5fd472d7d5fa1a","name":"facebook-research","fullname":"AI at Meta","avatar":"https://www.gravatar.com/avatar/7700baf1456a0ce14ca5fe9c5f27c32a?d=retro&size=100"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.30561.md"}">
VLM3: Vision Language Models Are Native 3D Learners
Abstract
Vision Language Models can be adapted for 3D understanding tasks through simple architectural modifications and text-based training, achieving performance comparable to specialized vision models without requiring complex designs or extensive data augmentation.
AI-generated summary
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.
Community
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.30561 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.30561 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.30561 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.