Spatial intelligence unfolds through a perception-action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke's core knowledge systems. Agents must decide what abilities to deploy - perception, locomotion, and manipulation - and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close.</p>\n","updatedAt":"2026-05-20T04:21:53.037Z","author":{"_id":"6431b64df76c34519e93d1ba","avatarUrl":"/avatars/ea577762b6b4798f87a7a3f1d53d082c.svg","fullname":"Yining Hong","name":"evelynhong","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9013887047767639},"editors":["evelynhong"],"editorAvatarUrls":["/avatars/ea577762b6b4798f87a7a3f1d53d082c.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.18746","authors":[{"_id":"6a0d368f65eb30f20d962d1f","name":"Yining Hong","hidden":false},{"_id":"6a0d368f65eb30f20d962d20","name":"Jiageng Liu","hidden":false},{"_id":"6a0d368f65eb30f20d962d21","name":"Han Yin","hidden":false},{"_id":"6a0d368f65eb30f20d962d22","name":"Manling Li","hidden":false},{"_id":"6a0d368f65eb30f20d962d23","name":"Leonidas Guibas","hidden":false},{"_id":"6a0d368f65eb30f20d962d24","name":"Li Fei-Fei","hidden":false},{"_id":"6a0d368f65eb30f20d962d25","name":"Jiajun Wu","hidden":false},{"_id":"6a0d368f65eb30f20d962d26","name":"Yejin Choi","hidden":false}],"publishedAt":"2026-05-18T00:00:00.000Z","submittedOnDailyAt":"2026-05-20T00:00:00.000Z","title":"ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop","submittedOnDailyBy":{"_id":"6431b64df76c34519e93d1ba","avatarUrl":"/avatars/ea577762b6b4798f87a7a3f1d53d082c.svg","isPro":false,"fullname":"Yining Hong","user":"evelynhong","type":"user","name":"evelynhong"},"summary":"Spatial intelligence unfolds through a perception-action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke's core knowledge systems. Agents must decide what abilities to deploy - perception, locomotion, and manipulation - and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close.","upvotes":3,"discussionId":"6a0d369065eb30f20d962d27","projectPage":"https://esi-bench.github.io/","githubRepo":"https://github.com/ESI-Bench/ESI-Bench","githubRepoAddedBy":"user","ai_summary":"Embodied spatial intelligence requires active perception-action loops where agents strategically explore environments to uncover hidden spatial structures, with performance limited by action selection rather than perception capabilities.","ai_keywords":["embodied spatial intelligence","perception-action loop","active exploration","3D grounding","spatial intelligence","embodied agents","action blindness","multi-view imaging","spatial reasoning","core knowledge systems"],"githubStars":11},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6431b64df76c34519e93d1ba","avatarUrl":"/avatars/ea577762b6b4798f87a7a3f1d53d082c.svg","isPro":false,"fullname":"Yining Hong","user":"evelynhong","type":"user"},{"_id":"6569a78844ce94a7017770dd","avatarUrl":"/avatars/e457adf2b42594858b844fbf60a08354.svg","isPro":false,"fullname":"Jiaqi Gu","user":"gujiaqivadin","type":"user"},{"_id":"679185119afe88fb031405e1","avatarUrl":"/avatars/aac8d1a818bfa9ee09cf982cf1d724b3.svg","isPro":false,"fullname":"Lily","user":"chenyingli","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.18746.md"}">
ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop
Abstract
Embodied spatial intelligence requires active perception-action loops where agents strategically explore environments to uncover hidden spatial structures, with performance limited by action selection rather than perception capabilities.
AI-generated summary
Spatial intelligence unfolds through a perception-action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke's core knowledge systems. Agents must decide what abilities to deploy - perception, locomotion, and manipulation - and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close.
Community
Spatial intelligence unfolds through a perception-action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke's core knowledge systems. Agents must decide what abilities to deploy - perception, locomotion, and manipulation - and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.18746 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.18746 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.18746 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.