Proposes a semantic-temporal world action model that combines DINOv3 future supervision with current-anchored history retrieval to improve robust manipulation under visual distribution shifts.</p>\n","updatedAt":"2026-08-05T04:40:07.862Z","author":{"_id":"686d185699645df570892710","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zjMBsWvDnRSxSHe0yJC7h.png","fullname":"wangmingxinthu","name":"wangmingxinthu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.666397750377655},"editors":["wangmingxinthu"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zjMBsWvDnRSxSHe0yJC7h.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.28993","authors":[{"_id":"6a72be0a1a375f948521c4b0","name":"Mingxin Wang","hidden":false},{"_id":"6a72be0a1a375f948521c4b1","name":"Bin Hu","hidden":false},{"_id":"6a72be0a1a375f948521c4b2","name":"Bin Qian","hidden":false},{"_id":"6a72be0a1a375f948521c4b3","name":"Kaitao Jiang","hidden":false},{"_id":"6a72be0a1a375f948521c4b4","name":"Haoning Wu","hidden":false},{"_id":"6a72be0a1a375f948521c4b5","name":"Feng Yan","hidden":false},{"_id":"6a72be0a1a375f948521c4b6","name":"Bowen Jing","hidden":false},{"_id":"6a72be0a1a375f948521c4b7","name":"Ruiyang Hao","hidden":false},{"_id":"6a72be0a1a375f948521c4b8","name":"Enyi Wang","hidden":false},{"_id":"6a72be0a1a375f948521c4b9","name":"Kangning Niu","hidden":false},{"_id":"6a72be0a1a375f948521c4ba","name":"Yandan Yang","hidden":false},{"_id":"6a72be0a1a375f948521c4bb","name":"Mu Xu","hidden":false},{"_id":"6a72be0a1a375f948521c4bc","name":"Yan Wang","hidden":false},{"_id":"6a72be0a1a375f948521c4bd","name":"Houde Liu","hidden":false},{"_id":"6a72be0a1a375f948521c4be","name":"Tianlun Li","hidden":false}],"publishedAt":"2026-07-31T00:00:00.000Z","submittedOnDailyAt":"2026-08-05T00:00:00.000Z","title":"ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts","submittedOnDailyBy":{"_id":"686d185699645df570892710","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zjMBsWvDnRSxSHe0yJC7h.png","isPro":false,"fullname":"wangmingxinthu","user":"wangmingxinthu","type":"user","name":"wangmingxinthu"},"summary":"World Action Models (WAMs) have emerged as a promising paradigm by jointly modeling robot actions and future visual dynamics. However, their reliance on pixel-generative future supervision can entangle action-relevant state transitions with task-irrelevant visual content, limiting robustness under visual distribution shifts. We identify Training-Distribution Hallucination, a recurring phenomenon in which futures conditioned on visually shifted observations hallucinate training-domain content rather than remain faithful to the current scene. A controlled frame-triplet diagnosis further shows that DINOv3 features remain more stable across visual shifts while better preserving task-state distinctions than Wan-VAE latents. Rather than correcting the predicted futures, we propose Semantic-Temporal WAM (ST-WAM) to improve action robustness by using DINOv3 as a shared semantic representation for future prediction and history retrieval while retaining fine-grained VAE dynamics. Its Dual-Space Future Experts (DSFE) jointly predict future VAE latents and DINO features, while Current-Anchored Intent Retrieval (CAIR) retrieves task-relevant evidence from recent DINO history under the current visual-language context. ST-WAM is trained end-to-end without additional embodied pretraining or task-specific annotations, and requires no explicit future generation at inference. It achieves 98.7% on LIBERO and 92.8% on RoboTwin 2.0; more importantly, compared with Fast-WAM, it improves zero-shot LIBERO-Plus performance by 21.3 percentage points and more than doubles real-world success under visual shifts from 25.8% to 61.5%. These results demonstrate that semantic-temporal modeling effectively complements pixel-generative dynamics for robust manipulation.","upvotes":1,"discussionId":"6a72be0a1a375f948521c4bf","projectPage":"https://thu-wangmx.github.io/st-wam/","githubRepo":"https://github.com/Thu-WangMX/ST-WAM-Semantic-Temporal-World-Action-Model","githubRepoAddedBy":"user","githubStars":5,"organization":{"_id":"628735cbc83a2d6ab8d14a66","name":"Tsinghua","fullname":"Tsinghua University","avatar":"https://www.gravatar.com/avatar/6c5c1441e3283e7543342e59277ea219?d=retro&size=100"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"686d185699645df570892710","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zjMBsWvDnRSxSHe0yJC7h.png","isPro":false,"fullname":"wangmingxinthu","user":"wangmingxinthu","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"628735cbc83a2d6ab8d14a66","name":"Tsinghua","fullname":"Tsinghua University","avatar":"https://www.gravatar.com/avatar/6c5c1441e3283e7543342e59277ea219?d=retro&size=100"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.28993.md","query":{}}">
ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts
Abstract
World Action Models (WAMs) have emerged as a promising paradigm by jointly modeling robot actions and future visual dynamics. However, their reliance on pixel-generative future supervision can entangle action-relevant state transitions with task-irrelevant visual content, limiting robustness under visual distribution shifts. We identify Training-Distribution Hallucination, a recurring phenomenon in which futures conditioned on visually shifted observations hallucinate training-domain content rather than remain faithful to the current scene. A controlled frame-triplet diagnosis further shows that DINOv3 features remain more stable across visual shifts while better preserving task-state distinctions than Wan-VAE latents. Rather than correcting the predicted futures, we propose Semantic-Temporal WAM (ST-WAM) to improve action robustness by using DINOv3 as a shared semantic representation for future prediction and history retrieval while retaining fine-grained VAE dynamics. Its Dual-Space Future Experts (DSFE) jointly predict future VAE latents and DINO features, while Current-Anchored Intent Retrieval (CAIR) retrieves task-relevant evidence from recent DINO history under the current visual-language context. ST-WAM is trained end-to-end without additional embodied pretraining or task-specific annotations, and requires no explicit future generation at inference. It achieves 98.7% on LIBERO and 92.8% on RoboTwin 2.0; more importantly, compared with Fast-WAM, it improves zero-shot LIBERO-Plus performance by 21.3 percentage points and more than doubles real-world success under visual shifts from 25.8% to 61.5%. These results demonstrate that semantic-temporal modeling effectively complements pixel-generative dynamics for robust manipulation.
Community
Proposes a semantic-temporal world action model that combines DINOv3 future supervision with current-anchored history retrieval to improve robust manipulation under visual distribution shifts.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.28993 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.28993 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.28993 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.