We’re excited to share <strong>PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control</strong>.</p>\n<p>Many robot tasks require remembering information that is no longer visible, such as a briefly shown target, an earlier instruction, or a demonstration observed at the beginning of an episode. While pretrained MLLMs already have strong long-context reasoning capabilities, most vision-language-action models do not directly use this context as memory for control.</p>\n<p>PonderPounce connects two pretrained systems:</p>\n<ul>\n<li><strong>Ponder</strong>, a System 2 MLLM, maintains observations, demonstrations, and prior cognition in its native causal context.</li>\n<li><strong>Pounce</strong>, a fast System 1 action model, controls the robot using the current observation together with Ponder’s latest continuous cognition token and its age.</li>\n</ul>\n<p>Ponder and Pounce operate on decoupled clocks and are trained jointly end-to-end, without a purpose-built memory module or separate bridge pretraining.</p>\n<p>On <strong>RoboMME</strong>, PonderPounce achieves <strong>60.83%</strong> average success with 1× training data and <strong>75.54%</strong> with 9× data. This corresponds to <strong>1.37×</strong> and <strong>1.31×</strong> the performance of the strongest non-oracle baseline, respectively.</p>\n<p>🌐 Project page: <a href=\"https://worv-ai.github.io/ponderpounce/\" rel=\"nofollow\">https://worv-ai.github.io/ponderpounce/</a><br>📄 Paper: <a href=\"https://arxiv.org/abs/2608.24115\" rel=\"nofollow\">https://arxiv.org/abs/2608.24115</a></p>\n","updatedAt":"2026-08-31T01:57:04.139Z","author":{"_id":"646484cfb90150b2706df03b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646484cfb90150b2706df03b/8ocSbXBSbrruhlhcxwzEt.png","fullname":"Jaeyoon Jung","name":"lastdefiance20","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":5,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8490726947784424},"editors":["lastdefiance20"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/646484cfb90150b2706df03b/8ocSbXBSbrruhlhcxwzEt.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.24115","authors":[{"_id":"6a94de07073195fee51571f5","name":"Suhwan Choi","hidden":false},{"_id":"6a94de07073195fee51571f6","name":"Jaeyoon Jung","hidden":false},{"_id":"6a94de07073195fee51571f7","name":"Sungkyung Kim","hidden":false},{"_id":"6a94de07073195fee51571f8","name":"Yunsung Lee","hidden":false},{"_id":"6a94de07073195fee51571f9","name":"Youngjae Yu","hidden":false}],"publishedAt":"2026-08-25T00:00:00.000Z","submittedOnDailyAt":"2026-08-31T00:00:00.000Z","title":"PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control","submittedOnDailyBy":{"_id":"646484cfb90150b2706df03b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646484cfb90150b2706df03b/8ocSbXBSbrruhlhcxwzEt.png","isPro":false,"fullname":"Jaeyoon Jung","user":"lastdefiance20","type":"user","name":"lastdefiance20"},"summary":"Multimodal large language models (MLLMs) can integrate long visual histories, reason under partial observability, and infer behavior from a few examples. Yet vision-language-action (VLA) models generally inherit pretrained representations without using this contextual capacity as episode memory. Memory-dependent policies address this gap through purpose-built history mechanisms. PonderPounce instead reuses an MLLM's native causal context as robot memory. Ponder, a System2 MLLM, accumulates episode observations, demonstrations, and prior cognition in its native causal context and can generate subgoal text and demonstration reasoning for internal use. Pounce, a System1 VLA, receives the current observation, instruction, and proprioception directly; through the Ponder--Pounce interface, it asynchronously receives only the newest continuous cognition token and its age. Both are jointly trained end to end without a purpose-built memory module or separate bridge pretraining. Optimized serving achieves p50 latencies of 78ms for cognition refresh and 25ms for action-model invocation, supporting 20Hz action playback. On RoboMME with base-scale training data, PonderPounce reaches 60.83% with 9B and 50.04% with 0.8B under the same Pounce architecture and interface, versus 44.51% for FrameSamp+Modul and 17.93% for the current-observation π_{0.5}. With 9x data, it reaches 75.54% versus 57.88% for FrameSamp+Modul. On RoboCasa-DC, the same interface learns from action supervision alone and reaches 12.5% versus 11.6% for the strongest published demonstration-conditioned baseline, falling to 8.6% when cognition is replaced by a learned null state.","upvotes":10,"discussionId":"6a94de08073195fee51571fa","projectPage":"https://worv-ai.github.io/ponderpounce","githubRepo":"https://github.com/worv-ai/PonderPounce","githubRepoAddedBy":"user","ai_summary":"PonderPounce leverages native causal context in multimodal language models as robot episode memory, jointly training a reasoning System2 module and a fast System1 action model to improve long-horizon policy performance without dedicated memory architectures.","ai_keywords":["multimodal large language models","vision-language-action models","causal context","System2 MLLM","System1 VLA","subgoal text","demonstration reasoning","Ponder--Pounce interface","RoboMME","RoboCasa-DC"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":2,"organization":{"_id":"64d32903129a210e56a3f3d3","name":"maum-ai","fullname":"maum-ai","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/646484cfb90150b2706df03b/QMrYBmsRfGZeU1DigPUzk.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"646484cfb90150b2706df03b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646484cfb90150b2706df03b/8ocSbXBSbrruhlhcxwzEt.png","isPro":false,"fullname":"Jaeyoon Jung","user":"lastdefiance20","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"65ec3449a69aaabb431db0da","avatarUrl":"/avatars/d7b507be0175a61a8fc21176eea45001.svg","isPro":false,"fullname":"Jin Hyeon Kim","user":"jinlovespho","type":"user"},{"_id":"67861f4658328c475597e540","avatarUrl":"/avatars/ff3d7b7912544cd0799d289e6c51db7a.svg","isPro":false,"fullname":"Seonghu Jeon","user":"SeonghuJeon","type":"user"},{"_id":"6a532d05fc4a19ceee5d2773","avatarUrl":"/avatars/bb6ec5a4097db53080c2432bb84481e5.svg","isPro":false,"fullname":"Jiwon Kang","user":"dryxco","type":"user"},{"_id":"669c95e0a2dc7c3fb84fbb11","avatarUrl":"/avatars/5e2f328c4be0d80a0cdf18823407b759.svg","isPro":false,"fullname":"Lee Hyunkoo","user":"pizb","type":"user"},{"_id":"670b90cbfb9aa61e417fd71f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/pT0vV0PiwnlEruarX7xb8.png","isPro":false,"fullname":"Woojin Cheong","user":"wj1128","type":"user"},{"_id":"68b1d8be868335455295e89f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/VH94dwDM6mWn5owTLp9Ju.png","isPro":false,"fullname":"Junsu KIM","user":"JJUNOS","type":"user"},{"_id":"63198bf1615c77c25d63e9ab","avatarUrl":"/avatars/6d591f87366e9990fed3c221dafdfae0.svg","isPro":false,"fullname":"Yunsung Lee","user":"Maangeek","type":"user"},{"_id":"632ced22ea6e62428aba1625","avatarUrl":"/avatars/0aae0ad956bdb2345732ca47fb28e3b6.svg","isPro":false,"fullname":"Suhwan Choi","user":"MilkClouds","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"64d32903129a210e56a3f3d3","name":"maum-ai","fullname":"maum-ai","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/646484cfb90150b2706df03b/QMrYBmsRfGZeU1DigPUzk.png"},"query":{}}">
PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control
Abstract
PonderPounce leverages native causal context in multimodal language models as robot episode memory, jointly training a reasoning System2 module and a fast System1 action model to improve long-horizon policy performance without dedicated memory architectures.
Multimodal large language models (MLLMs) can integrate long visual histories, reason under partial observability, and infer behavior from a few examples. Yet vision-language-action (VLA) models generally inherit pretrained representations without using this contextual capacity as episode memory. Memory-dependent policies address this gap through purpose-built history mechanisms. PonderPounce instead reuses an MLLM's native causal context as robot memory. Ponder, a System2 MLLM, accumulates episode observations, demonstrations, and prior cognition in its native causal context and can generate subgoal text and demonstration reasoning for internal use. Pounce, a System1 VLA, receives the current observation, instruction, and proprioception directly; through the Ponder--Pounce interface, it asynchronously receives only the newest continuous cognition token and its age. Both are jointly trained end to end without a purpose-built memory module or separate bridge pretraining. Optimized serving achieves p50 latencies of 78ms for cognition refresh and 25ms for action-model invocation, supporting 20Hz action playback. On RoboMME with base-scale training data, PonderPounce reaches 60.83% with 9B and 50.04% with 0.8B under the same Pounce architecture and interface, versus 44.51% for FrameSamp+Modul and 17.93% for the current-observation π_{0.5}. With 9x data, it reaches 75.54% versus 57.88% for FrameSamp+Modul. On RoboCasa-DC, the same interface learns from action supervision alone and reaches 12.5% versus 11.6% for the strongest published demonstration-conditioned baseline, falling to 8.6% when cognition is replaced by a learned null state.
Community
We’re excited to share PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control.
Many robot tasks require remembering information that is no longer visible, such as a briefly shown target, an earlier instruction, or a demonstration observed at the beginning of an episode. While pretrained MLLMs already have strong long-context reasoning capabilities, most vision-language-action models do not directly use this context as memory for control.
PonderPounce connects two pretrained systems:
- Ponder, a System 2 MLLM, maintains observations, demonstrations, and prior cognition in its native causal context.
- Pounce, a fast System 1 action model, controls the robot using the current observation together with Ponder’s latest continuous cognition token and its age.
Ponder and Pounce operate on decoupled clocks and are trained jointly end-to-end, without a purpose-built memory module or separate bridge pretraining.
On RoboMME, PonderPounce achieves 60.83% average success with 1× training data and 75.54% with 9× data. This corresponds to 1.37× and 1.31× the performance of the strongest non-oracle baseline, respectively.
🌐 Project page: https://worv-ai.github.io/ponderpounce/
📄 Paper: https://arxiv.org/abs/2608.24115
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.24115 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.24115 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.24115 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.