Meta-Cognitive Memory Policy Optimization for<br>Long-Horizon LLM Agents</p>\n","updatedAt":"2026-06-05T03:05:36.345Z","author":{"_id":"6777aeedcd9c943549b26527","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wTusgCoTbVqvu2jeaebcP.png","fullname":"liu","name":"ziyan2003","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6406147480010986},"editors":["ziyan2003"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wTusgCoTbVqvu2jeaebcP.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.30159","authors":[{"_id":"6a21484c3490a593e87b0e7c","user":{"_id":"6777aeedcd9c943549b26527","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wTusgCoTbVqvu2jeaebcP.png","isPro":false,"fullname":"liu","user":"ziyan2003","type":"user","name":"ziyan2003"},"name":"Ziyan Liu","status":"claimed_verified","statusLastChangedAt":"2026-06-04T12:37:14.871Z","hidden":false},{"_id":"6a21484c3490a593e87b0e7d","name":"Zhezheng Hao","hidden":false},{"_id":"6a21484c3490a593e87b0e7e","name":"Yeqiu Chen","hidden":false},{"_id":"6a21484c3490a593e87b0e7f","name":"Hong Wang","hidden":false},{"_id":"6a21484c3490a593e87b0e80","name":"Jingren Hou","hidden":false},{"_id":"6a21484c3490a593e87b0e81","name":"Ruiyi Ding","hidden":false},{"_id":"6a21484c3490a593e87b0e82","name":"Yongkang Yang","hidden":false},{"_id":"6a21484c3490a593e87b0e83","name":"Wence Ji","hidden":false},{"_id":"6a21484c3490a593e87b0e84","name":"Wei Xia","hidden":false},{"_id":"6a21484c3490a593e87b0e85","name":"Feng Liu","hidden":false}],"publishedAt":"2026-05-28T00:00:00.000Z","submittedOnDailyAt":"2026-06-05T00:00:00.000Z","title":"Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents","submittedOnDailyBy":{"_id":"6777aeedcd9c943549b26527","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wTusgCoTbVqvu2jeaebcP.png","isPro":false,"fullname":"liu","user":"ziyan2003","type":"user","name":"ziyan2003"},"summary":"Memory-augmented LLM agents tackle complex long-horizon tasks by recursively summarizing interaction trajectories into compact memory. However, existing approaches typically train these memory policies using outcome-based reinforcement learning, failing to localize where intermediate memory quality degrades. As interactions unfold, ambiguous recursive summaries progressively discard task-relevant information and introduce semantic noise. This exacerbates belief deviation, obscuring the agent's estimate of the latent task state and ultimately derailing long-horizon reasoning. We therefore argue that memory optimization should focus not merely on trajectory-level success, but on the clarity of the belief induced by intermediate summaries. To this end, we introduce Belief Entropy, a self-supervised proxy that probes how uncertain the model remains about the latent task state given its current memory. Based on this proxy, we propose Metacognitive Memory Policy Optimization (MMPO). Instead of relying only on sparse outcome-based signals, MMPO provides fine-grained, memory-specific supervision via explicitly penalizing summaries that induce high epistemic uncertainty. Experiments show that MMPO consistently outperforms existing methods on diverse long-horizon tasks, maintaining 97.1% performance even when scaled to 1.75M-token contexts.","upvotes":3,"discussionId":"6a21484d3490a593e87b0e86","ai_summary":"Memory-augmented language models struggle with long-horizon tasks due to information loss in recursive summaries, but a new method using belief entropy and metacognitive policy optimization improves performance by focusing on memory quality rather than just outcome success.","ai_keywords":["memory-augmented LLM agents","recursive summarization","reinforcement learning","belief deviation","belief entropy","metacognitive memory policy optimization","epistemic uncertainty","long-horizon tasks","latent task state"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"61d8000084231b832e5bbd99","name":"ustc","fullname":"university of science and technology of china","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1641545773772-61d7fdeb22a383817a543b68.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6777aeedcd9c943549b26527","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wTusgCoTbVqvu2jeaebcP.png","isPro":false,"fullname":"liu","user":"ziyan2003","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"6888c0ef00c018ee0d25c8e4","avatarUrl":"/avatars/b1ee2eb756dcd0eb6a7890d36587925e.svg","isPro":false,"fullname":"Yeqiu Chen","user":"cyqloveljj","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"61d8000084231b832e5bbd99","name":"ustc","fullname":"university of science and technology of china","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1641545773772-61d7fdeb22a383817a543b68.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.30159.md"}">
Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents
Published on May 28
· Submitted by liu on Jun 5 Abstract
Memory-augmented language models struggle with long-horizon tasks due to information loss in recursive summaries, but a new method using belief entropy and metacognitive policy optimization improves performance by focusing on memory quality rather than just outcome success.
Memory-augmented LLM agents tackle complex long-horizon tasks by recursively summarizing interaction trajectories into compact memory. However, existing approaches typically train these memory policies using outcome-based reinforcement learning, failing to localize where intermediate memory quality degrades. As interactions unfold, ambiguous recursive summaries progressively discard task-relevant information and introduce semantic noise. This exacerbates belief deviation, obscuring the agent's estimate of the latent task state and ultimately derailing long-horizon reasoning. We therefore argue that memory optimization should focus not merely on trajectory-level success, but on the clarity of the belief induced by intermediate summaries. To this end, we introduce Belief Entropy, a self-supervised proxy that probes how uncertain the model remains about the latent task state given its current memory. Based on this proxy, we propose Metacognitive Memory Policy Optimization (MMPO). Instead of relying only on sparse outcome-based signals, MMPO provides fine-grained, memory-specific supervision via explicitly penalizing summaries that induce high epistemic uncertainty. Experiments show that MMPO consistently outperforms existing methods on diverse long-horizon tasks, maintaining 97.1% performance even when scaled to 1.75M-token contexts.
Community
Meta-Cognitive Memory Policy Optimization for
Long-Horizon LLM Agents
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.30159 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.30159 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.30159 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.