[!IMPORTANT]\n> **Metis**: The first prototype of a memory foundation model, equipping foundation models with a persistent and dynamically evolving native memory state. \n\nWhat if memory were a native capability of foundation models, rather than an external module? \n\nRecent advances have increasingly internalized capabilities such as multimodal perception and reasoning into foundation models. Agent memory, however, is still primarily implemented through external retrieval, storage, and prompt-construction pipelines. \n\nWe introduce **memory foundation models**, which treat memory as a first-class capability of the model itself. We formalize **native memory** from two complementary perspectives: \n\n* **Native memory state:** a persistent and dynamically evolving state within the model backbone that participates directly in subsequent computation. \n* **Native memory procedures:** learned procedures that autonomously determine what to store and how to utilize the stored information. \n\nAt its core, each Metis Block combines a **Local Memory Block**, which maintains the persistent memory state, with a **Hyper Memory Block**, which learns the storage and utilization procedures. Historical information is compressed into a compact, fixed-size session state and accessed through memory attention, allowing later queries to use the memory without replaying the original history. \n\nMemory maintenance is gradient-free and requires only forward computation. At inference time, all learned model weights remain frozen, while the native memory state evolves through standard model computation. We release the implementation and Qwen3.5-based checkpoints at **4B, 9B, and 27B** scales.","html":"<blockquote class=\"important\">\n<p><strong>Metis</strong>: The first prototype of a memory foundation model, equipping foundation models with a persistent and dynamically evolving native memory state. </p>\n</blockquote>\n<p>What if memory were a native capability of foundation models, rather than an external module? </p>\n<p>Recent advances have increasingly internalized capabilities such as multimodal perception and reasoning into foundation models. Agent memory, however, is still primarily implemented through external retrieval, storage, and prompt-construction pipelines. </p>\n<p>We introduce <strong>memory foundation models</strong>, which treat memory as a first-class capability of the model itself. We formalize <strong>native memory</strong> from two complementary perspectives: </p>\n<ul>\n<li><strong>Native memory state:</strong> a persistent and dynamically evolving state within the model backbone that participates directly in subsequent computation. </li>\n<li><strong>Native memory procedures:</strong> learned procedures that autonomously determine what to store and how to utilize the stored information.</li>\n</ul>\n<p>At its core, each Metis Block combines a <strong>Local Memory Block</strong>, which maintains the persistent memory state, with a <strong>Hyper Memory Block</strong>, which learns the storage and utilization procedures. Historical information is compressed into a compact, fixed-size session state and accessed through memory attention, allowing later queries to use the memory without replaying the original history. </p>\n<p>Memory maintenance is gradient-free and requires only forward computation. At inference time, all learned model weights remain frozen, while the native memory state evolves through standard model computation. We release the implementation and Qwen3.5-based checkpoints at <strong>4B, 9B, and 27B</strong> scales.</p>\n","updatedAt":"2026-07-31T02:10:34.339Z","author":{"_id":"6961488e27354abbd898529c","avatarUrl":"/avatars/4769f91b400ee6e9de2943613d8b9384.svg","fullname":"Xichong Zhang","name":"astarkkk","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8968239426612854},"editors":["astarkkk"],"editorAvatarUrls":["/avatars/4769f91b400ee6e9de2943613d8b9384.svg"],"reactions":[],"isReport":false}},{"id":"6a6c07d5723248f520a5ffc0","author":{"_id":"6961488e27354abbd898529c","avatarUrl":"/avatars/4769f91b400ee6e9de2943613d8b9384.svg","fullname":"Xichong Zhang","name":"astarkkk","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-07-31T02:26:29.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"<div align=\"center\">\n <img src=\"https://cdn-uploads.huggingface.co/production/uploads/6961488e27354abbd898529c/wxDrzCHf8DzcfcC_RZ_z_.png\" width=\"50%\">\n</div>\n\n[Paper](https://arxiv.org/abs/2607.26760) · [Models](https://huggingface.co/collections/IAAR-Shanghai/metis) · [Code](https://github.com/MemTensor/Metis)\n\n## 💡 Overview\n\n**Metis is the first prototype of a memory foundation model, equipping foundation models with a persistent and dynamically evolving native memory state.**\n\nIt learns to autonomously store and utilize information through model computation, compressing historical context into native memory and accessing it through memory attention. At inference time, all model weights remain frozen, while memory is updated through gradient-free forward computation.\n\n<div align=\"center\">\n <img src=\"https://cdn-uploads.huggingface.co/production/uploads/6961488e27354abbd898529c/xs83rgQ1TKtjpaKfbzvLS.png\" width=\"100%\">\n</div>\n\nThe project explores three central ideas:\n\n- **Native memory state.** Dynamic parametric states live inside the backbone and participate directly in later forward passes.\n- **Native memory procedures.** Storage and utilization are learned from data instead of being implemented as separate retrieval, reranking, and prompt-construction rules.\n- **Fixed-size session state.** Historical information is compressed into a compact state, so later queries do not need to replay the original text history.\n\n### Architecture\n\nA **Metis Block** is inserted into Transformer layers and contains two components:\n\n- The **Local Memory Block** maintains the dynamic memory matrix and normalization state that persist across interaction steps.\n- The **Hyper Memory Block** learns token selection, memory key/value projections, a dedicated memory query, and the state-update procedure.\n\nAfter each memory step, Metis selects informative hidden states and updates the local memory. During a later query, memory attention reads that state and fuses the result with the original attention branch. The default implementation uses a **Gated Delta Network (GDN)** update. In the Qwen3.5 hybrid implementation, Metis is attached to full-attention layers, while linear-attention layers keep their original computation path.\n\n## Roadmap\n\n<div align=\"center\">\n <img src=\"https://cdn-uploads.huggingface.co/production/uploads/6961488e27354abbd898529c/agEpwCTEtEbvKAtAtDtn9.png\" width=\"100%\">\n</div>\n\nThe paper frames native memory as a progression from **stateful capability** to **self-managing memory**, **experience-driven learning**, **persistent cognition**, and ultimately **self-evolving capability**.\n\nMetis is an early research system rather than a complete replacement for external memory. Hybrid native–external memory remains an important direction.\n","html":"<div align=\"center\">\n <img src=\"https://cdn-uploads.huggingface.co/production/uploads/6961488e27354abbd898529c/wxDrzCHf8DzcfcC_RZ_z_.png\" width=\"50%\">\n</div>\n\n<p><a href=\"https://arxiv.org/abs/2607.26760\" rel=\"nofollow\">Paper</a> · <a href=\"https://huggingface.co/collections/IAAR-Shanghai/metis\">Models</a> · <a href=\"https://github.com/MemTensor/Metis\" rel=\"nofollow\">Code</a></p>\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"💡-overview\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#💡-overview\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\t💡 Overview\n\t</span>\n</h2>\n<p><strong>Metis is the first prototype of a memory foundation model, equipping foundation models with a persistent and dynamically evolving native memory state.</strong></p>\n<p>It learns to autonomously store and utilize information through model computation, compressing historical context into native memory and accessing it through memory attention. At inference time, all model weights remain frozen, while memory is updated through gradient-free forward computation.</p>\n<div align=\"center\">\n <img src=\"https://cdn-uploads.huggingface.co/production/uploads/6961488e27354abbd898529c/xs83rgQ1TKtjpaKfbzvLS.png\" width=\"100%\">\n</div>\n\n<p>The project explores three central ideas:</p>\n<ul>\n<li><strong>Native memory state.</strong> Dynamic parametric states live inside the backbone and participate directly in later forward passes.</li>\n<li><strong>Native memory procedures.</strong> Storage and utilization are learned from data instead of being implemented as separate retrieval, reranking, and prompt-construction rules.</li>\n<li><strong>Fixed-size session state.</strong> Historical information is compressed into a compact state, so later queries do not need to replay the original text history.</li>\n</ul>\n<h3 class=\"relative group flex items-baseline\">\n\t<a id=\"architecture\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#architecture\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\tArchitecture\n\t</span>\n</h3>\n<p>A <strong>Metis Block</strong> is inserted into Transformer layers and contains two components:</p>\n<ul>\n<li>The <strong>Local Memory Block</strong> maintains the dynamic memory matrix and normalization state that persist across interaction steps.</li>\n<li>The <strong>Hyper Memory Block</strong> learns token selection, memory key/value projections, a dedicated memory query, and the state-update procedure.</li>\n</ul>\n<p>After each memory step, Metis selects informative hidden states and updates the local memory. During a later query, memory attention reads that state and fuses the result with the original attention branch. The default implementation uses a <strong>Gated Delta Network (GDN)</strong> update. In the Qwen3.5 hybrid implementation, Metis is attached to full-attention layers, while linear-attention layers keep their original computation path.</p>\n<h2 class=\"relative group flex items-baseline\">\n\t<a id=\"roadmap\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#roadmap\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\tRoadmap\n\t</span>\n</h2>\n<div align=\"center\">\n <img src=\"https://cdn-uploads.huggingface.co/production/uploads/6961488e27354abbd898529c/agEpwCTEtEbvKAtAtDtn9.png\" width=\"100%\">\n</div>\n\n<p>The paper frames native memory as a progression from <strong>stateful capability</strong> to <strong>self-managing memory</strong>, <strong>experience-driven learning</strong>, <strong>persistent cognition</strong>, and ultimately <strong>self-evolving capability</strong>.</p>\n<p>Metis is an early research system rather than a complete replacement for external memory. Hybrid native–external memory remains an important direction.</p>\n","updatedAt":"2026-07-31T02:30:30.848Z","author":{"_id":"6961488e27354abbd898529c","avatarUrl":"/avatars/4769f91b400ee6e9de2943613d8b9384.svg","fullname":"Xichong Zhang","name":"astarkkk","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":2,"identifiedLanguage":{"language":"en","probability":0.7669006586074829},"editors":["astarkkk"],"editorAvatarUrls":["/avatars/4769f91b400ee6e9de2943613d8b9384.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.26760","authors":[{"_id":"6a6acca94463a8a84bdc3fdf","name":"Zeyu Zhang","hidden":false},{"_id":"6a6acca94463a8a84bdc3fe0","name":"Ziliang Guo","hidden":false},{"_id":"6a6acca94463a8a84bdc3fe1","name":"Yihang Sun","hidden":false},{"_id":"6a6acca94463a8a84bdc3fe2","user":{"_id":"6961488e27354abbd898529c","avatarUrl":"/avatars/4769f91b400ee6e9de2943613d8b9384.svg","isPro":false,"fullname":"Xichong Zhang","user":"astarkkk","type":"user","name":"astarkkk"},"name":"Xichong Zhang","status":"claimed_verified","statusLastChangedAt":"2026-07-30T08:45:04.406Z","hidden":false},{"_id":"6a6acca94463a8a84bdc3fe3","name":"Xixuan Hao","hidden":false},{"_id":"6a6acca94463a8a84bdc3fe4","name":"Zehao Lin","hidden":false},{"_id":"6a6acca94463a8a84bdc3fe5","name":"Yang Zhang","hidden":false},{"_id":"6a6acca94463a8a84bdc3fe6","name":"Xiaoyan Zhao","hidden":false},{"_id":"6a6acca94463a8a84bdc3fe7","user":{"_id":"68170f09096b10da567325b3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/rIML2Ql3I8klM8Mit3vIb.png","isPro":false,"fullname":"Shen Tong","user":"Arsenide","type":"user","name":"Arsenide"},"name":"Tong Shen","status":"claimed_verified","statusLastChangedAt":"2026-07-30T08:45:04.412Z","hidden":false},{"_id":"6a6acca94463a8a84bdc3fe8","name":"Bo Tang","hidden":false},{"_id":"6a6acca94463a8a84bdc3fe9","name":"Zhi-Qin John Xu","hidden":false},{"_id":"6a6acca94463a8a84bdc3fea","name":"Junchi Yan","hidden":false},{"_id":"6a6acca94463a8a84bdc3feb","name":"Haofen Wang","hidden":false},{"_id":"6a6acca94463a8a84bdc3fec","name":"Xu Chen","hidden":false},{"_id":"6a6acca94463a8a84bdc3fed","name":"Feiyu Xiong","hidden":false},{"_id":"6a6acca94463a8a84bdc3fee","name":"Zhiyu Li","hidden":false},{"_id":"6a6acca94463a8a84bdc3fef","name":"Tat-Seng Chua","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/6961488e27354abbd898529c/wxDrzCHf8DzcfcC_RZ_z_.png","https://cdn-uploads.huggingface.co/production/uploads/6961488e27354abbd898529c/agEpwCTEtEbvKAtAtDtn9.png","https://cdn-uploads.huggingface.co/production/uploads/6961488e27354abbd898529c/xs83rgQ1TKtjpaKfbzvLS.png"],"publishedAt":"2026-07-29T00:00:00.000Z","submittedOnDailyAt":"2026-07-31T00:00:00.000Z","title":"Metis: Memory Foundation Model","submittedOnDailyBy":{"_id":"6961488e27354abbd898529c","avatarUrl":"/avatars/4769f91b400ee6e9de2943613d8b9384.svg","isPro":false,"fullname":"Xichong Zhang","user":"astarkkk","type":"user","name":"astarkkk"},"summary":"Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.","upvotes":97,"discussionId":"6a6acca94463a8a84bdc3ff0","githubRepo":"https://github.com/MemTensor/Metis","githubRepoAddedBy":"user","githubStars":26,"organization":{"_id":"684d4f8e0bb9b6d7621cd53b","name":"MemTensor","fullname":"MemTensor","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/62a155e615eeab266b2f2243/2mVH99TFqle9MJVb95aDC.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"680856e92be9415acee8ee90","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/680856e92be9415acee8ee90/qAQQ16SNFA5ARdpT87Swp.webp","isPro":false,"fullname":"zl Guo","user":"rekrats","type":"user"},{"_id":"64d4615cf8082bf19b916492","avatarUrl":"/avatars/8e1b59565ec5e4b31090cf1b911781b9.svg","isPro":false,"fullname":"wongyukim","user":"wongyukim","type":"user"},{"_id":"69ca19837cc8762eb2973ff4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/046BzMNSoGlp_badcdKRL.jpeg","isPro":false,"fullname":"ziliang","user":"nworats","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"6961488e27354abbd898529c","avatarUrl":"/avatars/4769f91b400ee6e9de2943613d8b9384.svg","isPro":false,"fullname":"Xichong Zhang","user":"astarkkk","type":"user"},{"_id":"647af2849c71e428601e4be4","avatarUrl":"/avatars/62b8713818f2b0f1afd21a187c4507d6.svg","isPro":false,"fullname":"Zeyu Zhang","user":"Nuster1128","type":"user"},{"_id":"62a155e615eeab266b2f2243","avatarUrl":"/avatars/e89ef156e73af028e3ce3664e6cb4e62.svg","isPro":false,"fullname":"Zhiyu Li","user":"jimi888","type":"user"},{"_id":"64a27e5098fad0c8a5c236d4","avatarUrl":"/avatars/1f67bf3744786b5d3dc491968fe0c0d3.svg","isPro":false,"fullname":"Wenqiang Wei","user":"wwq38556399","type":"user"},{"_id":"64d9cd5cd38302bf80154d21","avatarUrl":"/avatars/a0408280a24c02582fd0966786f34813.svg","isPro":false,"fullname":"Ozzzp Ben","user":"Ozzzp","type":"user"},{"_id":"69c9b95825713d5b2a2dc94b","avatarUrl":"/avatars/515fc517cfd2cd202423b547a6bdd230.svg","isPro":false,"fullname":"Chris Yang","user":"Christopher333","type":"user"},{"_id":"6491aec41e161baf7894b54b","avatarUrl":"/avatars/0af10e24669125e4329c003fc27b603e.svg","isPro":false,"fullname":"huhelan","user":"hhhllan","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"684d4f8e0bb9b6d7621cd53b","name":"MemTensor","fullname":"MemTensor","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/62a155e615eeab266b2f2243/2mVH99TFqle9MJVb95aDC.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.26760.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2607.26760 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2607.26760 in a Space README.md to link it from this page.