Tweet: <a href=\"https://x.com/rosinality/status/2075142494753353776\" rel=\"nofollow\">https://x.com/rosinality/status/2075142494753353776</a></p>\n","updatedAt":"2026-07-09T12:11:47.795Z","author":{"_id":"5f1158120c833276f61f1a84","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608042047613-5f1158120c833276f61f1a84.jpeg","fullname":"Niels Rogge","name":"nielsr","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":1250,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7172717452049255},"editors":["nielsr"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1608042047613-5f1158120c833276f61f1a84.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.07386","authors":[{"_id":"6a4f8ff2d1879620f032b19b","name":"Loïc Cabannes","hidden":false},{"_id":"6a4f8ff2d1879620f032b19c","name":"Pierre-Emmanuel Mazaré","hidden":false},{"_id":"6a4f8ff2d1879620f032b19d","name":"Gergely Szilvasy","hidden":false},{"_id":"6a4f8ff2d1879620f032b19e","name":"Matthijs Douze","hidden":false},{"_id":"6a4f8ff2d1879620f032b19f","name":"Maria Lomeli","hidden":false},{"_id":"6a4f8ff2d1879620f032b1a0","name":"Ilze Amanda Auzina","hidden":false},{"_id":"6a4f8ff2d1879620f032b1a1","name":"Justin Carpentier","hidden":false},{"_id":"6a4f8ff2d1879620f032b1a2","name":"Gabriel Synnaeve","hidden":false},{"_id":"6a4f8ff2d1879620f032b1a3","name":"Hervé Jégou","hidden":false}],"publishedAt":"2026-07-08T00:00:00.000Z","submittedOnDailyAt":"2026-07-09T00:00:00.000Z","title":"Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity","submittedOnDailyBy":{"_id":"5f1158120c833276f61f1a84","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608042047613-5f1158120c833276f61f1a84.jpeg","isPro":false,"fullname":"Niels Rogge","user":"nielsr","type":"user","name":"nielsr"},"summary":"Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.","upvotes":4,"discussionId":"6a4f8ff2d1879620f032b1a4","ai_summary":"Sparse Delta Memory extends gated linear RNNs with sparse addressing to dramatically increase hidden state capacity for improved long-context learning and retrieval while maintaining computational efficiency.","ai_keywords":["linear attention models","softmax-attention","transformer architectures","gated linear RNNs","sparse addressing scheme","Gated DeltaNet","explicit memory","in-context learning","long-context retrieval","parametric memory"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"5e63d8713071d5be688861b8","name":"facebook","fullname":"AI at Meta","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1592839207516-noauth.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"},{"_id":"62a1280b88bfb47fc40fe75b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62a1280b88bfb47fc40fe75b/u6teJWcB6BWdD04G7g6uy.png","isPro":false,"fullname":"Gabriel Mongaras","user":"gmongaras","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"661ab1f1fa3b144a381fa454","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png","isPro":false,"fullname":"Urro","user":"urroxyz","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"5e63d8713071d5be688861b8","name":"facebook","fullname":"AI at Meta","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1592839207516-noauth.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.07386.md","query":{}}">
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Abstract
Sparse Delta Memory extends gated linear RNNs with sparse addressing to dramatically increase hidden state capacity for improved long-context learning and retrieval while maintaining computational efficiency.
Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.07386 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.07386 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.07386 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.