Excited to be bringing xLSTM to FLA — updates coming soon!</p>\n","updatedAt":"2026-06-11T11:21:15.671Z","author":{"_id":"64d35b95508a6313e319778a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64d35b95508a6313e319778a/_epY7-7H4rALNPXh9uZ7h.jpeg","fullname":"Anamaria-Roberta Hartl","name":"anamariarobertap","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":7,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8316769599914551},"editors":["anamariarobertap"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/64d35b95508a6313e319778a/_epY7-7H4rALNPXh9uZ7h.jpeg"],"reactions":[],"isReport":false}},{"id":"6a2aa6832a7cf6946211c05d","author":{"_id":"6960eca92f7ad9b043b5cbe0","avatarUrl":"/avatars/e68dcc7fd04f143d849d40414866e633.svg","fullname":"Noah","name":"noahml","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-06-11T12:13:55.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Neat paper. It is interesting to see a direct comparison between xLSTM, Mamba-2, and Gated DeltaNet, especially since the subquadratic space has been getting so crowded lately. I like that you focused on the memory dynamics to explain why xLSTM is pulling ahead in your tests.\n\nDo you think the advantages you found in state tracking for xLSTM would hold up if you scaled these models to even larger parameter counts, or is this primarily a feature of their current architecture?\n\nI made a podcast on it with ResearchPod, it makes it easy to get the key concepts on the go:\nhttps://researchpod.app/episode/6eb12b37-8233-48fb-9760-d6c870d6e6de","html":"<p>Neat paper. It is interesting to see a direct comparison between xLSTM, Mamba-2, and Gated DeltaNet, especially since the subquadratic space has been getting so crowded lately. I like that you focused on the memory dynamics to explain why xLSTM is pulling ahead in your tests.</p>\n<p>Do you think the advantages you found in state tracking for xLSTM would hold up if you scaled these models to even larger parameter counts, or is this primarily a feature of their current architecture?</p>\n<p>I made a podcast on it with ResearchPod, it makes it easy to get the key concepts on the go:<br><a href=\"https://researchpod.app/episode/6eb12b37-8233-48fb-9760-d6c870d6e6de\" rel=\"nofollow\">https://researchpod.app/episode/6eb12b37-8233-48fb-9760-d6c870d6e6de</a></p>\n","updatedAt":"2026-06-11T12:13:55.300Z","author":{"_id":"6960eca92f7ad9b043b5cbe0","avatarUrl":"/avatars/e68dcc7fd04f143d849d40414866e633.svg","fullname":"Noah","name":"noahml","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9575162529945374},"editors":["noahml"],"editorAvatarUrls":["/avatars/e68dcc7fd04f143d849d40414866e633.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.12364","authors":[{"_id":"6a2a83f7eb926374a38219bd","name":"Anamaria-Roberta Hartl","hidden":false},{"_id":"6a2a83f7eb926374a38219be","name":"Levente Zólyomi","hidden":false},{"_id":"6a2a83f7eb926374a38219bf","name":"David Stap","hidden":false},{"_id":"6a2a83f7eb926374a38219c0","name":"Pieter-Jan Hoedt","hidden":false},{"_id":"6a2a83f7eb926374a38219c1","name":"Niklas Schmidinger","hidden":false},{"_id":"6a2a83f7eb926374a38219c2","name":"Lukas Hauzenberger","hidden":false},{"_id":"6a2a83f7eb926374a38219c3","name":"Sebastian Böck","hidden":false},{"_id":"6a2a83f7eb926374a38219c4","name":"Günter Klambauer","hidden":false},{"_id":"6a2a83f7eb926374a38219c5","name":"Sepp Hochreiter","hidden":false}],"publishedAt":"2026-06-10T17:33:55.000Z","submittedOnDailyAt":"2026-06-11T00:00:00.000Z","title":"On Subquadratic Architectures: From Applications to Principles","submittedOnDailyBy":{"_id":"64d35b95508a6313e319778a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64d35b95508a6313e319778a/_epY7-7H4rALNPXh9uZ7h.jpeg","isPro":false,"fullname":"Anamaria-Roberta Hartl","user":"anamariarobertap","type":"user","name":"anamariarobertap"},"summary":"Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet. We evaluate these models on tasks with complex dependencies: (1) code-model pre-training, (2) distillation of code models from large language models, and (3) pre-training of time-series foundation models. Across these settings, xLSTM delivers the strongest overall performance. To explain xLSTM's advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing on state tracking and memory dynamics. Our results show that xLSTM enables more flexible and stable memory correction via its gating scheme. We corroborate these findings on controlled synthetic length-generalization tasks. Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.","upvotes":20,"discussionId":"6a2a83f8eb926374a38219c6","ai_summary":"xLSTM demonstrates superior performance in sequence modeling tasks compared to Mamba-2 and Gated DeltaNet due to enhanced state tracking and memory dynamics.","ai_keywords":["Transformers","attention","subquadratic architectures","xLSTM","Mamba-2","Gated DeltaNet","sequence modeling","code-model pre-training","distillation","time-series foundation models","state tracking","memory dynamics","gating scheme","length-generalization tasks"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6864f037f2004fd7264bc0fb","avatarUrl":"/avatars/b592be746703825d3d46b4ee2bfb2b46.svg","isPro":false,"fullname":"AntoniaEbner","user":"antoniaebner","type":"user"},{"_id":"6329a6a63f5b938fb5cab3c6","avatarUrl":"/avatars/dc9b9f987ad113a4746eaff1fba30f15.svg","isPro":false,"fullname":"Wilhelm Berghammer","user":"wilhelmberghammer","type":"user"},{"_id":"6943dcacce9c7bfa428cd2a1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/oJXt1AdCBVqYrATNC1HPE.jpeg","isPro":false,"fullname":"Gianluca Galletti","user":"gerkone","type":"user"},{"_id":"639d28d2b0496dc19b7b4322","avatarUrl":"/avatars/c32146392b68e45f421fba9356001f93.svg","isPro":false,"fullname":"Paul Setinek","user":"psetinek","type":"user"},{"_id":"6a031c4caa6d4635c4f97a2c","avatarUrl":"/avatars/16865f29a822b570de68436720deca07.svg","isPro":false,"fullname":"Leonardo Barcellona","user":"leonardo-nxai","type":"user"},{"_id":"6448e678de60e26710a71444","avatarUrl":"/avatars/9ea7b08f27267777aca3c665ec8960a8.svg","isPro":false,"fullname":"Johannes Schimunek","user":"Tschoui","type":"user"},{"_id":"62c53302662066c135ff6b24","avatarUrl":"/avatars/263e3906070a01f745ddd704d3b13433.svg","isPro":false,"fullname":"Daniil Yefimov","user":"danilyef","type":"user"},{"_id":"6733635a0ae3565538198583","avatarUrl":"/avatars/985f252b776e7120b9e006acfa29bf50.svg","isPro":false,"fullname":"Elias Bürger","user":"e-l-i-a-s","type":"user"},{"_id":"68c5a12c3fd744527981700d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/xZJCkVMJ0uEa-aiyrTlGh.png","isPro":false,"fullname":"Levi Zolyomi","user":"levi-zoly","type":"user"},{"_id":"6359028d6ca91ddc7401b757","avatarUrl":"/avatars/c45c7c3e8c25eb18cd41afce1d5e9d54.svg","isPro":false,"fullname":"David Stap","user":"davidstap","type":"user"},{"_id":"68d25efa6e7fd4172250c590","avatarUrl":"/avatars/2df642670dbed4bbdcee94a4e9086118.svg","isPro":false,"fullname":"Christian Ganhör","user":"tigxy","type":"user"},{"_id":"6303cca45c70c21d0eaa8edb","avatarUrl":"/avatars/10156b391d5bd6813f40188df80cca85.svg","isPro":false,"fullname":"gg","user":"CakeMixCore","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.12364.md"}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2606.12364 in a model README.md to link it from this page.

Cite arxiv.org/abs/2606.12364 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2606.12364 in a Space README.md to link it from this page.