Hugging Face Daily Papers · May 29, 2026 · 6 min read

Parallax: Parameterized Local Linear Attention for Language Modeling

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

LLA paper: <a href=\"https://arxiv.org/abs/2510.01450\" rel=\"nofollow\">https://arxiv.org/abs/2510.01450</a> LLA kernel was published in the github: <a href=\"https://github.com/Yifei-Zuo/FlashLLA\" rel=\"nofollow\">https://github.com/Yifei-Zuo/FlashLLA</a>\n","updatedAt":"2026-05-29T02:43:39.149Z","author":{"_id":"67fae9119e295dbf0a6edb80","avatarUrl":"/avatars/c6b3604553b3aa1b563e1df73a4633c7.svg","fullname":"Yifei Zuo","name":"YifeiZuo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8838603496551514},"editors":["YifeiZuo"],"editorAvatarUrls":["/avatars/c6b3604553b3aa1b563e1df73a4633c7.svg"],"reactions":[],"isReport":false}},{"id":"6a1a40631a986dda4c8d1ff6","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":359,"isUserFollowing":false},"createdAt":"2026-05-30T01:41:55.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Adaptive Memory Decay for Log-Linear Attention](https://huggingface.co/papers/2605.06946) (2026)\n* [MDN: Parallelizing Stepwise Momentum for Delta Linear Attention](https://huggingface.co/papers/2605.05838) (2026)\n* [Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion](https://huggingface.co/papers/2604.05688) (2026)\n* [Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility](https://huggingface.co/papers/2605.14037) (2026)\n* [Interdomain Attention: Beyond Token-Level Key-Value Memory](https://huggingface.co/papers/2605.24330) (2026)\n* [Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations](https://huggingface.co/papers/2605.28769) (2026)\n* [OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention](https://huggingface.co/papers/2605.13473) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. \nThe following papers were recommended by the Semantic Scholar API \n<ul>\n<li><a href=\"https://huggingface.co/papers/2605.06946\">Adaptive Memory Decay for Log-Linear Attention</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.05838\">MDN: Parallelizing Stepwise Momentum for Delta Linear Attention</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2604.05688\">Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.14037\">Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.24330\">Interdomain Attention: Beyond Token-Level Key-Value Memory</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.28769\">Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.13473\">OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention</a> (2026)</li>\n</ul>\n Please give a thumbs up to this comment if you found it helpful!\n If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code><a href=\"/librarian-bot\">@librarian-bot</a> recommend</code>\n","updatedAt":"2026-05-30T01:41:55.213Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":359,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7347705364227295},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.29157","authors":[{"_id":"6a18fc0c56b4bb14ec65cec4","user":{"_id":"67fae9119e295dbf0a6edb80","avatarUrl":"/avatars/c6b3604553b3aa1b563e1df73a4633c7.svg","isPro":false,"fullname":"Yifei Zuo","user":"YifeiZuo","type":"user","name":"YifeiZuo"},"name":"Yifei Zuo","status":"claimed_verified","statusLastChangedAt":"2026-05-29T08:51:03.110Z","hidden":false},{"_id":"6a18fc0c56b4bb14ec65cec5","name":"Dhruv Pai","hidden":false},{"_id":"6a18fc0c56b4bb14ec65cec6","name":"Zhichen Zeng","hidden":false},{"_id":"6a18fc0c56b4bb14ec65cec7","name":"Alec Dewulf","hidden":false},{"_id":"6a18fc0c56b4bb14ec65cec8","name":"Shuming Hu","hidden":false},{"_id":"6a18fc0c56b4bb14ec65cec9","name":"Zhaoran Wang","hidden":false}],"publishedAt":"2026-05-27T00:00:00.000Z","submittedOnDailyAt":"2026-05-29T00:00:00.000Z","title":"Parallax: Parameterized Local Linear Attention for Language Modeling","submittedOnDailyBy":{"_id":"67fae9119e295dbf0a6edb80","avatarUrl":"/avatars/c6b3604553b3aa1b563e1df73a4633c7.svg","isPro":false,"fullname":"Yifei Zuo","user":"YifeiZuo","type":"user","name":"YifeiZuo"},"summary":"Large Language Models (LLMs) have become the central paradigm in artificial intelligence, yet the core computational primitive of attention has remained structurally unchanged. Local Linear Attention (LLA) is an attention mechanism derived from nonparametric statistics in the test-time regression framework. In contrast to prior research on efficient attention variants, LLA upgrades the local constant estimate in softmax attention to a local linear estimate, yielding provably superior bias-variance tradeoffs for associative memory. However, LLA has not been scaled in LLM pretraining due to computational and numerical stability concerns. We introduce Parallax, a parameterized Local Linear Attention that is scalable for LLMs. Parallax eliminates the numerical solver in LLA and learns an extra query-like projector that probes the KV covariance. We place Parallax within a family of attention mechanisms connected by the bandwidth, the probe construction and the affine structure. We propose a hardware-aware algorithm that increases the arithmetic intensity over FlashAttention, shifting attention into a more compute bound regime. Our prototype decode kernel matches or outperforms FlashAttention 2/3 across diverse batch sizes and context lengths. We pretrain Parallax at 0.6B and 1.7B scales and find consistent perplexity improvements throughout pretraining with gains that transfer to downstream benchmarks. The advantage persists under both parameter-matched and compute-matched controls, demonstrating a Pareto improvement. We perform careful pretraining ablations and identify a novel phenomenon whereby Muon unlocks the capacity of Parallax. To our knowledge, this is the first empirical demonstration of strong architecture-optimizer codesign for attention mechanisms in the architecture research literature.","upvotes":1,"discussionId":"6a18fc0c56b4bb14ec65ceca","githubRepo":"https://github.com/Yifei-Zuo/Parallax","githubRepoAddedBy":"user","ai_summary":"Local Linear Attention is enhanced through parameterization and hardware-aware optimization to improve LLM training efficiency and performance while maintaining computational stability.","ai_keywords":["attention mechanism","local linear attention","softmax attention","associative memory","parameterized attention","query-like projector","KV covariance","bandwidth","affine structure","FlashAttention","arithmetic intensity","decode kernel","perplexity","pretraining","Muon","architecture-optimizer codesign"],"githubStars":30,"organization":{"_id":"680e1986c980bb2d024cdfaa","name":"northwestern-university","fullname":"Northwestern University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/680e15aacadcd7ce86cac9cf/0qOIptd897seA531DA6k2.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67fae9119e295dbf0a6edb80","avatarUrl":"/avatars/c6b3604553b3aa1b563e1df73a4633c7.svg","isPro":false,"fullname":"Yifei Zuo","user":"YifeiZuo","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"680e1986c980bb2d024cdfaa","name":"northwestern-university","fullname":"Northwestern University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/680e15aacadcd7ce86cac9cf/0qOIptd897seA531DA6k2.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.29157.md"}">

Papers

arxiv:2605.29157

Parallax: Parameterized Local Linear Attention for Language Modeling

Published on May 27

· Submitted by

Yifei Zuo on May 29

Northwestern University

Upvote

Authors:

Yifei Zuo ,

Abstract

Local Linear Attention is enhanced through parameterization and hardware-aware optimization to improve LLM training efficiency and performance while maintaining computational stability.

AI-generated summary

Large Language Models (LLMs) have become the central paradigm in artificial intelligence, yet the core computational primitive of attention has remained structurally unchanged. Local Linear Attention (LLA) is an attention mechanism derived from nonparametric statistics in the test-time regression framework. In contrast to prior research on efficient attention variants, LLA upgrades the local constant estimate in softmax attention to a local linear estimate, yielding provably superior bias-variance tradeoffs for associative memory. However, LLA has not been scaled in LLM pretraining due to computational and numerical stability concerns. We introduce Parallax, a parameterized Local Linear Attention that is scalable for LLMs. Parallax eliminates the numerical solver in LLA and learns an extra query-like projector that probes the KV covariance. We place Parallax within a family of attention mechanisms connected by the bandwidth, the probe construction and the affine structure. We propose a hardware-aware algorithm that increases the arithmetic intensity over FlashAttention, shifting attention into a more compute bound regime. Our prototype decode kernel matches or outperforms FlashAttention 2/3 across diverse batch sizes and context lengths. We pretrain Parallax at 0.6B and 1.7B scales and find consistent perplexity improvements throughout pretraining with gains that transfer to downstream benchmarks. The advantage persists under both parameter-matched and compute-matched controls, demonstrating a Pareto improvement. We perform careful pretraining ablations and identify a novel phenomenon whereby Muon unlocks the capacity of Parallax. To our knowledge, this is the first empirical demonstration of strong architecture-optimizer codesign for attention mechanisms in the architecture research literature.