Hugging Face Daily Papers · July 8, 2026 · 4 min read

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

A fully differentiable hierachical sparse attention with extraordinary length generalization ability.</p>\n","updatedAt":"2026-07-08T03:45:34.653Z","author":{"_id":"64992836929f17f581559e9b","avatarUrl":"/avatars/d478a82c64900405c888d2eecabfc495.svg","fullname":"Lei ZHU","name":"rayleizhu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":0,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.832429826259613},"editors":["rayleizhu"],"editorAvatarUrls":["/avatars/d478a82c64900405c888d2eecabfc495.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.02980","authors":[{"_id":"6a4dc5db25849b193a834bb6","name":"Xiang Hu","hidden":false},{"_id":"6a4dc5db25849b193a834bb7","name":"Xinyu Wei","hidden":false},{"_id":"6a4dc5db25849b193a834bb8","name":"Hao Gu","hidden":false},{"_id":"6a4dc5db25849b193a834bb9","name":"Minshen Zhang","hidden":false},{"_id":"6a4dc5db25849b193a834bba","name":"Tian Liang","hidden":false},{"_id":"6a4dc5db25849b193a834bbb","name":"Huayang Li","hidden":false},{"_id":"6a4dc5db25849b193a834bbc","name":"Lei Zhu","hidden":false},{"_id":"6a4dc5db25849b193a834bbd","name":"Yan Wang","hidden":false},{"_id":"6a4dc5db25849b193a834bbe","name":"Sirui Han","hidden":false},{"_id":"6a4dc5db25849b193a834bbf","name":"Yushi Bai","hidden":false},{"_id":"6a4dc5db25849b193a834bc0","name":"Kewei Tu","hidden":false},{"_id":"6a4dc5db25849b193a834bc1","name":"Haitao Mi","hidden":false},{"_id":"6a4dc5db25849b193a834bc2","name":"Leo Liang","hidden":false}],"publishedAt":"2026-07-03T00:00:00.000Z","submittedOnDailyAt":"2026-07-08T00:00:00.000Z","title":"Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling","submittedOnDailyBy":{"_id":"64992836929f17f581559e9b","avatarUrl":"/avatars/d478a82c64900405c888d2eecabfc495.svg","isPro":false,"fullname":"Lei ZHU","user":"rayleizhu","type":"user","name":"rayleizhu"},"summary":"Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse attention offers a promising alternative, but all existing methods fall short of full attention because of their inaccurate chunk selection. We propose Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss. HiLS factorizes attention hierarchically: each query performs attention independently with each retrieved chunk to extract chunk-specific information, and the resulting outputs are fused according to chunk retrieval scores. By incorporating retrieval scores into the forward attention computation, HiLS optimizes them directly with the LM loss, enabling end-to-end retrieval learning and native sparse training. Experimental results show that HiLS-Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths. Meanwhile, HiLS-Attention extrapolates more than 64times the training context length with 90% retrieval accuracy, far beyond full attention. Moreover, existing full-attention models can be converted to HiLS-Attention with lightweight continued pretraining, preserving in-domain performance while acquiring ultra-long-context extrapolation. Together with its sparse KV access and computation, HiLS-Attention breaks the usual efficiency-performance trade-off, enabling long-context LLMs that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.","upvotes":33,"discussionId":"6a4dc5dc25849b193a834bc3","githubRepo":"https://github.com/Tencent-Hunyuan/HiLS-Attention","githubRepoAddedBy":"user","ai_summary":"Hierarchical Landmark Sparse Attention enables efficient long-context language modeling by learning chunk selection end-to-end, achieving performance comparable to full attention while extrapolating beyond training context lengths.","ai_keywords":["large language models","dense attention","chunk-wise sparse attention","hierarchical landmark sparse attention","language-modeling loss","attention mechanism","retrieval scores","end-to-end learning","sparse KV access","long-context extrapolation"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":23,"organization":{"_id":"6645f953c39288df638dbdd5","name":"Tencent-Hunyuan","fullname":"Tencent Hunyuan","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/62d22496c58f969c152bcefd/woKSjt2wXvBNKussyYPsa.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64992836929f17f581559e9b","avatarUrl":"/avatars/d478a82c64900405c888d2eecabfc495.svg","isPro":false,"fullname":"Lei ZHU","user":"rayleizhu","type":"user"},{"_id":"643645baaa4211ef553f613c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/643645baaa4211ef553f613c/svUVeTqwLf5ZurprdTOUC.jpeg","isPro":false,"fullname":"TimLeung","user":"skytliang","type":"user"},{"_id":"64ed998b2d5805cb99894a04","avatarUrl":"/avatars/b58635ad5b664c8319af7192c26f1c2e.svg","isPro":false,"fullname":"QZX","user":"zexuanqiu22","type":"user"},{"_id":"665aaee7dd6389e91a8ca246","avatarUrl":"/avatars/ea5ec491854f61da3a7dbfe51b0acf73.svg","isPro":false,"fullname":"wxy","user":"weixy","type":"user"},{"_id":"65086ec114302b1d76ea3841","avatarUrl":"/avatars/8796fc15b361c48cca71255a4e261a7c.svg","isPro":false,"fullname":"Xiang Hu","user":"imhuim982","type":"user"},{"_id":"64fed23f0871bc5930598ab5","avatarUrl":"/avatars/080a4ef3e4634cd978528dfa899a4eb0.svg","isPro":false,"fullname":"ZhiWei LI","user":"Aragonaa","type":"user"},{"_id":"6321ae29a97afe3c4c647ffb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1677222836363-6321ae29a97afe3c4c647ffb.png","isPro":false,"fullname":"Zilin Zhu","user":"zhuzilin","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"641129818573c51c0458b793","avatarUrl":"/avatars/d4bc67c160a07146cf41c614678aa36b.svg","isPro":false,"fullname":"Tianqing Fang","user":"tqfang229","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6465dac7e9906a259f31ce9a","avatarUrl":"/avatars/a9bd8807a484822085a93302a37f08df.svg","isPro":false,"fullname":"DONGRYEOLLEE","user":"drlee1","type":"user"},{"_id":"64324f6571bf2c8bcf6fc359","avatarUrl":"/avatars/2f9748997161e7377bbeee19a087e5d7.svg","isPro":false,"fullname":"Minshen Zhang","user":"alexzms","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6645f953c39288df638dbdd5","name":"Tencent-Hunyuan","fullname":"Tencent Hunyuan","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/62d22496c58f969c152bcefd/woKSjt2wXvBNKussyYPsa.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.02980.md","query":{}}">

Papers

arxiv:2607.02980

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

Published on Jul 3

· Submitted by

Lei ZHU on Jul 8

Tencent Hunyuan

Upvote

Authors:

Abstract

Hierarchical Landmark Sparse Attention enables efficient long-context language modeling by learning chunk selection end-to-end, achieving performance comparable to full attention while extrapolating beyond training context lengths.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse attention offers a promising alternative, but all existing methods fall short of full attention because of their inaccurate chunk selection. We propose Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss. HiLS factorizes attention hierarchically: each query performs attention independently with each retrieved chunk to extract chunk-specific information, and the resulting outputs are fused according to chunk retrieval scores. By incorporating retrieval scores into the forward attention computation, HiLS optimizes them directly with the LM loss, enabling end-to-end retrieval learning and native sparse training. Experimental results show that HiLS-Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths. Meanwhile, HiLS-Attention extrapolates more than 64times the training context length with 90% retrieval accuracy, far beyond full attention. Moreover, existing full-attention models can be converted to HiLS-Attention with lightweight continued pretraining, preserving in-domain performance while acquiring ultra-long-context extrapolation. Together with its sparse KV access and computation, HiLS-Attention breaks the usual efficiency-performance trade-off, enabling long-context LLMs that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.