Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, making dynamic top-p selection more suitable than fixed top-k sparsification. Based on these insights, we propose RTPurbo, which retains the full KV cache only for retrieval heads and introduces a lightweight token indexer for sparse attention. By exploiting the model's intrinsic sparsity, RTPurbo achieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show that RTPurbo preserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36times prefill speedup at 1M context and about a 2.01times decode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining.</p>\n","updatedAt":"2026-05-22T06:06:30.480Z","author":{"_id":"6409f2eb9989bcb11721d127","avatarUrl":"/avatars/b25cad4b548281942c717a3acda0d96d.svg","fullname":"Richard ZHou","name":"zykRichard","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9039215445518494},"editors":["zykRichard"],"editorAvatarUrls":["/avatars/b25cad4b548281942c717a3acda0d96d.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.16928","authors":[{"_id":"6a0c536165eb30f20d962828","user":{"_id":"6409f2eb9989bcb11721d127","avatarUrl":"/avatars/b25cad4b548281942c717a3acda0d96d.svg","isPro":false,"fullname":"Richard ZHou","user":"zykRichard","type":"user","name":"zykRichard"},"name":"Yanke Zhou","status":"claimed_verified","statusLastChangedAt":"2026-05-20T17:12:43.333Z","hidden":false},{"_id":"6a0c536165eb30f20d962829","name":"Yiduo Li","hidden":false},{"_id":"6a0c536165eb30f20d96282a","name":"Hanlin Tang","hidden":false},{"_id":"6a0c536165eb30f20d96282b","user":{"_id":"671b4660a3fd72a462e97330","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/671b4660a3fd72a462e97330/QbIZR5DqA6sGBvzw0vstV.jpeg","isPro":false,"fullname":"Maohua Li","user":"Met4physics","type":"user","name":"Met4physics"},"name":"Maohua Li","status":"claimed_verified","statusLastChangedAt":"2026-05-20T17:12:40.557Z","hidden":false},{"_id":"6a0c536165eb30f20d96282c","name":"Kan Liu","hidden":false},{"_id":"6a0c536165eb30f20d96282d","name":"Lan Tao","hidden":false},{"_id":"6a0c536165eb30f20d96282e","name":"Lin Qu","hidden":false},{"_id":"6a0c536165eb30f20d96282f","name":"Yuan Yao","hidden":false},{"_id":"6a0c536165eb30f20d962830","name":"Xiaoxing Ma","hidden":false}],"publishedAt":"2026-05-16T00:00:00.000Z","submittedOnDailyAt":"2026-05-22T00:00:00.000Z","title":"Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps","submittedOnDailyBy":{"_id":"6409f2eb9989bcb11721d127","avatarUrl":"/avatars/b25cad4b548281942c717a3acda0d96d.svg","isPro":false,"fullname":"Richard ZHou","user":"zykRichard","type":"user","name":"zykRichard"},"summary":"Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, making dynamic top-p selection more suitable than fixed top-k sparsification. Based on these insights, we propose RTPurbo, which retains the full KV cache only for retrieval heads and introduces a lightweight token indexer for sparse attention. By exploiting the model's intrinsic sparsity, RTPurbo achieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show that RTPurbo preserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36times prefill speedup at 1M context and about a 2.01times decode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining.","upvotes":65,"discussionId":"6a0c536165eb30f20d962831","ai_summary":"RTPurbo leverages intrinsic sparsity in full-attention LLMs to achieve efficient long-context inference with minimal training overhead, enabling significant speedups while maintaining near-lossless accuracy.","ai_keywords":["full attention","long-context inference","attention heads","sparse attention","KV cache","token indexer","dynamic top-p selection","intrinsic sparsity","RTPurbo","prefill speedup","decode speedup"],"organization":{"_id":"6948e7d0a2a90d1cca14cbbc","name":"RTP-LLM","fullname":"RTP-LLM","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6426d1afbc4f1d51f5479914/lgUmPC4DXPxlhRBDnHybm.webp"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"671b4660a3fd72a462e97330","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/671b4660a3fd72a462e97330/QbIZR5DqA6sGBvzw0vstV.jpeg","isPro":false,"fullname":"Maohua Li","user":"Met4physics","type":"user"},{"_id":"6409f2eb9989bcb11721d127","avatarUrl":"/avatars/b25cad4b548281942c717a3acda0d96d.svg","isPro":false,"fullname":"Richard ZHou","user":"zykRichard","type":"user"},{"_id":"68cc1613b0e7121fdac012ee","avatarUrl":"/avatars/b01fcc068d822b90f6017b934dde5922.svg","isPro":false,"fullname":"Mark","user":"MasterMarkk","type":"user"},{"_id":"680724bc034aeb84419ad821","avatarUrl":"/avatars/9e61b71fc31c2392b58da9d155f1aa3c.svg","isPro":false,"fullname":"Wyllat","user":"Kirisyki","type":"user"},{"_id":"670fb308852576f6ffab836d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Ywd7Jw9LGtpKVWUP3_5xP.png","isPro":false,"fullname":"susong","user":"susong-ce","type":"user"},{"_id":"692bdf202b92f5ff3b7ae02e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/o6h-re7fTG7fBC6TbcX7x.png","isPro":false,"fullname":"Sayre Crane","user":"xbchhhh","type":"user"},{"_id":"66850b725e7c4d4b77f987e2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66850b725e7c4d4b77f987e2/Qn_6eCoyiB5-zE8anYOMd.png","isPro":false,"fullname":"Weizhong Huang","user":"huangwz","type":"user"},{"_id":"67d198a6ab69f5bd172fae3d","avatarUrl":"/avatars/e1754740281eed541a67365ce4acf4db.svg","isPro":false,"fullname":"Yong-Ming Tian","user":"TYM666","type":"user"},{"_id":"65745569839aa08899ea5d27","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4X8waDwiphbfKZySrYlFy.jpeg","isPro":false,"fullname":"Kailin Jiang","user":"kailinjiang","type":"user"},{"_id":"67d279948f1c958d671754e9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/oEjdAjQhZxK1VdSfbiK-x.png","isPro":false,"fullname":"Xiang-Sheng Deng","user":"dengxs1129","type":"user"},{"_id":"65818ecd20e57e0ebf5d90e9","avatarUrl":"/avatars/bd80a92013d930acf3fbc946d3a2bf67.svg","isPro":false,"fullname":"Richard Lee","user":"lixin4sky","type":"user"},{"_id":"660165de9e1cf5eb41fe4b0a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/660165de9e1cf5eb41fe4b0a/rpNxle6Px04AFTAomec0k.jpeg","isPro":false,"fullname":"Qianqian Xie","user":"mistletoe111","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":3,"organization":{"_id":"6948e7d0a2a90d1cca14cbbc","name":"RTP-LLM","fullname":"RTP-LLM","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6426d1afbc4f1d51f5479914/lgUmPC4DXPxlhRBDnHybm.webp"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.16928.md"}">
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
Abstract
RTPurbo leverages intrinsic sparsity in full-attention LLMs to achieve efficient long-context inference with minimal training overhead, enabling significant speedups while maintaining near-lossless accuracy.
AI-generated summary
Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, making dynamic top-p selection more suitable than fixed top-k sparsification. Based on these insights, we propose RTPurbo, which retains the full KV cache only for retrieval heads and introduces a lightweight token indexer for sparse attention. By exploiting the model's intrinsic sparsity, RTPurbo achieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show that RTPurbo preserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36times prefill speedup at 1M context and about a 2.01times decode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining.
Community
Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, making dynamic top-p selection more suitable than fixed top-k sparsification. Based on these insights, we propose RTPurbo, which retains the full KV cache only for retrieval heads and introduces a lightweight token indexer for sparse attention. By exploiting the model's intrinsic sparsity, RTPurbo achieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show that RTPurbo preserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36times prefill speedup at 1M context and about a 2.01times decode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.16928 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.16928 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.16928 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.