Hugging Face Daily Papers · June 12, 2026 · 3 min read

MiniMax Sparse Attention

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

our high-performance MSA kernel library is now open-source</p>\n","updatedAt":"2026-06-12T05:29:37.362Z","author":{"_id":"673463f29cb2775fbe62a626","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/673463f29cb2775fbe62a626/64k_zEitiRQD_U002ojyy.jpeg","fullname":"Ryanlee","name":"ryanlee-dev","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":36,"isUserFollowing":false,"primaryOrg":{"avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/676e38ad04af5bec20bc9faf/dUd-LsZEX0H_d4qefO_g6.jpeg","fullname":"MiniMax","name":"MiniMaxAI","type":"org","isHf":false,"details":"Intelligence with Everyone","plan":"team"}}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9124073386192322},"editors":["ryanlee-dev"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/673463f29cb2775fbe62a626/64k_zEitiRQD_U002ojyy.jpeg"],"reactions":[{"reaction":"❤️","users":["komixenon","celikburak","NETZkultur","Szczesny","plaue","Krasyliv"],"count":6}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.13392","authors":[{"_id":"6a2b99244957fcdd3aac0762","name":"Xunhao Lai","hidden":false},{"_id":"6a2b99244957fcdd3aac0763","name":"Weiqi Xu","hidden":false},{"_id":"6a2b99244957fcdd3aac0764","name":"Yufeng Yang","hidden":false},{"_id":"6a2b99244957fcdd3aac0765","name":"Qiaorui Chen","hidden":false},{"_id":"6a2b99244957fcdd3aac0766","name":"Yang Xu","hidden":false},{"_id":"6a2b99244957fcdd3aac0767","name":"Lunbin Zeng","hidden":false},{"_id":"6a2b99244957fcdd3aac0768","name":"Xiaolong Li","hidden":false},{"_id":"6a2b99244957fcdd3aac0769","name":"Haohai Sun","hidden":false},{"_id":"6a2b99244957fcdd3aac076a","name":"Haichao Zhu","hidden":false},{"_id":"6a2b99244957fcdd3aac076b","name":"Vito Zhang","hidden":false},{"_id":"6a2b99244957fcdd3aac076c","name":"Pengyu Zhao","hidden":false}],"publishedAt":"2026-06-11T00:00:00.000Z","submittedOnDailyAt":"2026-06-12T00:00:00.000Z","title":"MiniMax Sparse Attention","submittedOnDailyBy":{"_id":"673463f29cb2775fbe62a626","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/673463f29cb2775fbe62a626/64k_zEitiRQD_U002ojyy.jpeg","isPro":false,"fullname":"Ryanlee","user":"ryanlee-dev","type":"user","name":"ryanlee-dev"},"summary":"Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical speedups, we co-design MSA with a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access. On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4x at 1M context. Paired with our co-designed kernel, MSA achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. Our inference kernel is available at: https://github.com/MiniMax-AI/MSA. A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3.","upvotes":48,"discussionId":"6a2b99244957fcdd3aac076d","githubRepo":"https://github.com/MiniMax-AI/MSA","githubRepoAddedBy":"user","ai_summary":"MiniMax Sparse Attention enables efficient processing of ultra-long contexts in large language models through blockwise sparsity and optimized GPU execution, achieving significant speedups while maintaining performance.","ai_keywords":["sparse attention","Grouped Query Attention","Top-k selection","blockwise sparse attention","tensor-core utilization","prefill","decoding","attention compute","context length"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":145,"organization":{"_id":"6778fc29920093dbc0c24917","name":"MiniMaxAI","fullname":"MiniMax","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/676e38ad04af5bec20bc9faf/dUd-LsZEX0H_d4qefO_g6.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"673463f29cb2775fbe62a626","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/673463f29cb2775fbe62a626/64k_zEitiRQD_U002ojyy.jpeg","isPro":false,"fullname":"Ryanlee","user":"ryanlee-dev","type":"user"},{"_id":"66f64a246b0e782fa331c18b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66f64a246b0e782fa331c18b/bKoDOZP-thrDXEolaSlxS.png","isPro":false,"fullname":"Micko Lesmana","user":"komixenon","type":"user"},{"_id":"6940fc17b4ad5b6ca6c0a426","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6940fc17b4ad5b6ca6c0a426/n-_R0OLuBaF0SuNSiFHi0.jpeg","isPro":false,"fullname":"burak","user":"celikburak","type":"user"},{"_id":"65e7be93a5deaa480d51a88c","avatarUrl":"/avatars/44bf42dbde5ccc64114a36f1cfdad635.svg","isPro":false,"fullname":"Qihang Fan","user":"aldjalkdf","type":"user"},{"_id":"65b8cf96c514fe7afee6e27b","avatarUrl":"/avatars/341d2f54d3914200a8152ac813a00346.svg","isPro":false,"fullname":"mimi","user":"miku678","type":"user"},{"_id":"653f5fdc06a7bb4b1db1bc8c","avatarUrl":"/avatars/e191868c99505c5bd45cd3d1b7323a8d.svg","isPro":false,"fullname":"Jeff","user":"jeffltc","type":"user"},{"_id":"698eeb42a70d792a7dfa0d57","avatarUrl":"/avatars/d8a9e3fccf47f3c78163e4eb7d6e48e4.svg","isPro":false,"fullname":"wxll","user":"wxllhlh","type":"user"},{"_id":"64edc576f172ec32ddd857fc","avatarUrl":"/avatars/f8a5c83c144d12bf586db84bb77e3cf6.svg","isPro":false,"fullname":"Zhou Zijian","user":"bobbyzhou","type":"user"},{"_id":"6843f2585591792864ed3bcf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/OVDhC0SCW-ZJ7KXKiR587.png","isPro":false,"fullname":"boai","user":"Bo-AI","type":"user"},{"_id":"65473b5aaf7c27330d31c9fd","avatarUrl":"/avatars/2f2bc25298c8aeb492dc3d327116ad77.svg","isPro":false,"fullname":"horry","user":"horryhuang","type":"user"},{"_id":"67872a7f94aa7ee32ce8c15b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/avIek_Z84vkWxW87MHmdV.png","isPro":false,"fullname":"clara","user":"clara123","type":"user"},{"_id":"675192d780520a9ab578f4eb","avatarUrl":"/avatars/4155b597debfe05093855e4d972d3852.svg","isPro":false,"fullname":"ZIMA","user":"Chime316","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6778fc29920093dbc0c24917","name":"MiniMaxAI","fullname":"MiniMax","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/676e38ad04af5bec20bc9faf/dUd-LsZEX0H_d4qefO_g6.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.13392.md","query":{}}">

Papers

arxiv:2606.13392

MiniMax Sparse Attention

Published on Jun 11

· Submitted by

Ryanlee on Jun 12

MiniMax

Upvote

Authors:

Abstract

MiniMax Sparse Attention enables efficient processing of ultra-long contexts in large language models through blockwise sparsity and optimized GPU execution, achieving significant speedups while maintaining performance.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical speedups, we co-design MSA with a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access. On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4x at 1M context. Paired with our co-designed kernel, MSA achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. Our inference kernel is available at: https://github.com/MiniMax-AI/MSA. A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3.