Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.</p>\n","updatedAt":"2026-06-23T11:26:06.325Z","author":{"_id":"638828121901766b88076aa1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/638828121901766b88076aa1/rXlOO7eewmmaSN_hQIVz7.jpeg","fullname":"Vishesh Tripathi","name":"vishesh-t27","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":15,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8467386364936829},"editors":["vishesh-t27"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/638828121901766b88076aa1/rXlOO7eewmmaSN_hQIVz7.jpeg"],"reactions":[{"reaction":"❤️","users":["Alok277373","singhaks","udayallu","aksinghyaani","akanyaani"],"count":5}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.20945","authors":[{"_id":"6a3a6d28fdcd3514343bb834","name":"Vishesh Tripathi","hidden":false},{"_id":"6a3a6d28fdcd3514343bb835","name":"Abhay Kumar","hidden":false}],"publishedAt":"2026-06-18T00:00:00.000Z","submittedOnDailyAt":"2026-06-23T00:00:00.000Z","title":"Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention","submittedOnDailyBy":{"_id":"638828121901766b88076aa1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/638828121901766b88076aa1/rXlOO7eewmmaSN_hQIVz7.jpeg","isPro":false,"fullname":"Vishesh Tripathi","user":"vishesh-t27","type":"user","name":"vishesh-t27"},"summary":"Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.","upvotes":42,"discussionId":"6a3a6d29fdcd3514343bb836","ai_summary":"Grouped Query Experts (GQE) improves Transformer efficiency by selectively activating query heads based on token content while maintaining key-value cache benefits of grouped-query attention.","ai_keywords":["self-attention","Transformer","attention heads","query-head experts","grouped-query attention","mixture-of-experts","router","token interactions","attention cost","computational efficiency"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","organization":{"_id":"6968e79166053d81c63a0285","name":"FrontiersMind","fullname":"FrontiersMind","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/638828121901766b88076aa1/e2jdcDUMUF0ZD9BGViHuo.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"638828121901766b88076aa1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/638828121901766b88076aa1/rXlOO7eewmmaSN_hQIVz7.jpeg","isPro":false,"fullname":"Vishesh Tripathi","user":"vishesh-t27","type":"user"},{"_id":"62cd4b03c5cc157be82f0b56","avatarUrl":"/avatars/351e963c1c763d507ae78cbcd62966a3.svg","isPro":false,"fullname":"Abhay kumar","user":"akanyaani","type":"user"},{"_id":"6a0983809fc62a36e1d8d3f1","avatarUrl":"/avatars/8e919275d792689c29172862fa809a88.svg","isPro":false,"fullname":"Aakriti Singh","user":"aakritisinghfml","type":"user"},{"_id":"69cfe22dc0000c3e06060523","avatarUrl":"/avatars/5f8bd8a5139c66ecf3d661a556eb1206.svg","isPro":false,"fullname":"VSH","user":"vaishnofarm","type":"user"},{"_id":"67effec26dca39c7068cb352","avatarUrl":"/avatars/0da6c246be950c3a5d107931a61cc080.svg","isPro":false,"fullname":"abhay singh","user":"singhaks","type":"user"},{"_id":"67f01aac25628a9c32a8360d","avatarUrl":"/avatars/9d4b57c0361740f28f2912b51bd0bae7.svg","isPro":false,"fullname":"alex","user":"alexalml","type":"user"},{"_id":"67f019fd2c873f5ba9eacf36","avatarUrl":"/avatars/cdae026cf9b3672ab70229b1b50ee5a2.svg","isPro":false,"fullname":"Shashi Ranjan","user":"shashiaiml","type":"user"},{"_id":"6a0c2f87cdd91f803a7a1500","avatarUrl":"/avatars/a94ea6a101809aad9424f6a558117ce4.svg","isPro":false,"fullname":"Alok anyaani","user":"Alok277373","type":"user"},{"_id":"69ce6420847b84833d5ee3f5","avatarUrl":"/avatars/0457af704ff1bfe5a6468b0622e51378.svg","isPro":false,"fullname":"AK Singh","user":"aksinghyaani","type":"user"},{"_id":"6a0c2e46161eee21e13eea3b","avatarUrl":"/avatars/7f35169ef55e9170ee842fbbd0190c14.svg","isPro":false,"fullname":"Alok anyaani","user":"Aloksingh477","type":"user"},{"_id":"6a0c2cee52c0d1dc7482717d","avatarUrl":"/avatars/5a9170036cab38ec70306344bf6a5445.svg","isPro":false,"fullname":"Alok kumar","user":"alokkrty","type":"user"},{"_id":"69d0bbc0c7adf5a1e6b213aa","avatarUrl":"/avatars/d7e71bca5f7e01891e0d2e08273c35ca.svg","isPro":false,"fullname":"Abhishek Kumar","user":"ranasinghkumar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6968e79166053d81c63a0285","name":"FrontiersMind","fullname":"FrontiersMind","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/638828121901766b88076aa1/e2jdcDUMUF0ZD9BGViHuo.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.20945.md","query":{}}">
Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
Abstract
Grouped Query Experts (GQE) improves Transformer efficiency by selectively activating query heads based on token content while maintaining key-value cache benefits of grouped-query attention.
Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.
Community
Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.20945 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.20945 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.20945 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.