SeKV makes long-context LLM inference more efficient by organizing the KV cache into semantic spans and dynamically reconstructing only the relevant information, improving long-context performance while substantially reducing GPU memory usage.</p>\n","updatedAt":"2026-07-07T17:21:55.180Z","author":{"_id":"60e32baedc56466240084155","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60e32baedc56466240084155/Q-cJh3Q3-vvMbe749Gt5B.jpeg","fullname":"Amirhossein Abaskohi","name":"AmirhosseinAbaskohi","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":5,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8347264528274536},"editors":["AmirhosseinAbaskohi"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/60e32baedc56466240084155/Q-cJh3Q3-vvMbe749Gt5B.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.31145","authors":[{"_id":"6a4d356225849b193a834927","name":"Amirhossein Abaskohi","hidden":false},{"_id":"6a4d356225849b193a834928","name":"Giuseppe Carenini","hidden":false},{"_id":"6a4d356225849b193a834929","name":"Peter West","hidden":false},{"_id":"6a4d356225849b193a83492a","name":"Yuhang He","hidden":false}],"publishedAt":"2026-06-30T00:00:00.000Z","submittedOnDailyAt":"2026-07-07T00:00:00.000Z","title":"SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference","submittedOnDailyBy":{"_id":"60e32baedc56466240084155","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60e32baedc56466240084155/Q-cJh3Q3-vvMbe749Gt5B.jpeg","isPro":false,"fullname":"Amirhossein Abaskohi","user":"AmirhosseinAbaskohi","type":"user","name":"AmirhosseinAbaskohi"},"summary":"Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.","upvotes":6,"discussionId":"6a4d356225849b193a83492b","githubRepo":"https://github.com/AmirAbaskohi/SeKV","githubRepoAddedBy":"user","ai_summary":"SeKV introduces a resolution-adaptive semantic KV cache that compresses context into entropy-guided spans stored across GPU-CPU memory hierarchies, enabling efficient long-context processing with minimal memory overhead and preserved token-level detail.","ai_keywords":["KV cache","semantic spans","entropy-guided","GPU-CPU memory hierarchy","low-rank SVD basis","zoom-in mechanism","token-level reconstruction","context compression","long-context processing"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":2,"organization":{"_id":"5e6485f787403103f9f1055e","name":"microsoft","fullname":"Microsoft","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1583646260758-5e64858c87403103f9f1055d.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"60e32baedc56466240084155","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60e32baedc56466240084155/Q-cJh3Q3-vvMbe749Gt5B.jpeg","isPro":false,"fullname":"Amirhossein Abaskohi","user":"AmirhosseinAbaskohi","type":"user"},{"_id":"6955c60b454ba27bc5c5fc66","avatarUrl":"/avatars/6d34879d63ee9d1686870403358b567b.svg","isPro":false,"fullname":"He","user":"yuhanghe01","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6a147fd68e1e41b41f115600","avatarUrl":"/avatars/7708be212974200a5bdefaa3f7af9ed4.svg","isPro":false,"fullname":"诗雨 梁","user":"isaacmartinbj","type":"user"},{"_id":"69bcde629267623e872d9681","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uT07rNDNnY2Qaazbrkud-.png","isPro":false,"fullname":"木村七海","user":"julianhill45","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"5e6485f787403103f9f1055e","name":"microsoft","fullname":"Microsoft","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1583646260758-5e64858c87403103f9f1055d.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.31145.md","query":{}}">
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
Abstract
SeKV introduces a resolution-adaptive semantic KV cache that compresses context into entropy-guided spans stored across GPU-CPU memory hierarchies, enabling efficient long-context processing with minimal memory overhead and preserved token-level detail.
Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.
Community
SeKV makes long-context LLM inference more efficient by organizing the KV cache into semantic spans and dynamically reconstructing only the relevant information, improving long-context performance while substantially reducing GPU memory usage.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.31145 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.31145 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.31145 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.