<a href=\"https://cdn-uploads.huggingface.co/production/uploads/673f1513d8ab5f2d8d1a3475/U-cOtetSHOEqmd7Fs_GZ8.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/673f1513d8ab5f2d8d1a3475/U-cOtetSHOEqmd7Fs_GZ8.png\" alt=\"image\"></a></p>\n","updatedAt":"2026-06-26T15:55:14.071Z","author":{"_id":"673f1513d8ab5f2d8d1a3475","avatarUrl":"/avatars/6e62d945ef85e2dd7644f5af57fe7ce9.svg","fullname":"kai","name":"jsonopen1","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.4708739221096039},"editors":["jsonopen1"],"editorAvatarUrls":["/avatars/6e62d945ef85e2dd7644f5af57fe7ce9.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.26875","authors":[{"_id":"6a3e9fd70dbbc53604b66497","name":"Jushi Kai","hidden":false},{"_id":"6a3e9fd70dbbc53604b66498","name":"Zhuiri Xiao","hidden":false},{"_id":"6a3e9fd70dbbc53604b66499","name":"Alexandra Birch","hidden":false},{"_id":"6a3e9fd70dbbc53604b6649a","name":"Zhouhan Lin","hidden":false}],"publishedAt":"2026-06-25T00:00:00.000Z","submittedOnDailyAt":"2026-06-26T00:00:00.000Z","title":"Information-Aware KV Cache Compression for Long Reasoning","submittedOnDailyBy":{"_id":"673f1513d8ab5f2d8d1a3475","avatarUrl":"/avatars/6e62d945ef85e2dd7644f5af57fe7ce9.svg","isPro":false,"fullname":"kai","user":"jsonopen1","type":"user","name":"jsonopen1"},"summary":"Reasoning capability has advanced rapidly in large language models (LLMs), leading to an increasing size of key-value (KV) cache in both prefilling and decoding stages. Existing KV cache compression methods mainly rely on attention weights to estimate token importance. While attention effectively captures contextual relevance, it overlooks complementary information-theoretic signals related to predictive uncertainty and token informativeness. In this paper, we revisit token importance from a forward-looking perspective and introduce Forward Influence, a metric that measures how compressed tokens affect future contexts. Our analysis reveals that tokens selected by attention scores mainly influence nearby contexts, whereas tokens associated with high predictive uncertainty exhibit substantially stronger influence on distant future contexts. Based on the observation, we propose InfoKV, an entropy-aware KV cache compression framework that incorporates information-theoretic signals. It combines token-level predictive uncertainty with layer-wise representation evolution and integrates the resulting entropy scores with attention scores during reasoning. Experiments on long-context reasoning benchmarks with Llama-3.1, Llama-3.2, and DeepSeek-R1 demonstrate that InfoKV consistently outperforms existing attention-based KV compression methods in both long prefilling and decoding scenarios.","upvotes":9,"discussionId":"6a3e9fd70dbbc53604b6649b","ai_summary":"InfoKV is an entropy-aware KV cache compression framework that enhances long-context reasoning in LLMs by incorporating information-theoretic signals alongside attention weights.","ai_keywords":["KV cache","attention weights","token importance","predictive uncertainty","information-theoretic signals","Forward Influence","entropy-aware","long-context reasoning","LLMs","KV cache compression"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"673f1513d8ab5f2d8d1a3475","avatarUrl":"/avatars/6e62d945ef85e2dd7644f5af57fe7ce9.svg","isPro":false,"fullname":"kai","user":"jsonopen1","type":"user"},{"_id":"672244bb68ffb1aa63d98919","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Pg6l1js7VErkx3mXXzDr2.jpeg","isPro":false,"fullname":"paitsai","user":"Ryuuka917","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"},{"_id":"69f0bb9a53592156859aab90","avatarUrl":"/avatars/122aeb140c584b7842c50ae693c2a27e.svg","isPro":false,"fullname":"mini09999","user":"mini09999","type":"user"},{"_id":"696da0962b3e2d9587d0b35d","avatarUrl":"/avatars/4f6c177ad51fb687ca1be75d18f6f5d6.svg","isPro":false,"fullname":"mini","user":"mini0999","type":"user"},{"_id":"6725000a5854a657e8c98c9e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/glc1BF9Sn2FvSrShDQNjc.png","isPro":false,"fullname":"BunnyLutts","user":"BunnyLutts","type":"user"},{"_id":"69dcd9ed7280ac222b59a41d","avatarUrl":"/avatars/383e5a489ec9120a648b98d0a3e50b3d.svg","isPro":false,"fullname":"Jingzhi Wang","user":"jzwang666","type":"user"},{"_id":"66458107219ad12f47bc8fd4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66458107219ad12f47bc8fd4/8NqMRPB2Ko4GIOtL7ZzOj.jpeg","isPro":false,"fullname":"Yixuan Wang","user":"LuckyOrz","type":"user"},{"_id":"64c38a08e9e1506c56790f6d","avatarUrl":"/avatars/ea8fd3e97fdfbdb230fabd59d9ebe1b7.svg","isPro":false,"fullname":"Shixiang Song(SII)","user":"sxsong","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.26875.md","query":{}}">
Information-Aware KV Cache Compression for Long Reasoning
Published on Jun 25
· Submitted by kai on Jun 26 Abstract
InfoKV is an entropy-aware KV cache compression framework that enhances long-context reasoning in LLMs by incorporating information-theoretic signals alongside attention weights.
Reasoning capability has advanced rapidly in large language models (LLMs), leading to an increasing size of key-value (KV) cache in both prefilling and decoding stages. Existing KV cache compression methods mainly rely on attention weights to estimate token importance. While attention effectively captures contextual relevance, it overlooks complementary information-theoretic signals related to predictive uncertainty and token informativeness. In this paper, we revisit token importance from a forward-looking perspective and introduce Forward Influence, a metric that measures how compressed tokens affect future contexts. Our analysis reveals that tokens selected by attention scores mainly influence nearby contexts, whereas tokens associated with high predictive uncertainty exhibit substantially stronger influence on distant future contexts. Based on the observation, we propose InfoKV, an entropy-aware KV cache compression framework that incorporates information-theoretic signals. It combines token-level predictive uncertainty with layer-wise representation evolution and integrates the resulting entropy scores with attention scores during reasoning. Experiments on long-context reasoning benchmarks with Llama-3.1, Llama-3.2, and DeepSeek-R1 demonstrate that InfoKV consistently outperforms existing attention-based KV compression methods in both long prefilling and decoding scenarios.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.26875 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.26875 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.26875 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.