Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.</p>\n","updatedAt":"2026-06-03T04:00:25.325Z","author":{"_id":"63c8454e46421a2efe82709d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/4k_HR5F6cxkVH1b5KEsGi.webp","fullname":"Deqing Fu","name":"deqing","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":13,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9082589149475098},"editors":["deqing"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/4k_HR5F6cxkVH1b5KEsGi.webp"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.03928","authors":[{"_id":"6a1fa69be292c1c78ecb1389","name":"Ting-Yun Chang","hidden":false},{"_id":"6a1fa69be292c1c78ecb138a","name":"Harvey Yiyun Fu","hidden":false},{"_id":"6a1fa69be292c1c78ecb138b","name":"Deqing Fu","hidden":false},{"_id":"6a1fa69be292c1c78ecb138c","name":"Chenghao Yang","hidden":false},{"_id":"6a1fa69be292c1c78ecb138d","name":"Jesse Thomason","hidden":false},{"_id":"6a1fa69be292c1c78ecb138e","name":"Robin Jia","hidden":false}],"publishedAt":"2026-06-02T17:16:33.000Z","submittedOnDailyAt":"2026-06-03T00:00:00.000Z","title":"Value-Aware Stochastic KV Cache Eviction for Reasoning Models","submittedOnDailyBy":{"_id":"63c8454e46421a2efe82709d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/4k_HR5F6cxkVH1b5KEsGi.webp","isPro":true,"fullname":"Deqing Fu","user":"deqing","type":"user","name":"deqing"},"summary":"Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.","upvotes":3,"discussionId":"6a1fa69be292c1c78ecb138f","githubRepo":"https://github.com/terarachang/VaSE","githubRepoAddedBy":"user","ai_summary":"Value-aware stochastic KV cache eviction method improves reasoning model accuracy under compression by protecting large-magnitude states and promoting diverse eviction decisions.","ai_keywords":["KV cache eviction","sparse attention","reasoning models","FlashAttention2","value states","stochasticity","cache diversity"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0,"organization":{"_id":"66a403d0dcb5bbc6e98bb7d0","name":"UniversityofSouthernCalifornia","fullname":"University of Southern California","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a403728069e3c30e0d8524/tkYCfeIJfF1FxtYiRZ8bf.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63c8454e46421a2efe82709d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/4k_HR5F6cxkVH1b5KEsGi.webp","isPro":true,"fullname":"Deqing Fu","user":"deqing","type":"user"},{"_id":"62a9713c3d8a300f79f99379","avatarUrl":"/avatars/6cfed0d4f0d827763e5d83de213e3a3b.svg","isPro":false,"fullname":"Harvey Fu","user":"harveyfin","type":"user"},{"_id":"63bf2e3387619d1458cd1509","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63bf2e3387619d1458cd1509/qEMzIE10sYf7FC4qkh-Vn.jpeg","isPro":false,"fullname":"Xinyan Velocity Yu","user":"velocivirus","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"66a403d0dcb5bbc6e98bb7d0","name":"UniversityofSouthernCalifornia","fullname":"University of Southern California","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66a403728069e3c30e0d8524/tkYCfeIJfF1FxtYiRZ8bf.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.03928.md"}">
Value-Aware Stochastic KV Cache Eviction for Reasoning Models
Abstract
Value-aware stochastic KV cache eviction method improves reasoning model accuracy under compression by protecting large-magnitude states and promoting diverse eviction decisions.
Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.
Community
Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.03928 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.03928 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.03928 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.