Hugging Face Daily Papers · August 11, 2026 · 4 min read

OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

We proposed OasisKV, a KV-cache prefetching framework, to expand effective in-decode memory capacity using off-GPU memory.</p>\n","updatedAt":"2026-08-11T04:52:44.197Z","author":{"_id":"64ec4c04c782d648d28d70fc","avatarUrl":"/avatars/6975526fcf4b513cc934b5bc45370a48.svg","fullname":"Sukmin Cho","name":"zomss","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7398211359977722},"editors":["zomss"],"editorAvatarUrls":["/avatars/6975526fcf4b513cc934b5bc45370a48.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.08097","authors":[{"_id":"6a7a9b3c019ce76dc7b3aa9a","name":"Can Xiao","hidden":false},{"_id":"6a7a9b3c019ce76dc7b3aa9b","name":"Sukmin Cho","hidden":false},{"_id":"6a7a9b3c019ce76dc7b3aa9c","name":"Junbong We","hidden":false},{"_id":"6a7a9b3c019ce76dc7b3aa9d","name":"Zhixiong Niu","hidden":false},{"_id":"6a7a9b3c019ce76dc7b3aa9e","name":"Jianyi Cheng","hidden":false},{"_id":"6a7a9b3c019ce76dc7b3aa9f","name":"Yiren Zhao","hidden":false},{"_id":"6a7a9b3c019ce76dc7b3aaa0","name":"Youngjin Kwon","hidden":false},{"_id":"6a7a9b3c019ce76dc7b3aaa1","name":"Yongqiang Xiong","hidden":false},{"_id":"6a7a9b3c019ce76dc7b3aaa2","name":"Rui Ma","hidden":false},{"_id":"6a7a9b3c019ce76dc7b3aaa3","name":"Junyi Liu","hidden":false}],"publishedAt":"2026-08-08T00:00:00.000Z","submittedOnDailyAt":"2026-08-11T00:00:00.000Z","title":"OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching","submittedOnDailyBy":{"_id":"64ec4c04c782d648d28d70fc","avatarUrl":"/avatars/6975526fcf4b513cc934b5bc45370a48.svg","isPro":false,"fullname":"Sukmin Cho","user":"zomss","type":"user","name":"zomss"},"summary":"Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation, i.e., decode. In particular, HBM capacity has become a scarce and costly resource that heavily limits inference batch size and system throughput. This paper presents OasisKV, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding. Because decode-time attention is naturally sparse, OasisKV keeps only the KV entries of the most relevant tokens in HBMs for attention computation. We observe that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD). OasisKV employs an efficient attention background pipeline to identify important KV blocks. They are then prefetched from higher-capacity memory tiers (e.g., host or remote memory) and staged in HBMs before being used in the next decode step.\n We implement OasisKV based on vLLM. The lookahead prediction is accurate enough to keep accuracy within 0.7 points of full attention under a 2,048-token KV budget. This lets OasisKV turn sparsity into throughput gain: 1.69times over dense vLLM on the reasoning workload at 0.1 points of accuracy loss, and up to 2.1times on multi-GPU long-context serving. Under prefill--decode disaggregation, OasisKV reaches about 2times dense throughput while admitting each request with 6.5--9.7times less KV and holding 2.2-2.6 less decode-node host memory than full KV transfer.","upvotes":16,"discussionId":"6a7a9b3d019ce76dc7b3aaa4","ai_summary":"OasisKV improves LLM inference throughput by storing full KV caches in lower memory tiers and prefetching only relevant entries into HBM using speculative-decoding lookahead predictions.","ai_keywords":["large language model inference","KV cache","speculative decoding","lookahead tokens","attention sparsity","memory-centric inference","HBM capacity","vLLM","prefill-decode disaggregation"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"5e6485f787403103f9f1055e","name":"microsoft","fullname":"Microsoft","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1583646260758-5e64858c87403103f9f1055d.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64ec4c04c782d648d28d70fc","avatarUrl":"/avatars/6975526fcf4b513cc934b5bc45370a48.svg","isPro":false,"fullname":"Sukmin Cho","user":"zomss","type":"user"},{"_id":"64e5a1cd4c20016ec9020ec8","avatarUrl":"/avatars/d7ffe7fbbe39c0a013375357457c57b3.svg","isPro":false,"fullname":"Soyeong","user":"starsuzi","type":"user"},{"_id":"665868596e4d9a66a1b2c779","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ZiZpk_ZEjqSoBEj6rWgI_.png","isPro":false,"fullname":"SeungYoon Han","user":"seungyoonee","type":"user"},{"_id":"668cc6784587d0a82bc3b9c9","avatarUrl":"/avatars/01b46a6064a07e06677fa3cf1335b6f6.svg","isPro":false,"fullname":"Changgeon Ko","user":"pencaty","type":"user"},{"_id":"69a6d760e9106065ace4f75a","avatarUrl":"/avatars/17a6b92d4075e1ae5006afb9dc863b7d.svg","isPro":false,"fullname":"Jin Heo","user":"jinheo03","type":"user"},{"_id":"659384d933d72b44f0382c3e","avatarUrl":"/avatars/d8e5c8f6ca310ad5978fbd552791d40e.svg","isPro":false,"fullname":"Euijun Hwang","user":"Eddie-Hwang","type":"user"},{"_id":"5f03f5d5ff69163f6f59e9e7","avatarUrl":"/avatars/4b87aa1f0a52dab3826176b4a43394f3.svg","isPro":false,"fullname":"angi","user":"angiquer","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"676a387aef4c8dc3130b219e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vZjcTX7fAdTdgICLsoX9j.png","isPro":false,"fullname":"Jisu Shin","user":"ddindidu","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"68ef18625730be02088836bf","avatarUrl":"/avatars/7f0d70a518e86ab37dc00f659befb4bf.svg","isPro":false,"fullname":"Ray","user":"WalnetCracker","type":"user"},{"_id":"65f151d065610aef6477269c","avatarUrl":"/avatars/f8e786bbd3a216d89d753a572bee86b9.svg","isPro":false,"fullname":"Sangjin Choi","user":"sjchoi1111","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"5e6485f787403103f9f1055e","name":"microsoft","fullname":"Microsoft","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1583646260758-5e64858c87403103f9f1055d.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.08097.md","query":{}}">

Papers

arxiv:2608.08097

OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

Published on Aug 8

· Submitted by

Sukmin Cho on Aug 11

Microsoft

Upvote

Authors:

Abstract

OasisKV improves LLM inference throughput by storing full KV caches in lower memory tiers and prefetching only relevant entries into HBM using speculative-decoding lookahead predictions.

Generated by thinkingmachines/Inkling-Small

Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation, i.e., decode. In particular, HBM capacity has become a scarce and costly resource that heavily limits inference batch size and system throughput. This paper presents OasisKV, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding. Because decode-time attention is naturally sparse, OasisKV keeps only the KV entries of the most relevant tokens in HBMs for attention computation. We observe that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD). OasisKV employs an efficient attention background pipeline to identify important KV blocks. They are then prefetched from higher-capacity memory tiers (e.g., host or remote memory) and staged in HBMs before being used in the next decode step. We implement OasisKV based on vLLM. The lookahead prediction is accurate enough to keep accuracy within 0.7 points of full attention under a 2,048-token KV budget. This lets OasisKV turn sparsity into throughput gain: 1.69times over dense vLLM on the reasoning workload at 0.1 points of accuracy loss, and up to 2.1times on multi-GPU long-context serving. Under prefill--decode disaggregation, OasisKV reaches about 2times dense throughput while admitting each request with 6.5--9.7times less KV and holding 2.2-2.6 less decode-node host memory than full KV transfer.

View arXiv page View PDF Add to collection