Graft is a training-free speculative decoding framework that combines pruning and token retrieval to improve acceptance rates and accelerate large language model inference without incurring additional computational overhead.</p>\n","updatedAt":"2026-05-20T02:12:21.902Z","author":{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","fullname":"taesiri","name":"taesiri","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":301,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8452370166778564},"editors":["taesiri"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.20104","authors":[{"_id":"6a0d186d65eb30f20d962bb9","name":"Yuhao Shen","hidden":false},{"_id":"6a0d186d65eb30f20d962bba","name":"Tianyu Liu","hidden":false},{"_id":"6a0d186d65eb30f20d962bbb","name":"Xinyi Hu","hidden":false},{"_id":"6a0d186d65eb30f20d962bbc","name":"Quan Kong","hidden":false},{"_id":"6a0d186d65eb30f20d962bbd","name":"Baolin Zhang","hidden":false},{"_id":"6a0d186d65eb30f20d962bbe","name":"Jun Dai","hidden":false},{"_id":"6a0d186d65eb30f20d962bbf","name":"Jun Zhang","hidden":false},{"_id":"6a0d186d65eb30f20d962bc0","name":"Shuang Ge","hidden":false},{"_id":"6a0d186d65eb30f20d962bc1","name":"Lei Chen","hidden":false},{"_id":"6a0d186d65eb30f20d962bc2","name":"Yue Li","hidden":false},{"_id":"6a0d186d65eb30f20d962bc3","name":"Mingcheng Wan","hidden":false},{"_id":"6a0d186d65eb30f20d962bc4","name":"Cong Wang","hidden":false}],"publishedAt":"2026-05-19T00:00:00.000Z","submittedOnDailyAt":"2026-05-20T00:00:00.000Z","title":"Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding","submittedOnDailyBy":{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user","name":"taesiri"},"summary":"Speculative decoding (SD) accelerates large language model inference by leveraging a draft-then-verify paradigm. To maximize the acceptance rate, recent methods construct expansive draft trees, which unfortunately incur severe VRAM bandwidth and computational overheads that bottleneck end-to-end speedups. While dynamic-depth pruning can reduce this latency by removing marginal branches, it also discards potentially valid candidates, preventing the acceptance rate from reaching the upper bound of dense trees. In this paper, we identify a critical opportunity in resource allocation: the transition from dense to pruned drafting frees up significant computational budget. To break this Pareto tradeoff, we introduce Graft, a compensation framework that couples pruning and retrieval as mutually reinforcing operations. Pruning supplies sufficient budget for retrieval, while retrieval compensates for pruning-induced coverage loss and recovers accepted length. By employing a sequential `prune-then-graft' mechanism, Graft attaches highly predictive retrieved tokens into positions opened by pruning, filling the topological gaps with near-zero overhead. Graft is entirely training-free and lossless. Comprehensive evaluations show that Graft establishes a new Pareto frontier across practical deployment settings, including short-context generation, long-context generation, and large-scale models. On short-context benchmarks, it achieves up to 5.41times speedup and improves average speedup over EAGLE-3 by up to 21.8% on the large-scale Qwen3-235B. We also provide a preliminary exploration of applying Graft to the DFlash-style block drafting paradigm, offering initial evidence and insights for extending grafting beyond autoregressive draft trees.","upvotes":2,"discussionId":"6a0d186e65eb30f20d962bc5","ai_summary":"Graft is a training-free framework that enhances speculative decoding by dynamically combining pruning and retrieval operations to improve acceptance rates and inference speed without sacrificing accuracy.","ai_keywords":["speculative decoding","draft-then-verify paradigm","draft trees","VRAM bandwidth","computational overhead","dynamic-depth pruning","Pareto tradeoff","compensation framework","pruning","retrieval","sequential prune-then-graft mechanism","autoregressive draft trees","DFlash-style block drafting"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"643b62ac065961b2252abb7a","avatarUrl":"/avatars/c7fb4d11f0d795a52bdc771c04a69a20.svg","isPro":false,"fullname":"zuijiang","user":"zuijiang","type":"user"},{"_id":"661ab1f1fa3b144a381fa454","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/661ab1f1fa3b144a381fa454/IlpZBb9NCjo7ntFwMIH53.png","isPro":true,"fullname":"Urro","user":"urroxyz","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.20104.md"}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2605.20104 in a model README.md to link it from this page.

Cite arxiv.org/abs/2605.20104 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2605.20104 in a Space README.md to link it from this page.