Hugging Face Daily Papers · May 21, 2026 · 4 min read

Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

Mix-Quant is a phase-aware quantization paradigm for agentic LLMs, enabling fast NVFP4 prefilling while preserving high-quality BF16 decoding. It achieves up to 3× prefill speedup with minimal performance degradation on long-context and agentic benchmarks.\nPaper: <a href=\"https://arxiv.org/pdf/2605.20315\" rel=\"nofollow\">https://arxiv.org/pdf/2605.20315</a> Code: <a href=\"https://github.com/haiquanlu/Mix-Quant\" rel=\"nofollow\">https://github.com/haiquanlu/Mix-Quant</a> Project page: <a href=\"https://haiquanlu.github.io/Mix-Quant\" rel=\"nofollow\">https://haiquanlu.github.io/Mix-Quant</a>\n","updatedAt":"2026-05-21T05:53:51.079Z","author":{"_id":"689cb792f522165a63e55e4f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/689cb792f522165a63e55e4f/LIQv_bkx7rqZLax8CAuyV.jpeg","fullname":"Haiquan Lu","name":"haiquanlu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8682180643081665},"editors":["haiquanlu"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/689cb792f522165a63e55e4f/LIQv_bkx7rqZLax8CAuyV.jpeg"],"reactions":[{"reaction":"🚀","users":["decentmango"],"count":1},{"reaction":"🔥","users":["decentmango"],"count":1},{"reaction":"👀","users":["decentmango"],"count":1},{"reaction":"👍","users":["decentmango"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.20315","authors":[{"_id":"6a0e8b33164dbbc68a26c606","name":"Haiquan Lu","hidden":false},{"_id":"6a0e8b33164dbbc68a26c607","name":"Zigeng Chen","hidden":false},{"_id":"6a0e8b33164dbbc68a26c608","name":"Gongfan Fang","hidden":false},{"_id":"6a0e8b33164dbbc68a26c609","name":"Xinyin Ma","hidden":false},{"_id":"6a0e8b33164dbbc68a26c60a","name":"Xinchao Wang","hidden":false}],"publishedAt":"2026-05-19T00:00:00.000Z","submittedOnDailyAt":"2026-05-21T00:00:00.000Z","title":"Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs","submittedOnDailyBy":{"_id":"689cb792f522165a63e55e4f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/689cb792f522165a63e55e4f/LIQv_bkx7rqZLax8CAuyV.jpeg","isPro":false,"fullname":"Haiquan Lu","user":"haiquanlu","type":"user","name":"haiquanlu"},"summary":"LLM agents have recently emerged as a powerful paradigm for solving complex tasks through planning, tool use, memory retrieval, and multi-step interaction. However, these agentic workflows often introduce substantial input-side overhead, making the compute-intensive prefilling stage a key bottleneck in long-context, multi-turn inference. In this work, we propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference. We first investigate FP4 quantization in agentic LLM workflows and observe that quantizing the entire inference process can incur significant performance degradation. In contrast, the prefilling stage exhibits substantial quantization redundancy and can therefore be quantized with minimal accuracy loss, despite being the dominant source of computation. Based on this insight, we apply high-throughput NVFP4 quantization to the prefilling phase while preserving BF16 precision for decoding. By decoupling prefilling acceleration from decoding quality, Mix-Quant combines phase-aware algorithmic quantization with hardware-efficient NVFP4 execution to alleviate the inference bottleneck in LLM agents. Extensive experiments across long-context and agentic benchmarks demonstrate that Mix-Quant largely preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.","upvotes":17,"discussionId":"6a0e8b34164dbbc68a26c60b","projectPage":"https://haiquanlu.github.io/Mix-Quant/","githubRepo":"https://github.com/haiquanlu/Mix-Quant","githubRepoAddedBy":"user","ai_summary":"Mix-Quant is a phase-aware quantization framework that accelerates long-context, multi-turn LLM inference by applying high-throughput NVFP4 quantization to the prefilling phase while maintaining BF16 precision for decoding.","ai_keywords":["agentic LLM workflows","prefilling stage","FP4 quantization","NVFP4 quantization","BF16 precision","phase-aware quantization","long-context inference","multi-turn inference","hardware-efficient execution"],"githubStars":5,"organization":{"_id":"6508ab2b349930913196378b","name":"NationalUniversityofSingapore","fullname":"National University of Singapore","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/630ca0817dacb93b33506ce7/ZYUmpSMsa5Whihw3me2Bw.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"689cb792f522165a63e55e4f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/689cb792f522165a63e55e4f/LIQv_bkx7rqZLax8CAuyV.jpeg","isPro":false,"fullname":"Haiquan Lu","user":"haiquanlu","type":"user"},{"_id":"692d0b2a3ae240ec31098ac9","avatarUrl":"/avatars/4b385a4df0ac1ad8ee547bcd9130b6c8.svg","isPro":false,"fullname":"huang","user":"decentmango","type":"user"},{"_id":"65392d168838e131ace013d2","avatarUrl":"/avatars/564caf9b189c914dce3062cf244ccbbf.svg","isPro":false,"fullname":"alberta","user":"huanxinzhangggg","type":"user"},{"_id":"66839c7f7b0920f40d7ff3e0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66839c7f7b0920f40d7ff3e0/nuOh5acjmb9QZK3wNQxt1.jpeg","isPro":false,"fullname":"Dairu Liu","user":"dairuliu","type":"user"},{"_id":"668e740f1173ab43d9d9ed5e","avatarUrl":"/avatars/caa9b47c2a5f6d6d679759b8b234a0ab.svg","isPro":true,"fullname":"Zeqing Wang","user":"INV-WZQ","type":"user"},{"_id":"65811eeaa2284a018e51f1ba","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/dH8UZj6Kk5HJkI1DItCNm.jpeg","isPro":false,"fullname":"Zigeng Chen","user":"Zigeng","type":"user"},{"_id":"646a1939c37ca1e12308fe81","avatarUrl":"/avatars/752e9d86018e7d33ad8bcd741203fd86.svg","isPro":false,"fullname":"Gongfan Fang","user":"Vinnnf","type":"user"},{"_id":"634cfebc350bcee9bed20a4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png","isPro":false,"fullname":"Xingyi Yang","user":"adamdad","type":"user"},{"_id":"6860fe55a1ab4d5c885c3edf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/HNSwVGoTd33mgbyjxA-fc.jpeg","isPro":false,"fullname":"QIN ZHIBIN","user":"tuantuan0321","type":"user"},{"_id":"6761a439b4b56a823c8bce1b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/fUKU8eOtdru96Lfy5gFAa.png","isPro":false,"fullname":"liu","user":"linlinlinlinsdada","type":"user"},{"_id":"6624f53748e016b5ea587d40","avatarUrl":"/avatars/f8c16f45de0c3e32437f6e960a5b0959.svg","isPro":false,"fullname":"Shihua Zhang","user":"SuhZhang","type":"user"},{"_id":"640ebdfefdeaae139086f4d8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/640ebdfefdeaae139086f4d8/2N94gbHubplYD8njmUTPf.jpeg","isPro":true,"fullname":"Zhenxiong Tan","user":"Yuanshi","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6508ab2b349930913196378b","name":"NationalUniversityofSingapore","fullname":"National University of Singapore","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/630ca0817dacb93b33506ce7/ZYUmpSMsa5Whihw3me2Bw.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.20315.md"}">

Papers

arxiv:2605.20315

Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs

Published on May 19

· Submitted by

Haiquan Lu on May 21

National University of Singapore

Upvote

Authors:

Abstract

Mix-Quant is a phase-aware quantization framework that accelerates long-context, multi-turn LLM inference by applying high-throughput NVFP4 quantization to the prefilling phase while maintaining BF16 precision for decoding.

AI-generated summary

LLM agents have recently emerged as a powerful paradigm for solving complex tasks through planning, tool use, memory retrieval, and multi-step interaction. However, these agentic workflows often introduce substantial input-side overhead, making the compute-intensive prefilling stage a key bottleneck in long-context, multi-turn inference. In this work, we propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference. We first investigate FP4 quantization in agentic LLM workflows and observe that quantizing the entire inference process can incur significant performance degradation. In contrast, the prefilling stage exhibits substantial quantization redundancy and can therefore be quantized with minimal accuracy loss, despite being the dominant source of computation. Based on this insight, we apply high-throughput NVFP4 quantization to the prefilling phase while preserving BF16 precision for decoding. By decoupling prefilling acceleration from decoding quality, Mix-Quant combines phase-aware algorithmic quantization with hardware-efficient NVFP4 execution to alleviate the inference bottleneck in LLM agents. Extensive experiments across long-context and agentic benchmarks demonstrate that Mix-Quant largely preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.