Hi everyone, </p>\n<p>TL;DR: the community assumption for hybrid LLMs has been that the recurrent half (Gated DeltaNet) is too fragile for 4-bit — early quants of Qwen3.8-27B all kept it at 8/16-bit. We quantized all 496 linear layers to NVFP4 W4A4 — GDN and its gate projections included — with calibration-only PTQ (no QAT, no distillation), and it matches BF16 within seed noise on MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, while being ~2.9× smaller than BF16 and 14–19% faster at prefill than the partial quants.</p>\n<p> All four models were benchmarked head-to-head under one identical serving regime (vLLM, TP=1, single 96 GB Blackwell GPU):</p>\n<ul>\n<li>Qwen/Qwen3.8-27B — the BF16 baseline</li>\n<li>minima-ai/mnma_qwen3.8_27b_nvfp4 — ours, all 496 layers NVFP4 W4A4</li>\n<li>unsloth/Qwen3.8-27B-NVFP4 — keeps GDN + attention at FP8 W8A8</li>\n<li>RadixArk/Qwen3.8-27B-NVFP4 — likewise protects GDN/attention</li>\n</ul>\n<p>Three findings that surprised us:</p>\n<ul>\n<li>The \"fragile\" decay/write gates are actually the least sensitive layers in the model — their softplus/sigmoid parameterizations squash an ~11% GEMM error down to ~2% at the output.</li>\n<li>Quantization noise does not accumulate in the recurrent state. The delta rule overwrites the state along each new key direction, so an injected error is forgotten within hundreds of tokens — the 32K perplexity gap actually shrinks with position in the context.</li>\n<li>A serving trap: kernels that fuse GDN projections into one GEMM silently mis-scale per-module-calibrated NVFP4 checkpoints. We document the fix; our released checkpoint ships pre-harmonized and serves correctly out of the box.</li>\n</ul>\n<p> Weights (17.5 GiB, single Blackwell GPU): <a href=\"https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4\">https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4</a></p>\n","updatedAt":"2026-09-04T01:32:44.798Z","author":{"_id":"62618ef66dae705b2567f9c7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62618ef66dae705b2567f9c7/AxxkFDg4ckrhB-xrk41BK.png","fullname":"David Mayboroda","name":"dmayboroda","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false,"primaryOrg":{"avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a0621951931f3e8cee462b2/c93v-mCVGmOkYVm58qWFL.png","fullname":"Minima AI","name":"minima-ai","type":"org","isHf":false,"details":"AI inference optimisation","plan":"team"}}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8600409030914307},"editors":["dmayboroda"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/62618ef66dae705b2567f9c7/AxxkFDg4ckrhB-xrk41BK.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.04098","authors":[{"_id":"6a9a1dcd8f7c3b75572393c9","name":"Sergii Kozyrev","hidden":false},{"_id":"6a9a1dcd8f7c3b75572393ca","name":"Davyd Maiboroda","hidden":false}],"publishedAt":"2026-09-03T00:00:00.000Z","submittedOnDailyAt":"2026-09-04T00:00:00.000Z","title":"Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM","submittedOnDailyBy":{"_id":"62618ef66dae705b2567f9c7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62618ef66dae705b2567f9c7/AxxkFDg4ckrhB-xrk41BK.png","isPro":false,"fullname":"David Mayboroda","user":"dmayboroda","type":"user","name":"dmayboroda"},"summary":"Hybrid LLMs pair softmax attention with linear-attention layers such as Gated DeltaNet (GDN), whose recurrent state summarizes the context in fixed size. Early community 4-bit quantizations of Qwen3.8-27B (48 GDN layers, 16 attention layers) left the GDN block in 8- or 16-bit precision -- especially its decay and write-strength gates -- on the intuition that errors in a recurrence accumulate over long contexts. We test that intuition by building Minima: NVFP4 W4A4 on all 496 linear layers, GDN included. Across perplexity at 4K/32K, MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, Minima matches BF16 within seed noise (5-task average -0.52) while being the smallest (17.5 GiB) and fastest-prefill (+14-19%) recipe we compare, and its 32K perplexity gap shrinks with position. A four-part mechanism study explains why: (i) NVFP4's 16-element block scaling localizes the residual stream's extreme outliers, equalizing activation error across layer roles; (ii) the supposedly fragile gate projections are the least sensitive -- softplus/exponential and sigmoid parameterizations compress ~11% GEMM error to ~2% output error; (iii) the delta-rule recurrence holds injected noise at a flat plateau over 32K tokens and forgets a state impulse within hundreds of steps, because each write overwrites the state along the current key direction; (iv) the per-token quantization cost washes out with context instead of compounding. We also repair a global-scale mismatch that arises when per-module-calibrated NVFP4 checkpoints are served by kernels that fuse those modules into one GEMM, and show calibrated FP8 KV-cache scales are performance-free. The result: a practical recipe -- quantize everything, ship KV scales -- and a mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize. Checkpoint: https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4","upvotes":53,"discussionId":"6a9a1dcd8f7c3b75572393cb","ai_summary":"Fully quantizing hybrid LLMs—including recurrent Gated DeltaNet layers—to 4-bit NVFP4 preserves accuracy across long-context and reasoning benchmarks by localizing outliers and exploiting robust delta-rule dynamics.","ai_keywords":["Gated DeltaNet","linear-attention","softmax attention","4-bit quantization","NVFP4","W4A4","delta-rule recurrence","GEMM","KV-cache","FP8"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"6a0623b006f36fedeb62e39f","name":"minima-ai","fullname":"Minima AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a0621951931f3e8cee462b2/c93v-mCVGmOkYVm58qWFL.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"62618ef66dae705b2567f9c7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62618ef66dae705b2567f9c7/AxxkFDg4ckrhB-xrk41BK.png","isPro":false,"fullname":"David Mayboroda","user":"dmayboroda","type":"user"},{"_id":"6a69eb859ef85244dc4ca374","avatarUrl":"/avatars/21fc260ba33de7f03d663e84fba9199b.svg","isPro":false,"fullname":"Anthony Sanchez","user":"anthony-code","type":"user"},{"_id":"6a6a8229a5b9c4c08badf665","avatarUrl":"/avatars/d8d4deed04213f6dfbe4b504c2e0c1ec.svg","isPro":false,"fullname":"Timothy Garcia","user":"timothy-garcia","type":"user"},{"_id":"6a6a92931b3822fa45f64623","avatarUrl":"/avatars/c51da7089b42b77a78ee76de22250514.svg","isPro":false,"fullname":"Jessica Brown","user":"jessica-brown","type":"user"},{"_id":"6a6a92f51b3822fa45f64b3b","avatarUrl":"/avatars/8d2e9df93e7cdbe7a98b8511f0fa2efe.svg","isPro":false,"fullname":"Kevin Perez","user":"k-perez","type":"user"},{"_id":"6a6a95368acf46140bae93a3","avatarUrl":"/avatars/c64673a23a306cc73ed00382ec370e10.svg","isPro":false,"fullname":"Kevin Thompson","user":"rapidFox","type":"user"},{"_id":"6a6aa05b3550efadfe66fdbf","avatarUrl":"/avatars/4232e88a186fb67e814d28ed96574c02.svg","isPro":false,"fullname":"Joseph Jackson","user":"driftwisp","type":"user"},{"_id":"6a6aa169c125cc860a93b9f7","avatarUrl":"/avatars/4d7fa3c3bb83274d7ef3d3066621b801.svg","isPro":false,"fullname":"Karen Gonzalez","user":"vectorridge","type":"user"},{"_id":"6a6a9d2c28e0925e7302d8a1","avatarUrl":"/avatars/fdd0b6d29da96655180b28642951e87e.svg","isPro":false,"fullname":"Richard Thomas","user":"zenithridge","type":"user"},{"_id":"6a6aa30ba569279e390ed265","avatarUrl":"/avatars/47cc5c0662cc5244b3c6897886f1a16d.svg","isPro":false,"fullname":"Elizabeth Lee","user":"ZenithElizabeth","type":"user"},{"_id":"6a6aa75d2af00ac1140c64fd","avatarUrl":"/avatars/7d2b83a403ad93a0f77dc68a6d3e48fc.svg","isPro":false,"fullname":"George Wilson","user":"George-Wilson","type":"user"},{"_id":"6a6aa41d4c287dbb8e805dd3","avatarUrl":"/avatars/f109124053777f7667ff3235cd8aaf79.svg","isPro":false,"fullname":"Sarah Thompson","user":"rapidWing","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":3,"organization":{"_id":"6a0623b006f36fedeb62e39f","name":"minima-ai","fullname":"Minima AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a0621951931f3e8cee462b2/c93v-mCVGmOkYVm58qWFL.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.04098.md","query":{}}">
Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Abstract
Fully quantizing hybrid LLMs—including recurrent Gated DeltaNet layers—to 4-bit NVFP4 preserves accuracy across long-context and reasoning benchmarks by localizing outliers and exploiting robust delta-rule dynamics.
Hybrid LLMs pair softmax attention with linear-attention layers such as Gated DeltaNet (GDN), whose recurrent state summarizes the context in fixed size. Early community 4-bit quantizations of Qwen3.8-27B (48 GDN layers, 16 attention layers) left the GDN block in 8- or 16-bit precision -- especially its decay and write-strength gates -- on the intuition that errors in a recurrence accumulate over long contexts. We test that intuition by building Minima: NVFP4 W4A4 on all 496 linear layers, GDN included. Across perplexity at 4K/32K, MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, Minima matches BF16 within seed noise (5-task average -0.52) while being the smallest (17.5 GiB) and fastest-prefill (+14-19%) recipe we compare, and its 32K perplexity gap shrinks with position. A four-part mechanism study explains why: (i) NVFP4's 16-element block scaling localizes the residual stream's extreme outliers, equalizing activation error across layer roles; (ii) the supposedly fragile gate projections are the least sensitive -- softplus/exponential and sigmoid parameterizations compress ~11% GEMM error to ~2% output error; (iii) the delta-rule recurrence holds injected noise at a flat plateau over 32K tokens and forgets a state impulse within hundreds of steps, because each write overwrites the state along the current key direction; (iv) the per-token quantization cost washes out with context instead of compounding. We also repair a global-scale mismatch that arises when per-module-calibrated NVFP4 checkpoints are served by kernels that fuse those modules into one GEMM, and show calibrated FP8 KV-cache scales are performance-free. The result: a practical recipe -- quantize everything, ship KV scales -- and a mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize. Checkpoint: https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4
Community
Hi everyone,
TL;DR: the community assumption for hybrid LLMs has been that the recurrent half (Gated DeltaNet) is too fragile for 4-bit — early quants of Qwen3.8-27B all kept it at 8/16-bit. We quantized all 496 linear layers to NVFP4 W4A4 — GDN and its gate projections included — with calibration-only PTQ (no QAT, no distillation), and it matches BF16 within seed noise on MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, while being ~2.9× smaller than BF16 and 14–19% faster at prefill than the partial quants.
All four models were benchmarked head-to-head under one identical serving regime (vLLM, TP=1, single 96 GB Blackwell GPU):
- Qwen/Qwen3.8-27B — the BF16 baseline
- minima-ai/mnma_qwen3.8_27b_nvfp4 — ours, all 496 layers NVFP4 W4A4
- unsloth/Qwen3.8-27B-NVFP4 — keeps GDN + attention at FP8 W8A8
- RadixArk/Qwen3.8-27B-NVFP4 — likewise protects GDN/attention
Three findings that surprised us:
- The "fragile" decay/write gates are actually the least sensitive layers in the model — their softplus/sigmoid parameterizations squash an ~11% GEMM error down to ~2% at the output.
- Quantization noise does not accumulate in the recurrent state. The delta rule overwrites the state along each new key direction, so an injected error is forgotten within hundreds of tokens — the 32K perplexity gap actually shrinks with position in the context.
- A serving trap: kernels that fuse GDN projections into one GEMM silently mis-scale per-module-calibrated NVFP4 checkpoints. We document the fix; our released checkpoint ships pre-harmonized and serves correctly out of the box.
Weights (17.5 GiB, single Blackwell GPU): https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2609.04098 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2609.04098 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.