Qwen3.6-27B speculative decoding gets better on heavier quants
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
| I finished the speed leg of my spec-decode benchmarking for Qwen3.6-27B, main algorithms across quants. Overall: the heavier the quant, the more spec-decode buys you (10 of 10 speculative configs rank Q8 > Q6 > Q4 by multiplier). Acceptance is quant-independent at matched depth, the base step slows with weight bytes, draft+verify overhead does not. Exception is nvfp4/SGLang which is the fastest quant/engine pair overall. DFlash is the fastest algo overall. Weaver beats it but is the outlier: fork-only, and the head is per-target, so that is one model/quant combo, not a general option. MTP is a solid second everywhere. EAGLE3 is dominated by both, useful only as a fallback when no DFlash drafter exists. ngram I would skip: ~1.03× regardless of quant, and net-negative under concurrency. Interesting anomaly: llama.cpp's MTP path on UD-Q4 is pathologically slow. Q4 MTP-3 is slower in absolute tok/s than Q6 MTP-3 at identical acceptance, which bandwidth cannot explain. Output is correct, mechanism unknown. Caveat: a single, narrow data point on one hardware config. Spec-Bench with short outputs, greedy, batch 1 is close to a best case for speculative decoding. Under concurrency the per-stream gains shrink and longer contexts will compress them further, so treat these as an upper bound for this workload shape, not a general speedup. [link] [comments] |
More from r/LocalLLaMA
-
NVIDIA shipped OpenShell, an open source sandbox that gives local and open agents real runtime limits instead of prompt rules. Over 100 firms joined the safety stack. OpenAI did not.
Sep 28
-
3090 for $1500???
Sep 28
-
modified qwen 3.8 27b modifies windows credential dumper to bypass EDR detection
Sep 28
-
Minisforum MS-S1 MAX-P495 @ €7.799,00
Sep 28
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.