Qwen3.6-27B speculative decoding gets better on heavier quants
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
| I finished the speed leg of my spec-decode benchmarking for Qwen3.6-27B, main algorithms across quants. Overall: the heavier the quant, the more spec-decode buys you (10 of 10 speculative configs rank Q8 > Q6 > Q4 by multiplier). Acceptance is quant-independent at matched depth, the base step slows with weight bytes, draft+verify overhead does not. Exception is nvfp4/SGLang which is the fastest quant/engine pair overall. DFlash is the fastest algo overall. Weaver beats it but is the outlier: fork-only, and the head is per-target, so that is one model/quant combo, not a general option. MTP is a solid second everywhere. EAGLE3 is dominated by both, useful only as a fallback when no DFlash drafter exists. ngram I would skip: ~1.03× regardless of quant, and net-negative under concurrency. Interesting anomaly: llama.cpp's MTP path on UD-Q4 is pathologically slow. Q4 MTP-3 is slower in absolute tok/s than Q6 MTP-3 at identical acceptance, which bandwidth cannot explain. Output is correct, mechanism unknown. Caveat: a single, narrow data point on one hardware config. Spec-Bench with short outputs, greedy, batch 1 is close to a best case for speculative decoding. Under concurrency the per-stream gains shrink and longer contexts will compress them further, so treat these as an upper bound for this workload shape, not a general speedup. [link] [comments] |
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.