v0.32.10-rc0: nn: speed up prefill on double-scale nvfp4 models
Mirrored from Ollama releases for archival readability. Support the source by reading on the original site.
ModelOpt checkpoints apply a float32 global scale to every projection
output on top of the per-group quantization scales. Running the
multiply and the cast back to the activation dtype as separate eager
ops costs an extra kernel launch and a materialized intermediate per
projection.
Compile the multiply and cast into one kernel. On an M5 Max (medians
of order-swapped A/B runs against main; greedy outputs byte-identical):
qwen3.6:27b prefill 703 -> 769 t/s +7.9%
muse-glimmer:30b prefill 790 -> 843 t/s +6.7%
Speculative decode is unchanged within noise on both models. Only
checkpoints with a global scale are affected; single-scale nvfp4,
mxfp8, and affine checkpoints take the unchanged path.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.