Ollama releases · · 1 min read

v0.32.10-rc0: nn: speed up prefill on double-scale nvfp4 models

Mirrored from Ollama releases for archival readability. Support the source by reading on the original site.

ModelOpt checkpoints apply a float32 global scale to every projection
output on top of the per-group quantization scales. Running the
multiply and the cast back to the activation dtype as separate eager
ops costs an extra kernel launch and a materialized intermediate per
projection.

Compile the multiply and cast into one kernel. On an M5 Max (medians
of order-swapped A/B runs against main; greedy outputs byte-identical):

qwen3.6:27b        prefill  703 -> 769 t/s  +7.9%
muse-glimmer:30b   prefill  790 -> 843 t/s  +6.7%

Speculative decode is unchanged within noise on both models. Only
checkpoints with a global scale are affected; single-scale nvfp4,
mxfp8, and affine checkpoints take the unchanged path.

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from Ollama releases