mistral.rs v0.9.0: up to 1.8x faster CPU decode than llama.cpp on x86 and ARM!

https://preview.redd.it/nuk5rxceptbh1.png?width=1448&format=png&auto=webp&s=300344dd4c6552379e8536b81ba288be3d6dca3f

On Qwen3 4B Q4_K, mistral.rs decodes faster than llama.cpp at every context depth we measured, on x86 (Sapphire Rapids) and ARM (GB10).

We optimized mistral.rs at granular levels to achieve general speedups for all models. Additionally, our optimizations apply to CPUs of all calibers: from x86 with AVX2 or AVX512, to ARM processors with NEON.

We wanted to make sure this was a comparison in the best possible light for both engines. To ensure this, we swept various configs for mistral.rs and llama.cpp and tested at the best configuration per point for each engine.

Methodology, full tables, and repro scripts can be found here: https://github.com/EricLBuehler/mistral.rs/blob/master/releases/v0.9.0/report.md

If you'd like to try this out, it's super easy to install mistral.rs:

# Mac/Linux: curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh # Windows irm https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.ps1 | iex 

Then, you can run any of your favorite models (Qwen 3.5/3.6, Gemma 4, LFM 2.5) directly from Hugging Face using the mistral.rs ISQ system:

mistralrs run -m google/gemma-4-E4B-it --quant 4 

Reproductions are welcome, especially on hardware I haven't directly benchmarked!