There is a lot of discussion which quant to use. Especially a lot of post how much speed you gain with FP4, but not much about quality.
So i tested some of the models that I use on my R9700.
Models:
Qwen3.8-27B-UD-Q6_K_L.gguf
Swift-Qwen3.8-27B-Q6_K.gguf
Swift-1.5-Qwen3.8-27B-MXFP4-A-outQ6_K.gguf
Hypotheses:
Qwen3.8-27B-UD-Q6_K_L.gguf best in accuracy and slowest
Swift-Qwen3.8-27B-Q6_K.gguf second best in accuracy and speed
Swift-1.5-Qwen3.8-27B-MXFP4-A-outQ6_K.gguf worst in accuracy and fastest
Servers for the models:
Qwen3.8-27B-UD-Q6_K_L.gguf and Swift-Qwen3.8-27B-Q6_K.gguf: Llama.cpp
Swift-1.5-Qwen3.8-27B-MXFP4-A-outQ6_K.gguf: whirl
Environment:
The Pi agent set up gsm8k (math) and humaneval (code) tests
within the same docker container.
Results:
I was surprised that the MXFP4 quant was the best in math
and Qwen3.8-27B-UD-Q6_K_L the worst. Also that Swift-Qwen3.8-27B-Q6_K was the
slowest.
For better more precise results I would need to run whole test suits and run them multiple times. But that would take to much time on my hardware.
Can anyone explain how is MXFP4 is first in math and last in code?
submitted by
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.