There's an interactive chart and some extra data in the blog post if you're interested.

There are plenty of KL-divergence benchmarks for GGUF models, but most of them compare one GGUF quant against another. I wanted to know how those quants stack up against other commonly used formats (especially NVFP4).

I tested 16 quantizations of Qwen3.6 27B: GGUF models in llama.cpp and the others in vLLM. At each token in the test set, I compared the quantized model’s next-token probability distribution with that of an unquantized reference. The resulting KL divergence measures how far the quant has drifted from the original model; lower is better.

Weight-only GGUFs have the best quality-size tradeoffs

GGUF results occupy most of the lower envelope of the chart. For almost every size, a GGUF running in llama.cpp has the lowest measured KL divergence among nearby weight sizes. The main factor here is likely the activation quantization - GGUFs don’t quantize activations at all. Several vLLM checkpoints quantize weights, activations, and sometimes the KV cache.

vLLM quants vary substantially

Quantizations of similar size do not preserve the reference distribution equally well. Particularly of note is the Sakamakismile NVFP4 (W4A4) quant, which has substantially higher KLD compared to similarly sized (and even smaller) quants.

The two conventional Q4 GGUFs are consistent with each other. Bartowski Q4_K_L measures 0.2218 and Unsloth UD_Q4_K_XL measures 0.2273, with heavily overlapping intervals. AWQ and NVIDIA’s mixed NVFP4 are also nearly tied at 0.2776 and 0.2807.

What the KL number means

At every prompt position, the benchmark computes D_KL(P_reference || P_quant): how much the quantized model’s next-token distribution differs from the full-precision distribution. Zero means no measured change; larger values mean more of the reference distribution was displaced.

Both engines compute exact full-vocabulary softmax probabilities, but only the top 200 log probabilities per position are used. The benchmark solves for the minimum KL consistent with the two measured top-200 lists, their remaining probability budgets, and the fact that an unlisted quant token cannot exceed the quant’s smallest reported probability, in order to get a lower bound on full-vocabulary KL.

The mean reference tail mass outside the top 200 was 0.0025 for both engines in this run. Top-1 agreement does not depend on the tail approximation and provides a complementary check.

Top-1 agreement is the fraction of positions at which the quantized model and its reference assign the highest probability to the same token.

Methodology

Each quant was measured against a reference model in its own engine:

• GGUF quants were compared with a BF16 GGUF reference under llama.cpp.
• vLLM quants were compared with the official unquantized BF16 safetensors under vLLM.

I created my own dataset for the KL measurements, which ended up being 100 structured agentic tool-use conversations containing 182,306 tokens. Prompts range from 1,700 to 1,950 tokens.

Quantized checkpoints ran without changes, including any declared compute dtype, activation quantization, or KV-cache scheme, in order to measure the true fidelity of each quant recipe.

The size measurement includes MTP/NextN layers and excludes KV/recurrent caches, activations, workspaces, CUDA graphs, runtime context, and unloaded multimodal components. It is not total serving memory. Take these measurements with a grain of salt, as they’ll vary in actual deployment depending on your configuration.

Practical takeaways

• Quantization format alone is not enough to predict quality. Look at the quantization recipe to determine if it fits your needs.
• Activation quantization can improve throughput on supported hardware, but this comes at the cost of quality.
• If quality per loaded GiB is the priority, the tested GGUF recipes provide the strongest tradeoffs.
• GGUF Q5 for Qwen3.6 27B seems to be the sweet spot from the results.

Final notes

KLD benchmarks may be able to show the relative differences in quantization quality, but this doesn’t translate perfectly into real-world performance. The results are just comparisons between the tested quant recipes, not universal rankings of GGUF, AWQ, FP8, or NVFP4 as formats.