Hey r/LocalLLaMA!

Apparently, if you draw enough arrows between proxy rankings like KLD, perplexity, and BPW, and real deployment measurements, quantization evaluation starts to look like abstract modern art. Check it out in the second figure!

TL;DR: KLD and perplexity can help rank quantized models once degradation becomes meaningfully large. But for near-baseline quants, they are effectively unable to tell you which quant is actually better. We did a deep dive to identify where this breakdown occurs and analyzed many individual samples to understand what is actually happening.

Our chaos graph compares 28 quantized models across two dimensions:

• KLD rank versus benchmark-quality rank
• BPW rank versus measured tokens-per-second rank

The circles show where the proxies place each quant. The arrowheads show where the quant actually lands. Throughput was measured on an RTX 6000.

The rankings get scrambled quite a bit. In particular, the quant with the lowest KLD is not necessarily the best model to deploy.

The KLD silent zone

In our experiments, KLD-versus-quality plots consistently split into two regions (see first figure):

• Silent zone: Below a certain KLD threshold, there is no meaningful relationship between KLD and actual model quality. Lower KLD does not imply better quality.
• Lossy zone: Above that threshold, KLD becomes significantly correlated with quality degradation.

We tested 14 fidelity-metric variants across different datasets, context lengths, response-only scoring, aggregation methods, top-k KLD, perplexity, and top-token agreement.

None of them recovered a reliable ranking within the silent zone.

The usual argument for KLD relies on two assumptions:

1. KLD measures how far the quantized model moves from the BF16 reference.
2. Moving farther from BF16 must mean worse quality.

The first assumption holds. The second often does not.

KLD still measures something useful: disagreement volume, meaning how many benchmark outcomes change relative to BF16.

Within the silent zone:

• KLD correlates +0.94 with disagreement volume.
• KLD correlates 0.00 with quality.

For example, suppose the prompt is:

“What is the capital of France?”

The BF16 model answers:

“What a great question. The capital of France is Paris.”

Now imagine the quantized model answers:

“It is Paris.”

Both answers are correct, but the quantized model may receive a worse KLD score because its token distribution and wording deviate more from the BF16 reference.

Meanwhile, another quantized model might answer:

“What a great question. The capital of France is Rome.”

This answer is factually wrong, but because it follows the BF16 model’s wording and token pattern more closely, it could still achieve a better KLD score. Of course that’s not how KLD is exactly measured, but it is close enough for this discussion.

What KLD is missing is direction.

Some changes correct mistakes made by BF16. Others introduce new mistakes. Within the silent zone, KLD has almost no relationship with whether those changes are helpful or harmful, with ρ = −0.13.

Once models enter the lossy zone, divergence becomes more predictably harmful, and the correlation with direction reaches ρ = −0.67. But by that point, the models are no longer near baseline.

KLD measures displacement. Benchmark quality depends on the direction of that displacement.

BPW has a similar problem

BPW is useful for estimating model size and whether a model will fit on a given device. But it does not reliably rank similarly sized quants by tokens per second.

Quantization format, kernels, tensor shapes, hardware, workload, and inference implementation can all change the ordering.

We wrote up the full results here:

[Series landing page]: Three part blog with all the details.

[Preprint]: Full methodology, robustness analyses, and Devstral replication.

I am curious whether this matches what others have observed.

Have you ever seen a quant that had higher KLD yet worked better for your needs? Have you ever run a larger quant that was faster than a smaller one?