In standard AWQ, per-channel scales and quantization ranges are picked in separate steps: scales first, then the quantization parameters. But they're not independent, i.e., the rounding error from one depends on the choice of the other, so optimizing them in sequence leaves quality on the table. Our cyankiwi AWQ 26.05 update jointly fits scales and quantization ranges against a reconstruction objective.

We benchmarked cyankiwi AWQ 26.05 update against every major 4-bit method on Llama-3 as examples, measuring KL Divergence vs the BF16 baseline on GPQA Diamond responses.

Result: cyankiwi posts the lowest KLD on all three base models. Lower is better.

Llama-3.2-3B-Instruct

Quantized Model	Method	KLD
cyankiwi/Llama-3.2-3B-Instruct-AWQ-INT4	cyankiwi AWQ INT4	0.00510
unsloth/Llama-3.2-3B-Instruct-unsloth-bnb-4bit	unsloth BNB NF4	0.00785
unsloth/Llama-3.2-3B-Instruct-bnb-4bit	BNB NF4	0.00896
nvidia/Meta-Llama-3.2-3B-Instruct-ONNX-INT4	AWQ INT4	0.01494
casperhansen/llama-3.2-3b-instruct-awq	AWQ INT4	0.02437

Llama-3.1-8B-Instruct

Quantized Model	Method	KLD
cyankiwi/Llama-3.1-8B-Instruct-AWQ-INT4	cyankiwi AWQ INT4	0.00478
RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w4a16	GPTQ INT4	0.00729
unsloth/Meta-Llama-3.1-8B-Instruct-unsloth-bnb-4bit	unsloth BNB NF4	0.00769
unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit	BNB NF4	0.00835
RedHatAI/Llama-3.1-8B-Instruct-NVFP4	SmoothQuant NVFP4	0.01059
nvidia/Llama-3.1-8B-Instruct-NVFP4	NVFP4	0.01190

Llama-3.3-70B-Instruct

Quantized Model	Method	KLD
cyankiwi/Llama-3.3-70B-Instruct-AWQ-INT4	cyankiwi AWQ INT4	0.02826
unsloth/Llama-3.3-70B-Instruct-unsloth-bnb-4bit	unsloth BNB NF4	0.04444
casperhansen/llama-3.3-70b-instruct-awq	AWQ INT4	0.04859
unsloth/Llama-3.3-70B-Instruct-bnb-4bit	BNB NF4	0.06879
nvidia/Llama-3.3-70B-Instruct-NVFP4	NVFP4	0.08307
RedHatAI/Llama-3.3-70B-Instruct-quantized.w4a16	GPTQ INT4	0.09272

https://preview.redd.it/uicubbg6951h1.png?width=6400&format=png&auto=webp&s=2f7f1d4e46c9953f00c68518b3c2aa058fc34e32