r/LocalLLaMA · · 1 min read

We quantized Qwen 3.8 27B and compared the quants on an RTX 6000

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

We quantized Qwen 3.8 27B and compared the quants on an RTX 6000

Me and my team made Atomic Dynamic GGUF quants for Qwen 3.8 27B, so we wanted to see the difference between them by giving each quant the same voxel island creation task

First of all we were surprised at how well Qwen 3.8 27B handled the 3D scenes in general, though part of that is probably because all the scenes were voxels

quant size top-1 vs BF16 mean KLD decode, RTX PRO 6000
AD-Q4_K_M 17.1 GB 95.6% 0.0113 67 tok/s
AD-Q5_K_M 20.2 GB 97.3% 0.0042 57 tok/s
AD-Q6_K 25.0 GB 98.7% 0.0011 49 tok/s
Q8_0 28.9 GB 98.9% 0.0006 50 tok/s

We think that each quant handled the scenes in a pretty similar way, the difference isn't that drastic, to the point that sometimes we preferred the Q4 output overall, though for the safest pick we recommend AD-Q6_K

We ran the test inside atomic.chat and watched the output right there, the quants are available to download directly inside the app or on huggingface ( https://huggingface.co/collections/AtomicChat/qwen-38-27b ) (any feedback is appreciated, we're trying to make the product and models as good for you guys as possible)

submitted by /u/Fun-Meaning-6474
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA