TL;DR llama.cpp fork with more KV cache quantization features, with all claims supported by benchmarks: KVarN, KV cache precision tail, additional types of standard KV cache (q2_0-q3_1, q6_0, q6_1), and more.
BeeLLama v0.4.1 is here, building up on top of v0.4.0 feature set, now with better backend and model support.
- KVarN. Variance-normalized KV-cache quantization (paper) with better precision per bit. Although it was already introduced a few weeks ago in v0.3.2 Preview, that was a very raw implementation, with performance issues and VRAM usage spikes. Now in v0.4.1 it's the real deal: the precision is still above what usual quants offer for the same bit width, but now with very modest sacrifices to prefill, decode, and memory.
- KV cache precision tail. A promising new feature in the domain of mixed-precision KV cache. It allows to specify a specific numbers of recent tokens that will be stored in BF16 or F16, with the rest of KV cache being quantized as usual. This way we can store the hottest tokens in a lossless fashion, preventing a model from misreading your task details, code, or data.
- Additional types of standard KV cache.
q6_0 and q6_1 join the high end of the ladder, allowing to fine-tune precision vs VRAM in-between upstream's q5_0/1 and q8_0 types. q2_0, q2_1, q3_0 and q3_1 are added as a replacement for turbo3 and turbo2 for cases where KVarN doesn't work well, but you just can't fit everything into VRAM without extreme quantization.
Please note that for SWA architecture (Gemma, GPT-OSS) the precision of KVarN and KVPT is the same, but VRAM and performance costs are higher due to complications between SWA ring and mixed precision KV cache.
GitHub repo: https://github.com/Anbeeld/beellama.cpp
KLD results for Qwen 3.6 27B Q5_K_S 64k
Here are all symmetrical qX_0 pairs and kvarnX pairs where X >= 4 with tail 0/1024/2048, compared against q8_0 t0 from the same benchmarks, and sorted by ratio between median KLD and VRAM costs. Full benchmark data and analysis: KV Cache Precision Tail: Implementation and Benchmarks.
| Cache | Tail | KV MiB | Size vs q8_0 | Median/size vs q8_0 | Median vs q8_0 | P99.9 vs q8_0 |
kvarn4 | 1024 | 1232.00 | 56.6% | 1.62 | 91.4% | 102.9% |
kvarn4 | 2048 | 1296.00 | 59.6% | 1.60 | 95.5% | 95.6% |
kvarn4 | 0 | 1184.00 | 54.4% | 1.50 | 81.8% | 82.5% |
q4_0 | 1024 | 1248.00 | 57.4% | 1.50 | 86.0% | 89.0% |
q4_0 | 2048 | 1312.00 | 60.3% | 1.48 | 89.2% | 100.6% |
kvarn5 | 0 | 1440.00 | 66.2% | 1.48 | 98.1% | 107.5% |
kvarn5 | 1024 | 1488.00 | 68.4% | 1.48 | 101.3% | 106.1% |
kvarn5 | 2048 | 1552.00 | 71.3% | 1.43 | 101.9% | 105.6% |
q5_0 | 1024 | 1504.00 | 69.1% | 1.40 | 96.9% | 105.6% |
q5_0 | 2048 | 1568.00 | 72.1% | 1.36 | 98.0% | 103.7% |
kvarn6 | 0 | 1696.00 | 77.9% | 1.31 | 102.2% | 104.5% |
kvarn6 | 1024 | 1744.00 | 80.1% | 1.29 | 103.4% | 109.9% |
kvarn6 | 2048 | 1808.00 | 83.1% | 1.25 | 103.8% | 108.1% |
q6_0 | 0 | 1664.00 | 76.5% | 1.24 | 94.7% | 102.1% |
q6_0 | 1024 | 1760.00 | 80.9% | 1.24 | 100.1% | 109.2% |
q5_0 | 0 | 1408.00 | 64.7% | 1.22 | 78.8% | 95.8% |
q6_0 | 2048 | 1824.00 | 83.8% | 1.20 | 100.6% | 103.5% |
kvarn8 | 0 | 2208.00 | 101.5% | 1.03 | 104.4% | 104.9% |
kvarn8 | 1024 | 2256.00 | 103.7% | 1.01 | 104.4% | 106.2% |
q8_0 | 0 | 2176.00 | 100.0% | 1.00 | 100.0% | 100.0% |
q8_0 | 1024 | 2272.00 | 104.4% | 0.97 | 101.3% | 106.1% |
kvarn8 | 2048 | 2320.00 | 106.6% | 0.97 | 103.6% | 104.7% |
q8_0 | 2048 | 2336.00 | 107.4% | 0.95 | 101.6% | 106.8% |
q4_0 | 0 | 1152.00 | 52.9% | 0.93 | 49.2% | 60.2% |
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.