1M context with 17 GB model in 24 GB VRAM: "for the first time I was able to load a context of almost 1M tokens and extract 7 needles from various parts of the text"

https://preview.redd.it/xxjh11f38jih1.png?width=1852&format=png&auto=webp&s=76850ed51e29a8bc86c2ca718d4320075eed4363

Just wanted to share a user report that I found to be very interesting. Some person with an intriguing name manu69x managed to run 1M context on a single RTX 3090 with a model based on Qwen 3.5 35B A3B, which was taking up some 17 GB of VRAM alone. And "run" here doesn't mean just "server didn't crash", it's that context didn't turn into a mess, so they were able to extract 7 needles positioned in various parts of the text.

They used KVarN 4-bit for both K and V with my BeeLlama.cpp fork, build v0.4.3 preview. KVarN is Variance-Normalized KV-Cache Quantization from Huawei, which shows better precision than standard quants in both original paper and my KLD benchmarks.

Honestly, just really cool to see someone push it to the very limit. Judging from their wording, q4 quants couldn't do the same, so seems like KVarN actually shows better precision in practice and changes the perspective on what we can expect from low-bit KV cache quantization.