ConvRot Quant method now in llama-cpp-turboquant
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
It started here , and now https://github.com/TheTom/llama-cpp-turboquant/ has it.
Imagine a Q6 quant with nearly Q8 KLD/PPL.
Q6_CR and Q5_CR have a slight improvement over their base counterparts.
Also while you are there check out --moe-cache auto to help improve running MoE models bigger than your VRAM.
I am hoping that with this we may be able to recover some lost quality from turbo4/3/2 , but I haven't test that out yet.
PR's has the breakdown of the tests, we did have some some decode and crashing issues but they are now resolved.
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.