DeepSeek-v4-Flash-Mini 54GB GGUF running at ~20.5 t/s

Took the REAP adaptation of DeepSeek-V4-Flash (0xSero/DeepSeek-V4-Flash-0731-REAP) along with antirez/deepseek-v4-gguf as inspiration, and decided to see how aggressive we could get with standard quant tricks to create a budget-friendly "Mini" build.

For the lulz, naturally.

Started with the full 95GB bf16 GGUF and crushed it down to an IQ2_XXS variant with mixed quantization (w2Q2K-AProjQ8-OutQ8).

Is extreme 2-bit quantization practical for complex reasoning? Debatable. Did it shave off over 40GB of VRAM/RAM footprint and still generate coherently? Absolutely.

Science isn't about why, it's about why not.

prompt eval time = 372.26 ms / 12 tokens (31.02 ms per token, 32.24 tokens per second) eval time = 81141.35 ms / 1667 tokens (48.68 ms per token, 20.54 tokens per second) total time = 81513.62 ms / 1679 tokens graphs reused = 1804 

The File Sizes:

-rw-rw-r-- 1 jabbatheduck jabbatheduck 95G Aug 4 16:10 deepseek-v4-flash-bf16.gguf -rw-rw-r-- 1 jabbatheduck jabbatheduck 54G Aug 4 17:41 DeepSeek-V4-Flash-REAP-IQ2XXS-w2Q2K-AProjQ8-OutQ8-chat-v2.gguf -rw-rw-r-- 1 jabbatheduck jabbatheduck 353M Aug 4 16:37 DeepSeek-V4-Flash-REAP-IQ2XXS-w2Q2K-AProjQ8-OutQ8-chat-v2-imatrix-0731.gguf