Q2 DeepSeek V4 Flash on 2x 3080 20GB, 64GB DDR5 | 17 tk/s gen, 270 tk/s prefill

Hey, it's my first time posting here and I thought I'd share my progress on getting Antirez's imatrix Q2 DeepSeek V4 Flash GGUF (86.7 GB) running on my build. I used this llama.cpp fork which fixed the model's output when KV cache is quantised to Q8.

Specs:
- Ryzen 7 7800X3D
- Asus ProArt B850-CREATOR WIFI NEO
- 64GB DDR5-5200
- 2x RTX 3080 20GB (both running at PCIe 4.0 x8, power limited at 280W)

Running the Nvidia 610.43.02 drivers on Debian 13.

Using these launch parameters:

./llama-server \ --model ~/models/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf \ -ngl 99 \ -np 1 \ --n-cpu-moe 28 \ --split-mode layer \ -ts 130,43 \ -b 2048 \ -ub 2048 \ -fa on -c 131072 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --temp 1.0 \ --top-p 1.0 \ --min-p 0.0 \ --no-mmap \ --host 0.0.0.0 \ --port 8080 

(note: I had to force a VRAM allocation bias towards GPU 0 using -ts 130,43 otherwise the server would crash with an OOM error)

I am able to achieve 17 tk/s gen and 270 tk/s prefill at 128k context:

15.07.104.694 I slot print_timing: id 0 | task 3236 | prompt eval time = 6269.96 ms / 1713 tokens ( 3.66 ms per token, 273.21 tokens per second) 15.07.104.697 I slot print_timing: id 0 | task 3236 | eval time = 130667.97 ms / 2290 tokens ( 57.06 ms per token, 17.53 tokens per second) 15.07.104.697 I slot print_timing: id 0 | task 3236 | total time = 136937.93 ms / 4003 tokens 

I'm sure there's a million more ways I could further optimise it, but I think it's not too bad for the hardware.