DeepSeek-V4-Flash-0731 on Bosgame M5 with RTX PRO 6000 Max-Q eGPU
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Here are my numbers:
| Quant | Size | Layout | Decode | Prefill | Draft acceptance |
|---|---|---|---|---|---|
| UD-Q8_K_XL | 150.8 GiB | 20 layers CUDA0 / 23 ROCm0 + drafter | 44.0 t/s | 564 t/s | 0.535 |
| UD-Q4_K_XL | 144.4 GiB | 22 / 21 + drafter | 48.4 t/s | 585 t/s | 0.532 |
| UD-Q2_K_XL | 90.2 GiB | entirely on CUDA0, no drafter | 59.5 t/s | 1513 t/s | — |
I let claude port the DSpark drafter from the closed PR to current main. https://github.com/haraldh/llama.cpp/tree/dspark-dsv4
UD-Q2_K_XL
sh llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q2_K_XL \ --host 0.0.0.0 --port 8000 \ --temp 1.0 --top-p 1.0 --min-p 0.0 \ --no-mmap -fa on -np 1 \ --device CUDA0 \ -ub 2048 -b 4096 \ -c 200000 --cache-ram 65536
UD-Q4_K_XL
sh llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL \ --host 0.0.0.0 --port 8000 \ --temp 1.0 --top-p 1.0 --min-p 0.0 \ --no-mmap -fa on -np 1 \ --device CUDA0,ROCm0 --split-mode layer --tensor-split 100,0 \ -ot 'blk\.(2[2-9]|3[0-9]|4[0-2])\.ffn_(gate|up|down)_exps\.weight=ROCm0' \ -md DSV4-Flash-0731-DSpark-draft-bf16.gguf \ --spec-type draft-dspark --spec-draft-n-max 5 \ --device-draft CUDA0 --spec-draft-p-min 0.3 \ -ub 2048 -b 4096 \ -c 200000 --cache-ram 32768
UD-Q8_K_XL
As above with :UD-Q8_K_XL and the expert boundary two layers lower, since its dense part is 6.3 GiB larger:
-ot 'blk\.(2[0-9]|3[0-9]|4[0-2])\.ffn_(gate|up|down)_exps\.weight=ROCm0'
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.