Achievable 253 t/s - unsloth/Muse Glimmer 30B UD-Q5_K_M on a 5090
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Benchmarked Muse Glimmer 30B on my RTX 5090 (32GB), 262k context, UD-Q5_K_M + dflash-kquant + mmproj.
| Workload | Stock master + DFlash | ngram-simple | PR #26842 + DFlash |
|---|---|---|---|
| Code patch | 78 t/s | 57 t/s | 220-253 t/s |
| Mixed agent turn | 77 t/s | 68 t/s | 188-213 t/s |
| Tool-call JSON | 71 t/s | 75 t/s | 155-181 t/s |
| Heavy reasoning | 52 t/s | 58 t/s | 120-130 t/s |
PR #26842 moves the DFlash draft argmax from CPU to GPU, which was the bottleneck. I cherry-picked it onto master (it branched before the Muse merge, one conflict to resolve manually) and it builds clean. Code generation now matches Meta's published 233 t/s, which I could not reproduce on stock master.
Notes:
- ngram-simple loses to DFlash on every coding workload.
- Server caps context at the model's metadata context_length, use
--override-kvfor 262k. - The reasoning budget flags do not work with this template. This is verified: with the budget set to 64, the model still burned 2000+ chars thinking and the budget message never appeared. Leave max_tokens headroom for the reasoning block.
Flags:
llama-server ^ --model Muse-Glimmer-30B-UD-Q5_K_M.gguf ^ --mmproj mmproj-kquant.gguf ^ -c 262144 --parallel 1 ^ --override-kv "muse-glimmer.context_length=int:262144,dflash.context_length=int:262144" ^ --kv-unified -ngl all -ngld all -fa on --warmup ^ -b 4096 -ub 512 --cache-prompt --cache-reuse 256 --slot-prompt-similarity 0.5 ^ -t 8 -tb 8 -td 4 -tbd 4 -fit off --load-mode mmap ^ --spec-draft-model dflash-kquant.gguf --spec-draft-ngl 999 --spec-draft-n-max 15 ^ --spec-type draft-dflash --spec-draft-type-k f16 --spec-draft-type-v f16 ^ --reasoning-preserve ^ --temp 1.0 --top-p 0.95 --min-p 0.0 --top-k 64 --jinja --port 9090 Caveat: #26842 is a draft PR, use at your own risk. Rebuild from master once merged.
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.