r/LocalLLaMA · · 1 min read

Achievable 253 t/s - unsloth/Muse Glimmer 30B UD-Q5_K_M on a 5090

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

Benchmarked Muse Glimmer 30B on my RTX 5090 (32GB), 262k context, UD-Q5_K_M + dflash-kquant + mmproj.

Workload Stock master + DFlash ngram-simple PR #26842 + DFlash
Code patch 78 t/s 57 t/s 220-253 t/s
Mixed agent turn 77 t/s 68 t/s 188-213 t/s
Tool-call JSON 71 t/s 75 t/s 155-181 t/s
Heavy reasoning 52 t/s 58 t/s 120-130 t/s

PR #26842 moves the DFlash draft argmax from CPU to GPU, which was the bottleneck. I cherry-picked it onto master (it branched before the Muse merge, one conflict to resolve manually) and it builds clean. Code generation now matches Meta's published 233 t/s, which I could not reproduce on stock master.

Notes:

  • ngram-simple loses to DFlash on every coding workload.
  • Server caps context at the model's metadata context_length, use --override-kv for 262k.
  • The reasoning budget flags do not work with this template. This is verified: with the budget set to 64, the model still burned 2000+ chars thinking and the budget message never appeared. Leave max_tokens headroom for the reasoning block.

Flags:

llama-server ^ --model Muse-Glimmer-30B-UD-Q5_K_M.gguf ^ --mmproj mmproj-kquant.gguf ^ -c 262144 --parallel 1 ^ --override-kv "muse-glimmer.context_length=int:262144,dflash.context_length=int:262144" ^ --kv-unified -ngl all -ngld all -fa on --warmup ^ -b 4096 -ub 512 --cache-prompt --cache-reuse 256 --slot-prompt-similarity 0.5 ^ -t 8 -tb 8 -td 4 -tbd 4 -fit off --load-mode mmap ^ --spec-draft-model dflash-kquant.gguf --spec-draft-ngl 999 --spec-draft-n-max 15 ^ --spec-type draft-dflash --spec-draft-type-k f16 --spec-draft-type-v f16 ^ --reasoning-preserve ^ --temp 1.0 --top-p 0.95 --min-p 0.0 --top-k 64 --jinja --port 9090 

Caveat: #26842 is a draft PR, use at your own risk. Rebuild from master once merged.

submitted by /u/patricious
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA