r/LocalLLaMA · · 1 min read

Qwen 3.6 27B Q5 on 3x2080ti: 55tps with llama.cpp. Can I squeeze out more?

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

CPU: Threadripper 3970X
RAM: 128GB DDR4
GPUs: 3x2080ti 11GB

The current best parameters to run it:

llama-server \ --model Qwen3.6-27B-Q5_K_S.gguf \ --n-gpu-layers 999 \ --split-mode tensor \ --flash-attn on \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --ctx-size 16384 \ --batch-size 2048 \ --ubatch-size 1024 \ --threads 4 \ --spec-type draft-mtp \ --spec-draft-n-max 3 \ --no-mmap 
submitted by /u/AccountGotLocked69
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA