4090 + 5060 Ti + 64GB RAM: 206 t/s on a 35B-A3B, and a 122B at 37 t/s
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
I've been benchmarking a two-card box for a few weeks and I still can't quite get over some of these numbers, so I'm dumping them here.
Box: RTX 4090 (24GB) + RTX 5060 Ti (16GB), i9-13900K, 64GB DDR5. WSL2 with 47GB allocated to the VM, CUDA 12.8 (12.8 specifically,13.1 segfaults llama.cpp's MMQ kernel on Blackwell and silently falls back to cuBLAS, which cost me ~6x on prompt processing before I figured that out). llama.cpp built for 89;120. Everything below is 131K context with q8_0 KV cache, measured on short-code generation.
| Model | Placement | MTP on | MTP off |
|---|---|---|---|
| Qwen3.6-27B dense, Q4_K_XL | 4090 only | 101–118 t/s | 44 t/s |
| Qwen3.6-27B dense, Q6_K_XL | both cards, layer split | 64 t/s | |
| Qwen3.6-35B-A3B, Q4_K_XL | 4090 + 6 expert layers spilled | 206 t/s | 113 t/s |
| Qwen3.5-122B-A10B, IQ3_S | 4090 + 5060 Ti + ~15GB in RAM | 37–41 t/s | 24 t/s |
The 122B one is the one I keep re-reading. That's a 122-billion-parameter model with 17 of its 49 layers living in system RAM, generating faster than most people's 8B setups. My own napkin estimate before I ran it was 20–30 t/s and I thought I was being optimistic.
Scripts and all the raw numbers are in a repo I put up (github.com/04RR/qServer). it's my own, mostly llama.cpp launch flags and regression gates rather than anything clever, but the RESULTS and LEARNINGS files have the full sweeps if anyone wants the ugly details (generated by Claude code ofc) .
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.