Qwen3.8-27B Q4_K_M on 2x3060

For the last few days, I've been using two computers to run multiple agents.

My 4x3090 machine is running Qwen 3.8 27B with parallel=2, so I can run two agents at the same time. My pi instances are running on a machine with 2x3060, running/managing smaller models such as Gemma 26B A4B or Qwen 35B A3B.

Today, however, I needed my 4x3090 machine for some vLLM work, so I was missing my AI. I decided to try running Qwen 3.8 27B on the 2x3060 machine instead.

Here is the command:

#!/bin/bash ~/git/llama.cpp/build/bin/llama-server \ -sm tensor \ -lv 4 \ -m ~/LLMs-huge/Qwen3.8-27B-UD-Q4_K_M.gguf \ -c 50000 \ --host 0.0.0.0 \ --jinja \ -fa on \ --keep 4096 \ -b 8192 \ -ub 512 \ --no-kv-unified \ --fit-target 1024 \ --ctx-checkpoints 12 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0 \ --presence-penalty 0 \ --repeat-penalty 1.0 \ --spec-type ngram-mod \ --spec-type draft-mtp \ --spec-draft-n-max 3 \ --chat-template-kwargs '{"preserve_thinking":true}' 

And here are some real-world speeds from actual usage:

7.21.365.095 I slot print_timing: id 3 | task 4179 | prompt eval time = 873.12 ms / 48 tokens ( 18.19 ms per token, 54.98 tokens per second) 7.21.365.099 I slot print_timing: id 3 | task 4179 | eval time = 2760.36 ms / 139 tokens ( 20.00 ms per token, 49.99 tokens per second) 7.21.365.099 I slot print_timing: id 3 | task 4179 | total time = 3633.48 ms / 187 tokens (...) 7.37.423.312 I slot print_timing: id 3 | task 4225 | prompt eval time = 1640.55 ms / 325 tokens ( 5.05 ms per token, 198.10 tokens per second) 7.37.423.315 I slot print_timing: id 3 | task 4225 | eval time = 14163.13 ms / 630 tokens ( 22.52 ms per token, 44.41 tokens per second) 7.37.423.316 I slot print_timing: id 3 | task 4225 | total time = 15803.68 ms / 955 tokens (...) 7.44.951.668 I slot print_timing: id 3 | task 4445 | prompt eval time = 888.79 ms / 40 tokens ( 22.22 ms per token, 45.01 tokens per second) 7.44.951.672 I slot print_timing: id 3 | task 4445 | eval time = 6364.50 ms / 277 tokens ( 23.06 ms per token, 43.37 tokens per second) 7.44.951.673 I slot print_timing: id 3 | task 4445 | total time = 7253.29 ms / 317 tokens 

Hopefully this helps anyone wondering how usable 3060s still are for local LLM, the main problem is short context (too short for long agentic session)

https://preview.redd.it/bjtsld02btrh1.png?width=1854&format=png&auto=webp&s=e974f8668ba9ba9c5b16c8ab27273dd1562900bd