Dear 24G owners, try VLLM you might be able to run Qwen3.8 27B INT4, 144K FP8 KV on RTX 3090 with better speed. (TLDR VLLM AOT)

VLLM Benchmark:
Prefill, Prompt processing
- avg, 871.93 tok/s (3 hours constant running xhigh)
- 10K prompt, 1000.26 tok/s (16 runs)
- 90K prompt, 743,59 tok/s (16 runs)

Decode, tok gen
- avg, 38.39 tok/s (3 hours constant running xhigh)
- 10K, 42.3 tok/s (16 runs)
- 90K, 34 tok/s (16 runs)

Preamble: I am on WSL2. Running the 27B Q5 UD GGUF through llama.cpp with 81,920 context plus MTP gives me around 25-30 tok/s. Then I found this GitHub repo: https://github.com/noonghunna/club-3090

It is basically a recipe and Docker configuration for running the model.

So, 30 tok/s itself is fine, but I just got bored waiting for RunPod to open its GPUs. I finally brought my vLLM tuning back from the back burner, and here I am.

I often forget that Inductor/Triton compilation and CUDA Graph capture require additional VRAM while testing the configuration and kernel calls. When JIT compilation failed because of an OOM, I never bothered trying AOT.

FYI, AOT and JIT are compilation strategies. AOT means Ahead of Time, while JIT means Just in Time.

If you OOM on the first startup, try it one more time. Inductor might have already compiled and cached part of the configuration before the OOM, allowing the next run to reuse it if the configuration has not changed. This is not guaranteed, but it worked for me.

And yes, it was trial and error. It was kinda tedious and pain in the ass, starting from 32K, then 64K, 80K, 128K, and finally 144K. The practical ceiling for my conf at 154K, but I chose 144K. I also started the batch size at 256 and climbed to 1024, although I might be able to squeeze in 1280-1536.

Also, beware of your vLLM compilation cache. It might grow to 5-6GB after testing many configs. Personally, I delete the old cache and run the final configuration again twice so it rebuilds only what I currently use.

My current setup runs Qwen3.8-27B with INT4 AutoRound weights through vLLM while using an FP8 E4M3 KV cache. It fits on one GPU with a configured context window of 147,456 tokens.

Although I should say, with GDN, or really any linear-attn, vLLM can be kinda bad at predicting how much VRAM the KV and state-cache pools will require.

Benchmark:

https://github.com/noonghunna/benchlocal-cli

This is the deterministically scored, no-Docker portion of BenchLocal: 75 scenarios covering tool calling, instruction following, structured output, data extraction, and reasoning/math.

Pack	Score	p50
ToolCall	14/15 (93%)	3.21s
InstructFollow	15/15 (100%)	6.97s
StructOutput	14/15 (93%)	7.74s
DataExtract	14/15 (93%)	11.63s
ReasonMath	14/15 (93%)	9.50s
Total	71/75 (94.7%)	—

Thinking was forced on with reasoning_effort=low. The run took about 15 minutes. Yep, even with low reasoning effort and INT4 weights, it passed 71/75.

Setup

• GPU: RTX 3090 245W, 24GB, 48G RAM DDR4 (Exposed from WSL2), E5 2690V4
• Model: https://huggingface.co/Avuja/Qwen3.8-27B-int4-AutoRound
• Inference engine: vLLM 0.27.1 with patch from Club 3090

The important vLLM settings were:

• --dtype bfloat16
• --tensor-parallel-size 1
• --max-model-len 147456
• --gpu-memory-utilization 0.9475 (This is the painful one to redo.)
• --max-num-seqs 1 (Yep single serving only, you could change this to 2, but the KV will be cut ofc active requests will have to share the same total KV capacity.)
• --max-num-batched-tokens 1024 (Prefill stuff / prompt processing)
• --long-prefill-token-threshold 1024 (Prefill stuff / prompt processing)
• --kv-cache-dtype fp8_e4m3
• --enable-prefix-caching
• --enable-chunked-prefill
• --mamba-cache-mode align (GDN stuff)
• --prefix-match-unit 16
• --language-model-only

Full command : https://gist.github.com/komikndr/b17955e1a80ce6ede9a3115f16216bc5#vllm-qwen-27b-38-just-remove-or-add-flag-as-you-like

Forgot to mention, no MTP and no MultiModal, i max the CTX, multimodal is at 64-65K ish but at that point i'll just use Llamacpp. And also again this is WSL2, if you are on baremetal, you could improve more speed