Hi everyone,

I've spent about four days now trying to find the eight configuration for running Qwen 27b in production using VLLM but have been getting significant decreases in performance with the FP8 safetensors in comparison to the llama.cpp variants, especially under load/high concurrency. Our quality benchmarks drop by almost 20% in absolute terms. Would some mind sharing their VLLM config they use in production that execute correctly? I'm at a loss at this point :)

Thanks in advance!

(Update) Current config:

Bash Run Script: ``` set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" CONFIG="${1:-${SCRIPT_DIR}/qwen3.6-27b-fp8.yaml}"

gguf2ct-converted Unsloth UD-Q4_K_XL (compressed-tensors W4A16/W8A16 mixed,

MTP weights included; text-only — requires language-model-only in the yaml).

MODEL="Qwen/Qwen3.6-27B-FP8"

---------------------------------------------------------------------------

Environment Variables

---------------------------------------------------------------------------

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

if omitted, hallucinates tool names

export VLLM_ENFORCE_STRICT_TOOL_CALLING=1

---------------------------------------------------------------------------

Launch vLLM

---------------------------------------------------------------------------

echo "Starting vLLM server..." echo "Endpoint will be available at http://0.0.0.0:667" echo "Press Ctrl+C to stop." echo ""

exec "${VLLM_BIN}" serve "${MODEL}" --config "${CONFIG}" ```

Model Config ``` model: Qwen/Qwen3.6-27B-FP8 served-model-name: qwen3.6-27b trust-remote-code: true

I have tried both

chat-template: /nfs/models/Qwen3/Qwen3.6-27b-chat-template-v2.jinja

---------------------------------------------------------------------------

Server

---------------------------------------------------------------------------

host: 0.0.0.0 port: 667 api-key: "default"

tensor-parallel-size: 1 pipeline-parallel-size: 1

gpu-memory-utilization: 0.95 dtype: bfloat16

---------------------------------------------------------------------------

max-model-len: 131072 max-num-seqs: 256 max-num-batched-tokens: 8192 enable-prefix-caching: true

async-scheduling: true

speculative-config: '{"method":"mtp","num_speculative_tokens":2}'

tool-call-parser: qwen3_coder enable-auto-tool-choice: true structured-outputs-config: '{"reasoning_parser":"qwen3","enable_in_reasoning":true}'

override-generation-config: '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repetition_penalty": 1.0}' reasoning-config: '{"reasoning_start_str": "<think>", "reasoning_end_str": "\n[Budget reached: Reasoning truncated.]</think>"}' default-chat-template-kwargs: '{"preserve_thinking": true}' ```