After a couple of days of faffing around, here's my current config in case it helps anyone out. Some of this is also valuable for a single Spark.

Benchmarks:

Dual-Spark TP2, eager + MTP k=3, 262k context (warmed medians, exact tokenizer counts): ┌─────────┬──────────────────┬───────────────────┬─────────────────────────────────┐ │ Streams │ Decode aggregate │ Decode per-stream │ Prefill aggregate (10k prompts) │ ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤ │ 1 │ 45.9 t/s │ 45.9 t/s │ 2,940 t/s │ ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤ │ 4 │ 120.2 t/s │ 30.0 t/s │ 2,524 t/s │ ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤ │ 8 │ 222.2 t/s │ 27.8 t/s │ 3,098 t/s │ ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤ │ 13 │ 265.9 t/s │ 20.5 t/s │ 2,960 t/s │ └─────────┴──────────────────┴───────────────────┴─────────────────────────────────┘ Prefill vs prompt depth (single stream, exact tokens): ┌─────────────┬───────────┐ │ Prompt size │ Prefill │ ├─────────────┼───────────┤ │ 11k │ 2,875 t/s │ ├─────────────┼───────────┤ │ 100k │ 2,655 t/s │ ├─────────────┼───────────┤ │ 200k │ 2,463 t/s │ └─────────────┴───────────┘ 

AI slop below for you to read and paste into your own agent :)

Stack

• vLLM PR #53896 (release/qwen38next branch) — NOT main; main doesn't have the model, and the recipe image's vLLM commit isn't in the public repo
• sm_121 support is a 2-file patch (details below) — GB10 is arch 12.1 and the NVFP4 E2M1 conversion needs a software fallback
• PLE n-gram table served from internal NVMe via mmap:(48 GB, MADV_RANDOM is essential — it's a 30× read-amplification difference on hash-scattered row lookups), with a node-local CPU-worker process doing gathers over ZMQ + pinned buffers + CUDA-IPC outputs so the gather is graph-safe and TP2-safe
• TP2 across both Sparks: native venv (NOT docker — see dead ends), eager + MTP k=3

Numbers (TP2, warmed medians, count-20 structured / 3-paragraph prose)

Dead ends

1. PIECEWISE cudagraphs cost ~28% decode under MTP+TP2.
2. Same tree, same everything, graphs on vs off: 36 vs 50 tok/s. Graphs help single-node no-MTP decode (+80%, 9→16.5), but combined with MTP at TP2 they're a straight loss — MTP already amortizes the launch overhead the graphs were eliminating. We chased a "tree-level MTP regression" for a day before realizing the config was the regression.
3. Docker silently degrades NCCL to TCP sockets.
4. Default containers get no IB device → NET/Socket → TP2 prefill at ~40% of native. You need --privileged (or IB device passthrough) AND the right HCA name — see next point. Native venv runs just get RoCE.
5. RoCE device names are not stable across reboots, and can differ between two identical machines
6. (ours: rocep1s0f1 on one, rocep1s0f0 on the other, resolved dynamically). Hardcoded NCCL_IB_HCA will silently fail on one node. Enumerate with ibdev2netdev, match port-ACTIVE to your rail netdev, and verify with NCCL_DEBUG=INFO that you see NET/IB, not NET/Socket.
7. flashinfer must be 0.6.18
8. on GB10 — 0.6.17 crashes the NVFP4 MoE fallback kernel.
9. vLLM's MoE shuffleInputRowsKernel has an uninitialized-permutation OOB read that shows up as a fake CUTLASS status=7 GEMM failure — a two-line bounds guard fixes it (already in PR #53896).

The 2-file sm_121 patch (vs PR #53896 base)

1. CMakeLists.txt: add 12.1 to CUDA_SUPPORTED_ARCHS, SCALED_MM_ARCHS, CUTLASS_MOE_DATA_ARCHS, FP4_SM120_ARCHS
2. csrc/libtorch_stable/quantization/fp4/nvfp4_utils.cuh: ~90-line software E2M1 conversion guarded by __CUDA_ARCH__ == 1210 (GB10 has no cvt.rn.satfinite.e2m1x2.f32)

That's the whole delta. Build offline with the usual dependency cache; CUDA arch 121.

Startup command(s)

Head node (rank 0):

export VLLM_PLE_MMAP=1 VLLM_PLE_MMAP_WORKERS=64 VLLM_PLE_MMAP_PREWARM=1 export VLLM_PLE_MMAP_DIR=$HOME/ple-table # internal NVMe copy of the table export VLLM_PLE_CPU_OFFLOAD=1 # activates the IPC CPU-worker gather path export NCCL_SOCKET_IFNAME=<your-rail-if> # bootstrap only; NCCL finds RoCE itself export GLOO_SOCKET_IFNAME=<your-rail-if> python -m vllm.entrypoints.openai.api_server \ --model <path-to-qwen38-flashnext-nvfp4> \ --host 0.0.0.0 --port 8086 --load-format safetensors \ --tensor-parallel-size 2 --nnodes 2 --node-rank 0 \ --master-addr <head-rail-ip> --master-port 29511 \ --distributed-executor-backend mp \ --max-model-len 262144 --max-num-seqs 13 --gpu-memory-utilization 0.85 \ --no-enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 8192 \ --long-prefill-token-threshold 4096 --enforce-eager \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' 

Worker node (rank 1): identical, but --node-rank 1 --headless, launched ~30s after the head. Note --enforce-eager — that's not a compromise, it's the fast path (see dead end #1).