Ran DeepSeek V4-Flash-0731 — the full official checkpoint, not a re-quant — on commodity used hardware. Sharing because I couldn't find anyone else publishing Ampere results for this engine.

Why bother with a 2018 server

The model is 156 GB. That number decides everything before speed matters:

Platform	Memory	Bandwidth	Price	Runs DS4-Flash?
Mac Studio M3 Ultra	96 GB max¹	819 GB/s	$3,999+	❌ won't load
DGX Spark	128 GB	273 GB/s	$4,699²	⚠️ 4-bit re-quant only, ~10 GB headroom
AMD Ryzen AI Halo	128 GB	~256 GB/s	$3,999	⚠️ same
RTX PRO 6000 Blackwell	96 GB	1,792 GB/s	~$9,000	❌ won't load
6× RTX 3090	144 GB	936 GB/s	~$6,600 cards alone	✅ (+ a chassis that takes 6 cards)
Used R940 + 2× 3090	512–768 GB	141 GB/s × 4 nodes	~$6K	✅ full checkpoint

¹ Apple pulled the 512 GB M3 Ultra option in March 2026 and the 256 GB in May — 96 GB is the current ceiling. ² Up from $3,999 at launch, explicitly attributed to DRAM costs.

Unified-memory boxes give you bandwidth in a small pool. A 4-socket server gives you a huge pool at lower per-node bandwidth — but four independent memory controllers running in parallel. For sparse MoE, where only ~13B of 284B params activate per token, capacity wins.

Inference platform

Lvllmds4-x v2.3.8 — guqiong96's SM80+ DeepSeek V4 specialization. A vLLM fork (base: yhfgyyf/vllm-deepseek-v4-sm89) with the lk_moe v2.3.1 CPU-GPU hybrid MoE engine doing NUMA-aware expert compute in system RAM. Prebuilt cp312 wheel from the GitHub release, no compiling.

Model

• DeepSeek V4-Flash-0731 · 284B total / 13B active MoE · official safetensors, 156 GB (48 shards)
• Quantization-aware trained — routed experts (~96% of params) ship natively in MXFP4. Nothing re-quantized. FP8 linears run weight-only, activations BF16, KV cache fp8_ds_mla.
• The sm_86 trick: no native FP8/FP4 compute on Ampere, so the fork routes everything through Marlin weight-only kernels (MXFP4 MoE backend + MarlinFP8 linears). That's how a Blackwell-era checkpoint runs on 2020 GPUs.
• DSpark speculative decoding (built into the checkpoint, 5 draft tokens) — where most of the single-stream speed comes from.

Hardware (all used/eBay-class)

• Dell PowerEdge R940 · 4× Xeon Platinum 8268 (96C/192T, Cascade Lake, AVX512-VNNI, no AMX)
• 768 GB DDR4-2933 (24× 32 GB, 6 channels/socket, 4 NUMA nodes)
• 2× RTX 3090 24 GB (sm_86), both PCIe x16, TP=2
• NVMe + SATA SSD for model storage

Current eBay pricing (Aug 2026): 96-core R940 with 128 GB runs $2,000–2,800; 512 GB around $3,800; 768 GB around $7,600. Add ~$2,200–2,600 for a pair of used 3090s.

You don't need 768 GB to run it. One instance needs ~170 GB, and with --membind pinning that has to fit on a single NUMA node — so 512 GB (128 GB/node) is roughly the entry point at ~$6K all-in. The extra RAM buys instances, not speed: going 22→24 DIMMs moved throughput ~5%, within noise.

Resource footprint while serving

• VRAM: 6.6 GB weights + KV per card (21.6/24 GB used) — GPUs sit at ~25% util
• System RAM: ~170 GB per instance (experts live in DRAM, streamed by CPU via lk_moe AVX512-VNNI kernels)
• Power (iDRAC/Redfish + nvidia-smi measured): ~1,000 W chassis under decode, 435 W idle. GPUs draw only 136–145 W avg (189 W peak). I power-capped both 3090s 350 W → 250 W and throughput didn't move a single tok/s — the cap never engages. ~95% of the load delta is 96 Xeon cores streaming experts from DRAM. At $0.13/kWh that's ~$94/month worst-case 24/7, far less at realistic duty cycle.

Results

128-token completions, temp 0, 22K max context, max-num-seqs 4, spec depth 5.

Concurrent	Aggregate	Per user
1	33 tok/s	33
4	53–68 tok/s	13–17
8	47–63 tok/s	6–8

(Ranges = cold first pass → warm steady state with prefix cache.)

For scale: the same box running the same model on ik_llama.cpp hybrid does 12.2 tok/s single-stream. The spec-decode + Marlin path is a 2.6× single / ~3× aggregate jump on identical hardware.

What didn't matter

Three separate things I expected to help and didn't:

• +2 DIMMs (22→24, symmetric 192 GB/node): ~5%, within noise
• GPU power cap 350→250 W: zero effect
• More GPUs: wouldn't help — they're at 25% util and 6.6 GB of 24

All three point the same way: the bottleneck is CPU-side DRAM bandwidth. This workload wants DDR5 and AMX (Sapphire Rapids), not more Ampere. If you're planning a build around this, spend on memory channels, not cards.

Gotchas that cost me hours

1. TileLang JIT-compiles kernels at runtime with whatever nvcc it finds — system CUDA 12.0 fails with cryptic lambda syntax errors. Point CUDA_HOME at the pip-bundled toolkit inside the venv (site-packages/nvidia/cu13). No system CUDA install needed.
2. The wheel's pip CUDA packages ship internally mismatched (nvcc 13.2 vs runtime headers 13.0) → CCCL "compiler and toolkit headers are incompatible". Fix: pip install nvidia-cuda-runtime==13.2.86 nvidia-cuda-nvrtc==13.2.86.
3. Undocumented DSpark constraint, found the hard way: max_num_seqs × (spec_tokens + 1) must be ≤ 32 or engine warmup dies with a tensor-size mismatch. seqs=4 × spec=5 is the sweet spot — wider batches with shallower spec were slower everywhere.
4. MiniMax and other non-DeepSeek MoE on this fork still hit the sm_86 vectorized_gather_kernel assert from generic LvLLM. The Ampere fixes are DS4-path only.

Happy to share the full launch command / venv recipe in comments.