r/LocalLLaMA · · 1 min read

MTP decoding patched for pre-Ampere GPUs (Kepler/Maxwell/Pascal/Turing)

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

MTP decoding patched for pre-Ampere GPUs (Kepler/Maxwell/Pascal/Turing)

The current implementation of multi token prediction (MTP) in llama cpp could trigger BF16 compute selection on GPUs that don't support BF16, causing cuBLAS crashes on older architectures (tested on Kepler).

I patched llama.cpp's CUDA backend to add a robust capability check: - BF16 supported → use BF16 - No BF16 but fast FP16 available → fall back to FP16 - Older GPUs (Kepler etc.) → fall back to FP32

This keeps MTP working on older cards without affecting newer GPUs.

Tested with: - Qwen3.6 35B A3B q4xl - Q4XL quantization - Tesla K40c (Kepler, Overclocked) (4x)

Performance: - Stock llama.cpp: ~22 tok/s - MTP disabled: ~17.5 tok/s - MTP enabled (2 tokens): ~25 tok/s

submitted by /u/TheSpicyBoi123
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA