Qwen3.8-Flash-Next in llama.cpp from CPU-only to 96GB VRAM: 8.5 to 109 tok/s, max context and parameters test. My findings on RTX 6000 PRO.
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
| | Hey guys, I tested Qwen3.8 Flash with llama.cpp from CPU-only to the full 96GB of my RTX PRO 6000. Short version:
Setup
I started a fresh server for each configuration. I waited for the previous VRAM allocation to disappear and for the GPU to cool. Each run saved the resolved configuration, server log, output, memory use and GPU telemetry. Important note about the VRAM rangesI used the same RTX PRO 6000 for every GPU test. A helper process reserved GPU memory, so llama.cpp saw a smaller usable VRAM pool. This tests VRAM capacity and CPU offload. It does not simulate the compute power or bandwidth of a real 8GB or 24GB GPU. The 8GB result does not mean that every 8GB card will reach the same speed. VRAM resultsAll numbers below use a 2,048-token prompt. CPU-only:
All GPU tests use the same RTX PRO 6000. The limits simulate memory capacity, not smaller GPU performance. 1. The model runs on the CPUAt a 2K prompt, CPU-only reached 182.64 tok/s prefill and 8.34 tok/s decode. This is enough for an interactive chat. The MoE design helps because the model activates only 6B parameters for each token. 2. The VRAM tiers converge at long contextAt a 2K prompt, 96GB was 2.796x faster than 24GB. At a 245K prompt, the advantage decreased to 1.451x. Decode at 245,760 prompt tokens:
Every configuration loses speed at long context. The fastest configuration loses most of its lead. Only 12 of the 48 layers keep a growing attention cache. The other 36 use Gated DeltaNet. This keeps context memory relatively low, but it does not make long-context decode free. 3. PLE on CUDA was 55.6x slower in this buildThe GGUF contains a 27.2 GiB per-layer token embedding table. I tested two placements:
The CUDA placement was 55.6x slower on decode. I verified that the tensor moved, but I did not isolate the cause of the slowdown. I repeated the test in A-B-B-A order. The order effect was 0.56%. The memory data also confirmed the placement. GPU use increased by approximately 27.5 GiB when the table moved to CUDA. The CPU placement needs approximately 9.2 ms per decode token. The CUDA placement needs approximately 513 ms per decode token. That delay is too large to explain with arithmetic alone. It looks more like synchronization or a per-token transfer, but I have not proved the cause. My conclusion is limited to this build:
4. RAM-resident loading gave 1.87x more prefillI compared mmap with RAM-resident loading at the same 48GB tensor placement. At the 2K prompt:
The decode ratio was 0.998, so decode was effectively unchanged. The option is:
This means that llama.cpp does not use file mapping. It does not mean that the model is not loaded. The mode needs enough free system RAM. 5. KV layout changed concurrencyI tested unified and non-unified KV layouts from 1 to 16 concurrent requests. Both started at 59.0 tok/s with one request. At concurrency 16:
Non-unified KV gave more total output at high concurrency. Each individual request still became slower. Concurrency increased total server capacity. It did not make one request faster. Non-unified KV also divides the available context between slots, so it is not always the correct setting. Results I left out for nowThe report also contains:
The final numbers are present in my repo and video, ResourcesGitHub with the report, scripts, configurations, results and graphs: https://github.com/lukaLLM/Qwen3.8-Flash-Next-VRAM-Benchmark Full video: PS: AI was abused while making edits My main question is about the PLE CUDA result. Did anybody reproduce the same slowdown with My other question is about keeping earlier thinking in the conversation. I ran the same five-turn coding conversation with earlier reasoning either kept or removed from later prompts. Keeping it reduced prompt tokens recomputed from 18,403 to 267 because the history stayed append-only. However, the turn-5 prompt grew from 18,387 to 63,223 tokens, and decode ended at 48.9 instead of 65.5 tok/s. This was only one run per arm with temperature 1.0. The arms also started at different speeds, 110.2 and 96.0 tok/s, so I do not trust the exact 69x and 25% figures yet. Did anybody reproduce this tradeoff? Does keeping the earlier reasoning normally save this much prompt recomputation, and how much should the longer prompt reduce decode speed? Or any other finding to increase the efficiency of this model. [link] [comments] |
More from r/LocalLLaMA
-
yandex/AliceAI-Foundation-80B-A3B-Base: Russian-developed competitor to Qwen 35B and DeepSeek V4 Flash
Sep 21
-
Huawei shelves global AI chip rollout as China's own demand outstrips supply — AMD and Nvidia no longer have to worry.
Sep 21
-
A better coder for the small-GPU/small-RAM crowd!
Sep 21
-
How it feels watching prices go up
Sep 21
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.