Pipeline parallelism in llama.cpp may be wasting your VRAM

By default, llama.cpp enables pipeline parallelism, presumably to speed up inference. In my testing, I found that pipeline parallelism has no speed benefit and comes at a significant cost of VRAM.

This cost can be avoided by compiling llama.cpp with the -DGGML_SCHED_MAX_COPIES=1 option. This prevents llama.cpp from allocating a much larger compute buffer when pipeline parallelism is enabled.

Pipeline parallelism is enabled when --split-mode layer is used (the default) and all model layers and all compute is offloaded to the GPU. If compiled with the default options, llama.cpp allocates four sched copies instead of one when pipeline parallelism is enabled. I don't know exactly what a sched copy is, but it's a significant contributor to the size of the compute buffer in VRAM. Four copies consume significantly more VRAM, especially when context cache quantization is used.

I did a whole lot of testing to confirm that, with my setup at least, allocating those four sched copies is a complete waste of VRAM. There is no speedup whatsoever.

edit: Multiple users have pointed out in the comments that pipeline parallelism is beneficial when submitting parallel requests. I didn't test that. If you often submit parallel requests, you can test for yourself and see if the speedup is worth the VRAM cost.

For this test, I compared three builds of llama.cpp, all using the Vulkan backend. The first build used the default option, GGML_SCHED_MAX_COPIES=4. The second used GGML_SCHED_MAX_COPIES=1. The third used GGML_BLAS=ON GGML_BLAS_VENDOR=OpenBLAS which, I discovered, coincidentally disables pipeline parallelism.

This is the llama.cpp command I ran with each of the three builds:

./llama-server -m models/Qwen3.6-27B-MTP/Qwen3.6-27B-UD-Q5_K_XL.gguf \ --verbosity 4 \ --no-op-offload \ -fa on \ --mlock \ -ngl 999 \ --temp 0.6 --top-k 20 --top-p 0.95 --presence-penalty 0.0 \ --cache-type-k f16 --cache-type-v q8_0 \ --host 0.0.0.0 

Here are the data I collected after three trials:

As you can see, inference speed was virtually identical in all configurations. However, the compute buffer size was much larger with pipeline parallelism and 4 sched copies, which is the llama.cpp default. It consumed an additional 1.5 GB of VRAM with my specific model and settings compared to the other configurations.

The compute buffer bloat seems to be much worse if context cache quantization is used. I tried the same test without the --cache-type-k f16 --cache-type-v q8_0 options and got the following results:

In this test, the compute buffer was "only" about 0.5 GB bigger with pipeline parallelism and 4 sched copies.

The compute buffer bloat with four sched copies and context quantization is so severe that it partially cancels out the VRAM savings of quantizing the cache!

Of course, all of these findings are specific to my computer and llama.cpp settings. Your results may vary.

Here's more information about my system: