I didn't know it was possible to compile llamacpp to run cuda + vulkan at the same time..

cmake -B build -G "Visual Studio 17 2022" -A x64 -DCUDAToolkit_ROOT="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1" -DGGML_CUDA=ON -DGGML_VULKAN=ON -DGGML_FLASH_ATTN=ON -DGGML_BLAS=OFF -DGGML_NATIVE=OFF -DGGML_RPC=ON -DGGML_BACKEND_DL=ON -DCMAKE_CUDA_ARCHITECTURES="120a" -DGGML_CUDA_FA_ALL_QUANTS=true

I was optimizing Vulkan for the w7800 using ds4 on opencode and ds4 told me to "compile cuda + vulkan" to take advantage of the two cards. I had already tried cuda + hip but not vulkan.

It's late now, but tomorrow I'll try some benchmarks to understand the real benefits.

Has anyone tried it yet?

I have about +10% tokens/sec (decoding) with MiniMax-M3-UD-IQ2_M-00001-of-00004.gguf