r/LocalLLaMA · · 1 min read

Are there more easy techniques than --tensor-split to fill VRAM in llama.cpp?

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

Using 4 GPUs with llama.cpp, with MoE models mainly, I try to fit as much in VRAM as I can. --fit does a terrible job and always causes oom by trying to put way too much on 1 gpu or stupid things like that, so I do --ngl 999 and --n-cpu-moe and adjust till I get enough into vram, then use --tensor-split and spend a while tweaking the numbers until I manage to balance the layers across GPUs. Whenever I try a new model it usually takes a good few hours of playing around to find the exact right numbers to fit as much as I can into VRAM, find the optimal context size and speed tradeoff etc.

But, with this, I often do have something like 2-5gb of free VRAM on each GPU, because even shifting the layer numbers by one will cause one gpu to have too much on it and oom, so I have to balance them to the point where it all fits, but I feel like I'm always leaving like 8-12gb of vram on the table that I can't seem to fill. I can increase context size to get a bit more on there, but when I don't need context that high and just want extra speed, I can't seem to get any more of the model loaded on there just using --tensor-split.

Do I need to get into the crazy giant commands people have overriding specific tensors to help fill the space?

submitted by /u/GregoryfromtheHood
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA