Llama.cpp : Split Mode Tensor Fix Incoming?
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Appears thay have been cooking and we might see a fix soon released for crashes on split mode tensor
Multi-gpu folks keep watch -
( In my tests SM Tensor has a ~35% uplift in TG over Layer but ofc crashes every 90-120 minutes due to vram exhaustion this fix is supposed to stop that )
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.