GLM 5.2 and ik_llama.ccp
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Running GLM-5.2 (the new glm-dsa arch), Unsloth UD-Q4_K_XL, on a 4-socket Xeon E7-8880 v4 box with 1TB RAM and a single RTX 3060 12GB. ik_llama.cpp, experts on CPU (--cpu-moe), 24 attention layers on the GPU. Works great at 8k context — rock solid, ~3.7 tok/s gen.
Problem: the second I raise context (anywhere past ~32–64k), generation crashes on the very first token. Fatal error in llama-sampling.cpp, and the dumped probabilities.txt shows every logit is nan (max = nan, sump = nan). 8k always works, long context always dies.
Stuff I've already tried, none of it helped:
-dsa on and off
-fa off (and on)
shrinking -ub all the way down to 32/64
-nkvo (KV on CPU)
both f16 and q8 KV cache
pulled the latest commits with the GLM-DSA indexer fixes (#2075, #2090, #2093)
Same all-NaN crash every time. Feels like an f16 overflow somewhere in the DSA/indexer path on the GPU, but I can't flag my way around it. Model's trained for 1M context so I'd really like to actually use it.
Is this a known glm-dsa long-context bug? Any workaround to keep it on the GPU, or is CPU-only genuinely the only path right now?
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.