r/LocalLLaMA · · 1 min read

GLM 5.2 and ik_llama.ccp

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

Running GLM-5.2 (the new glm-dsa arch), Unsloth UD-Q4_K_XL, on a 4-socket Xeon E7-8880 v4 box with 1TB RAM and a single RTX 3060 12GB. ik_llama.cpp, experts on CPU (--cpu-moe), 24 attention layers on the GPU. Works great at 8k context — rock solid, ~3.7 tok/s gen.

Problem: the second I raise context (anywhere past ~32–64k), generation crashes on the very first token. Fatal error in llama-sampling.cpp, and the dumped probabilities.txt shows every logit is nan (max = nan, sump = nan). 8k always works, long context always dies.

Stuff I've already tried, none of it helped:

-dsa on and off

-fa off (and on)

shrinking -ub all the way down to 32/64

-nkvo (KV on CPU)

both f16 and q8 KV cache

pulled the latest commits with the GLM-DSA indexer fixes (#2075, #2090, #2093)

Same all-NaN crash every time. Feels like an f16 overflow somewhere in the DSA/indexer path on the GPU, but I can't flag my way around it. Model's trained for 1M context so I'd really like to actually use it.

Is this a known glm-dsa long-context bug? Any workaround to keep it on the GPU, or is CPU-only genuinely the only path right now?

submitted by /u/naunen
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA