llama.cpp releases · May 14, 2026 · 1 min read

b9148

Mirrored from llama.cpp releases for archival readability. Support the source by reading on the original site.

unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr… (#22110)

unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regression tests

Add unicode_regex_split_custom_qwen35() to src/unicode.cpp, a non-backtracking handler for Qwen3.5's [\p{L}\p{M}]+ regex (letters + combining marks).
Register the handler in the custom tokenizer dispatch table to prevent stack overflows on long inputs (fixes #21919).
Add models/ggml-vocab-qwen35.gguf (test vocab), models/ggml-vocab-qwen35.gguf.inp (test cases), and models/ggml-vocab-qwen35.gguf.out (expected output) for regression testing.
Update tests/CMakeLists.txt to include the new test entry.

This mirrors the Qwen2 fix (commit 0d049d6), but adapts for Qwen3.5's regex. Ensures robust Unicode tokenization and prevents std::regex stack overflows.

Closes #21919.

fix: enhance regex handling for Qwen3.5 tokenizer to include accent marks
cont : remove trailing whitespace

Co-authored-by: Kabir [email protected]
Co-authored-by: Alde Rojas [email protected]

macOS/iOS:

Linux:

Android:

Android arm64 (CPU)

Windows:

openEuler:

Discussion (0)

No comments yet. Sign in and be the first to say something.

Discussion (0)

More from llama.cpp releases