llama.cpp releases · · 1 min read

b10707

Mirrored from llama.cpp releases for archival readability. Support the source by reading on the original site.

kv-cells: stop the sequence scan once all sequences are seen (#28011)

for_each_token_in tested all LLAMA_MAX_SEQ sequences for every used cell,
while a cell almost always belongs to one. The scan now stops once the
cell's own sequences have been seen. Same visit order, same callback
arguments, so behaviour is unchanged.

get_prev_tokens is the only caller, so this affects the n-gram path.

RTX PRO 6000, Qwen3.8-Flash-Next UD-Q4_K_XL, fa on, warm runs:

55k context generation 56.3 -> 74.3 t/s
132k context generation 33.6 -> 50.9 t/s

Prompt processing is unchanged, the scan is amortised over the ubatch
there. The gain follows the number of used cells, so it grows with
context and is invisible on short prompts.

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from llama.cpp releases