llama.cpp releases
500 articles archived · Visit source ↗ · RSS
-
llama.cpp releases dev-tools 1mo ago
b9956
cli: fix crash on wrong server base url ( #25497 ) llama-cli: fix crash on wrong server base url by catching exceptions and graceful exit review: leaner catch group: json error and standard exception macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI…
20 -
b9952
llama : make all KQ masks f16 if FA is used, remove zero attention bias, remove raw_k repeats in DeepSeek V4 ( #25370 ) llama : make all KQ masks (except the lightning indexer one) f16 if FA is used and remove zero attention bias in DeepSeek V4 llama : remove dead code that…
9 -
b9950
llama-batch: add unit test ( #25471 ) llama-batch: add unit test fix win32 builds add not implemented assertion in unused methods remove unreachable code macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS…
23 -
b9949
opencl: cluster-parallel decode FA for Adreno ( #25473 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64…
37 -
b9948
ggml : process data in smaller chunks in CUDA ggml_top_k() and ggml_argsort() to reduce temporary buffers memory usage ( #24776 ) ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage ggml : allocate tmp_dst only only…
14 -
b9947
cli: add --output option ( #25484 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64…
31 -
b9951: ggml-et: Initial ET backend (#24179)
ggml-et: Add performance logging ggml-et: Quants helpers ggml-et: Add MUL_MAT kernel ggml-et: Add ROPE kernel ggml-et: Add RMS_NORM kernel ggml-et: Add GLU kernel ggml-et: Add SOFT_MAX kernel ggml-et: Add GET_ROWS kernel ggml-et: Add CONT kernel ggml-et: Add SET_ROWS kernel…
23 -
b9946
hexagon: tiling, tracing and optimizations for unary ops ( #25474 ) hexagon: tile wide rows in pointwise unary ops to avoid VTCM overflow unary: reject permuted tensors for now (not used by models) hex-unary: replace divs with fastdiv hex-unary: add vtcm layout and host computed…
24 -
b9945
server : move chat-template thinking probe inside the init try/catch ( #24093 ) A model whose chat template parses at init but fails parser generation at apply time (e.g. uses {% call %}) throws std::invalid_argument from common_chat_templates_support_enable_thinking(), which…
4 -
b9941
Only index by compile times + always multiply/add ( #25445 ) The first one avoids relying on compile to optimize local memory away, and the second is cheaper than issuing control flow statements macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled)…
28 -
b9940
llama-bench : init params.offline ( #25476 ) Signed-off-by: Adrien Gallouët [email protected] macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x…
19 -
b9939
metal : add CONV_2D_DW (depthwise convolution) support ( #21565 ) metal : add CONV_2D_DW (depthwise 2D convolution) support test : add perf cases for CONV_2D_DW metal : use 3D dispatch for CONV_2D_DW kernel metal : add channel-tiled CONV_2D_DW kernel for non-contiguous layouts…
10 -
b9938
ggml-hip: enable -funsafe-math-optimizations ( #24668 ) CUDA is compiled with fast math and AMD/HIP is not — this flag lets AMD use fast math too. We can't use -ffast-math: it implies -ffinite-math-only, which won't compile (ggml uses INFINITY for masking) and produces NaNs.…
33 -
b9937
cuda: align snake fusion matcher with the other backends ( #25460 ) cuda: fix snake fusion type predicate, a and inv_b are F32 The matcher required a->type == x->type while launch_snake reads both as const float *, matching the CPU and Metal contract where a and inv_b stay F32.…
21 -
b9936
server : respect min-step when splitting prompt batches ( #25420 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan)…
12 -
b9935
hexagon: add VISION RoPE support ( #25216 ) hexagon: add VISION RoPE support hexagon: support RoPE on strided half-dim views for all modes hex-rope: decouple src0 DMA copy size from row stride hex-rope: support non-contiguous dst for RoPE hex-rope: fix dst spad pitch for…
15 -
b9934
ggml-webgpu: tune subgroup split (d_split) in flash_attn_vec ( #25418 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64…
20 -
b9933
opencl: Q6_K GEMM/GEMV fix for ne01 of weights that are not multiples of 128. ( #25464 ) opencl: fix garbled output for Q6_K weights with ne01 % 128 != 0 on Adreno Observed with granite-3.1-3b-a800m-instruct, whose vocab is an odd number. Route Q6_K dense mul_mat with ne01 % 128…
36 -
b9932
vulkan: disable FA mask_opt on GCN to improve performance ( #24362 ) vulkan: disable FA mask_opt on GCN to improve performance reenable mask opt over attention head size 256 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel…
27 -
b9931
opencl: ragged-tile MoE prefill FP16 GEMM optimization (skip padded expert tiles) ( #25433 ) opencl: ragged-tile MoE prefill GEMM (skip padded expert tiles) The MoE prefill GEMM groups tokens into TILESIZE_N=32 per-expert tiles; at low tokens-per-expert most tiles are mostly…
34 -
b9930
llama-batch: fix allowed decreasing pos in a seq ( #25449 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu…
28 -
b9929
vulkan: for small AMD GPUs, reduce submission threshold based on CU count ( #25240 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU)…
17 -
b9928
hexagon: new vtcm layouts and improved pipelines for MUL_MAT, MUL_MAT_ID and FLASH_ATTN_EXT ( #25425 ) hex-fa: refactor kernel param compute to use common layout builder hmx: add explicit compiler barriers to make hmx funcs more robust hex-vtcm: more generic vtcm layout builder…
15 -
b9927
cli : move to HTTP-based implementation ( #24948 ) cli: move to HTTP-based implementation wip working remote server ok cli support router mode Co-authored-by: Piotr Wilkin [email protected] case: router with only one model Apply suggestions from code review Co-authored-by: Piotr…
12 -
b9925
cuda : add support for f16->f16 GGML_OP_SET_ROWS ( #25367 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu…
22 -
b9924
llama: refactor fused ops ( #24646 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64…
24 -
b9923
server-stream: follow-up on SSE Replay Buffer ( #23226 ) ( #25047 ) server-stream : pimpl server-stream: prefix free functions with server_stream_ address review from ggerganov: scope the public stream functions under the server_stream_ prefix, matching…
34 -
b9922
llama-batch: add n_keep_tail in split_equal for recurrent models ( #25278 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64…
36 -
b9918
metal : add set_rows with src0 f16 ( #25434 ) Co-authored-by: Georgi Gerganov [email protected] macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu…
9 -
b9916
ggml : fix A indexing in simd_gemm scalar tail-column path ( #25390 ) simd_gemm() has an incorrect A-matrix index in the scalar tail-column path for full row blocks. macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64)…
25 -
b9915
ggml : add support for CPU f16->f16 GGML_OP_SET_ROWS ( #25344 ) ggml : add support for CPU f16->f16 GGML_OP_SET_ROWS ggml : add missing type checks in f16 GGML_OP_SET_ROWS ggml : merge ggml_compute_forward_set_rows_f32() and ggml_compute_forward_set_rows_f16() into…
19 -
b9914
opencl: fix potential crash in aos reconstruct ( #25383 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu…
28 -
b9913
Add Q2_0 quantization: type definition and CPU backend ( #24448 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan)…
30 -
b9912
spec : fix naming, spacing ( #25410 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu…
30 -
b9911
CUDA: Fuse MMVQ post-scale for NVFP4 ( #24481 ) CUDA: Fuse MMVQ for NVFP4 and BS 1 TODO: Add tests to test-backend-ops (did verify correctness manually for one model) Reorder bias/scale once PRs for NVFP4 are merged/landed Add dense MMVQ fusion as well Perf numbers on B4500.…
14 -
b9917: fix: OOB reads in UGM tokenizer (precompiled_charsmap handling) (#18750)
fix: OOB reads in UGM tokenizer (precompiled_charsmap handling) Validate minimum size (4 bytes) before reading xcda_blob_size Use strnlen with bounds check instead of unsafe strlen Both issues allow heap-buffer-overflow from malicious T5/UGM GGUF files. Replace unsafe strnlen()…
21 -
b9910
server : fix draft model fit vs load inconsistency ( #25056 ) fix: draft model fit vs load inconsistency refactor(server): unify draft/mtp parameter initialization, model, and context load moves speculative init to speculative.cpp changes server_context_impl model_dft and…
29 -
b9909
server : add timings and progress to /responses API stream ( #25348 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan)…
38 -
b9908
server: enforce prompt cache RAM limit ( #25070 ) Before this commit, --cache-ram was not a hard limit: The cache always kept at least one entry, even if that entry exceeded the RAM/token limits. Old entries were only evicted for the RAM/token limits after saving the new one,…
23 -
b9907
common : add missing include in common.h ( #25220 ) Signed-off-by: zhangrunda [email protected] macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU)…
18 -
b9906
ggml-hip : add -fno-finite-math-only alongside -ffast-math ( #25373 ) -ffast-math implies -ffinite-math-only under ROCm/clang 22, which disables INFINITY/NaN and triggers -Wnan-infinity-disabled (errors under -Werror in CI). Re-enable infinity handling without dropping the rest…
11 -
b9905
llama: fix quantized kv-cache for dsv4 ( #25202 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64…
35 -
b9904
[SYCL] fix unsupported UT cases of CONT & CPY ( #25231 ) fix unsupported UT cases of CONT & CPY update ops.md rm unused head file macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64…
10 -
b9902
[SYCL] support OP cross_entropy_loss, cross_entropy_loss_back ( #25236 ) support OP cross_entropy_loss, cross_entropy_loss_back correct format issue macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework…
12 -
b9901
sycl : set K_QUANTS_PER_ITERATION to 1 on DMMV path ( #25063 ) sycl: add supported types to ggml_sycl_supports_reorder_dmmv The reordered feature is implemented in ggml_sycl_op_dequantize_mul_mat_vec, but gated by ggml_sycl_supports_reorder_dmmv. This commit fixes the gate.…
11 -
b9899
sycl : enhance argsort to support all UT cases ( #25125 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu…
30 -
b9898
sycl : use sycl func to fix AOT double type issue ( #25081 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu…
19 -
b9897
sycl : rename the env vars from "disable" to "enable" ( #25042 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan)…
17 -
b9895
speculative : fix out-of-bounds read in ngram-map on prompt shrink ( #23936 ) speculative : fix out-of-bounds read in ngram-map on prompt shrink speculative : fix ngram-map cleanup cutoff after prompt shrink macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64,…
24 -
b9894
vulkan : check src0 type in GGML_OP_SET_ROWS to avoid failures due to unimplemented f16 support ( #25351 ) vulkan : check src0 type in GGML_OP_SET_ROWS to avoid failures due to unimplemented f16 support chore : get rid of else Co-authored-by: Stanisław Szymczyk…
32