llama.cpp releases
500 articles archived · Visit source ↗ · RSS
-
llama.cpp releases dev-tools 1mo ago
b10019
sycl : support kernel type fp16 for conv2d_dw ( #25653 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64…
21 -
b10018
sycl : implement xielu op ( #25550 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64…
34 -
b10017
sycl: Increase minimum buffer size for USM system allocations ( #25525 ) Raise the threshold for minimum buffer size from 1 GiB to 4 GiB, based on real-world experiments of overcommitting device memory with model weights larger than available VRAM, for example Qwen3.5-35B-A3B-Q8…
28 -
b10016
[SYCL] Flash Attention with XMX engine via oneDNN ( #25222 ) [SYCL] F16 (default) Flash Attention with XMX engine via oneDNN graph API; Qwen3.6-27b-Q8_0 prefill speed up x1.21 at p=512 and x4.26 at p=80k [SYCL] Address review on FA oneDNN path. Result: llama-bench---pp512; 32%…
8 -
b10015
opencl: do not use clCreateBufferWithProperties when targeting CL 2.x ( #25673 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu…
6 -
b10012
hexagon: fix hmx-queue signal enum-narrowing problem ( #25677 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan)…
15 -
b10011
server : refactor prompt cache state ownership ( #25649 ) server : clear checkpoints upon prompt clear server : move the prompt state data to the server_prompt_cache Assisted-by: pi:llama.cpp/Qwen3.6-27B server : handle batched slot being cleared macOS/iOS: macOS Apple Silicon…
38 -
b10010
server: add --cors-* options ( #25655 ) server: add --cors-* options add special "localhost" value add tests fix test add link to PR macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64…
33 -
b10007
opencl: fix a dp4a bug for devices where cl_khr_integer_dot_product is unavailable ( #25639 ) opencl: do not fail backend init on devices without cl_khr_integer_dot_product opencl: do not call dp4 kernels when dp is unavailable Co-authored-by: Li He [email protected]…
10 -
b10005
DeepseekV4: fix seq_rm ( #25588 ) DeepseekV4: fix seq_rm implement proper seq_cp create actual update context macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64…
25 -
b10004
vulkan/cpu: Support f16 as SET_ROWS src. ( #25432 ) vulkan/cpu: Support f16 as SET_ROWS src. This adds full support for f16 SET_ROWS (equivalent to f32) to vulkan and CPU backends, and adds more backend tests. Set DenormPreserve 16 when supported, to try to fix failures on Intel…
6 -
b10003
tokenize : align usage by using common args ( #25516 ) Migrate the tokenize tool to common_params_parse, replacing its hand-rolled argv parsing, Windows UTF-8 handling and file reading with the shared common helpers. Expose the model-sourcing flags (-m, -mu, -dr, -hf, -hff,…
16 -
b10002
ggml : add a set of functions for checking contiguity of inner tensor dimensions ( #25650 ) Co-authored-by: Stanisław Szymczyk [email protected] macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework…
27 -
b10001
tests: export-graph-ops: exit gracefully when called w/o arguments ( #25619 ) Fixes a segfault when test-export-graph-ops is called without any arguments. macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS…
10 -
b10000
ggml: uniformize im2col dst_type for all conv ops ( #23660 ) ggml: uniformize im2col dst_type for all conv ops Update ggml/src/ggml.c Co-authored-by: Georgi Gerganov [email protected] ggml : uniformize im2col casting logic across all conv ops fix : allow im2col_f16 to accept…
4 -
b9999
kleidiai : add SME2 f32 kernel ( #24414 ) kleidiai : add SME2 f32 kernel enable dynamic scheduling for SME2 f32 kernel macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu…
30 -
b9996
arg: Flush log before exiting after usage() ( #25504 ) Under certain conditions, it's possible for messages emitted via LOG() to get lost before exit, apparently because they are emitted by another thread. common_params_print_usage() uses printf directly, and is not affected.…
36 -
b9995
sycl: set fattn_vec_nthreads to 256 for Battlemage ( #25205 ) Currently detects lunarlake + battlemage / xe2 and sets the value to 256. Keeps default at 128, Intel's ARC Alchemist's prefered value. macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI…
15 -
b9994
metal : add Q2_0 support ( #25419 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64…
29 -
b9993
model: add Hy3 (hy_v3) support with MTP speculative decoding ( #25395 ) model: add Hy3 (hy_v3) architecture support Adds Tencent Hunyuan 3 (HF architecture HYV3ForCausalLM, GGUF arch hy_v3): a MoE decoder stack with per-head Q/K RMSNorm, a sigmoid router with expert selection…
29 -
b9992
CUDA: refactor MMQ kernel configuration ( #24127 ) CUDA: refactor MMQ kernel configuration fix Blackwell config remove legacy code macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64…
4 -
b9990
spec: add Minimax2 eagle3 support Fix nullptr in minimax2 EAGLE3 minor : add newline Co-authored-by: Georgi Gerganov [email protected] macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu…
27 -
b9988
tests: Harmonize header use ( #25616 ) tests: Harmonize the use of private ggml includes tests: In test-backend-ops, use quoted includes As with all other tests. This is to ensure that the build uses shipped headers over possibly system-installed ones. macOS/iOS: macOS Apple…
17 -
b9987
gguf : add tensor shape accessor ( #24405 ) gguf : add tensor shape accessors gguf : return tensor shape as const int64_t * gguf : remove n_dims accessor, keep only gguf_get_tensor_ne macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED…
33 -
b9986
chat : fix reasoning leak with force-opened bare templates ( #24674 ) chat : fix reasoning leak with force-opened bare templates The reasoning start tag inferred from prior turns can carry trailing whitespace (e.g. \n) while a force-open template prefills a bare . Trim the tag…
9 -
b9985
sycl: add fused top-k MoE ( #25217 ) sycl: add fused top-k MoE sycl: address review: GGML_SYCL_ENABLE_FUSION env, move fusion dispatch to topk-moe sycl: print GGML_SYCL_ENABLE_FUSION at startup like other env vars Co-Authored-By: Claude Fable 5 [email protected]…
8 -
b9984
sycl: add Q2_K to DMMV reorder path ( #25064 ) Signed-off-by: Todd Malsbary [email protected] macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu…
30 -
b9982
server: honour per-request reasoning_budget_tokens in chat completions ( #23116 ) server: honour per-request reasoning_budget_tokens in chat completions The reasoning-budget block in oaicompat_chat_params_parse read only the server-level default (opt.reasoning_budget, typically…
4 -
b9981
vendor : update cpp-httplib to 0.50.1 ( #25576 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64…
13 -
b9980
server: Don't consider models with --no-mmproj-auto as multimodal ( #25590 ) If mmproj is explicitly disabled via the model preset or command-line parameters then the model won't be able to handle image/audio inputs and this shouldn't be declared as supported input modality on…
25 -
b9979
mtmd: fix silent prompt truncation on embedded NUL ( #25548 ) mtmd: fix silent prompt truncation on embedded NUL mtmd_input_text carried the prompt as a bare const char* with no length, so a NUL byte in message content cut the prompt at the tokenizer boundary and dropped every…
32 -
b9978
server : evict checkpoints within min-step of each other ( #25472 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan)…
33 -
b9977
server : fix image blocks in tool_result being dropped during Anthropic OpenAI conversion ( #22536 ) server : fix image blocks in tool_result being dropped during Anthropic→OpenAI conversion server_chat_convert_anthropic_to_oai() silently discarded image blocks inside Anthropic…
28 -
b9975
gguf : reject empty metadata keys ( #24917 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan)…
21 -
b9974
cuda: Don't crash when querying memory on device with no free memory. ( #25157 ) If a Cuda device has no or limited available memory, the actual call to cudaMemGetInfo() itself can cause a fatal crash due to a cuda out of memory error (there is not enough memory to actually…
38 -
b9973
DeepseekV4: clear cache only for seq rather than full ( #25521 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan)…
15 -
b9972
server: allow stream for exec_shell_command ( #25526 ) init stream add stream for shell tool add test nits update docs macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu…
27 -
b9971
server: refactor server_stream ( #25541 ) server: refactoring, remove spipe from server_http_res wip remove non-thread-safe rd.stop() call move server_res_spipe nits improve server_stream_create_spipe server-stream: update dev docs for the improved API Co-authored-by: Pascal…
28 -
b9970
ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer ( #24231 ) ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer ggml : remove scale parameters from lightning indexer OP, add f16 mask parameter tests : add…
14 -
b9969
Vulkan: route large matmuls to medium tile on Adreno ( #24877 ) [Vulkan] Fixes llama-cli breaking over longer promts sizes The llama-cli was breaking for longer promts sizes for q4_0 quantized networks. Causing due to insufficient shared memory. Removed the un-used Adreno device…
10 -
b9968
opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GPUs ( #25537 ) opencl: add int8 dp4 dense and moe GEMM opencl: refactor Co-authored-by: Li He [email protected] macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED…
30 -
b9966
llama : make tensor-split regex patterns static ( #24710 ) llama_meta_device_get_split_state() recompiled 29 std::regex on every call. In -sm tensor mode the callback runs once per tensor per token, so this dominated the decode thread in profiling. Mark them static const so they…
10 -
b9965
hexagon: improve ARGSORT performance for small tensors ( #25512 ) hex-sort: add efficient bitomic sort in hvx regs up to 1024 elements hex-sort: fix inverted vrors hex-sort: specialize sort functions for the common cases hex-sort: add tracing and local context macOS/iOS: macOS…
33 -
b9976
Fix conditional to display 'LLAMA_SPLIT_MODE_TENSOR not implemented f…
37 -
b9964
arg: prevent duplicate spec model downloads ( #25527 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64…
35 -
b9963
mtmd: deepseek-ocr v1 multi-tile ( #24717 ) mtmd: deepseek-ocr v1 multi-tile dynamic resolution + unified image-preprocessors for both versions (ds-ocr v1 and v2) remove hacky API fuse row into a long image almost working adapt to new preprocessor api rm debugging printf improve…
17 -
b9960
server: remove loading.html ( #25500 ) server: remove loading.html apply ui changes macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU)…
4 -
b9959
sync : ggml macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64 (ROCm 7.2) Ubuntu x64…
26 -
b9957
server: improve tools, remove apply_diff ( #25498 ) server: improve tools, remove apply_diff improve edit tool add tools_io abstraction add tools_io_basic fix build move utils to class member add const macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI…
9 -
b9967: server: accept null sampling params (#25538)
server: accept null sampling params Extend the schema validation to treat a null value as absent, so clients can send null on nullable params (temperature, top_p, ...) to request the server default. This matches the OpenAI spec and the json_value convention used elsewhere. Add…
17