News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow r/LocalLLaMA community 27d ago SigLIP 2 text embedding on CPU with Rust + ONNX We’re building a robotics data platform with a lot of images, video, and text metadata. For search, we use SigLIP 2. GPUs handle batched asynchronous image/video embedding and indexing, while this small Rust + ONNX Runtime service handles live text queries on CPU. Both land in… 37 llama.cpp releases dev-tools 27d ago b10066 opencl: load and use kernel_gemm_moe_q6_k_f32_ns from bin kernel lib ( #25797 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU)… 19 r/LocalLLaMA community 27d ago Prism ternary 27b, how is it? Is it actually comparable to the qwen3.6 27b? Can it be used for GPU poor? Or do I stick to the 35b moe?   submitted by   /u/Adventurous-Gold6413 [link]   [comments] 25 r/LocalLLaMA community 27d ago [RESEARCH] Breaking the 1-bit Floor: Achieving "Negative-Bit Quantization" (NBQ) via Phase-Inverted Tensor Embedding (satire) Hey everyone, I’ve spent the last three weeks compiling custom llama.cpp forks and running imatrix maps on a modified CUDA kernel setup, and the numbers don’t lie. We’ve been looking at model compression completely wrong. Everyone in the community has assumed that 1-bit… 37 r/LocalLLaMA community 27d ago DeepSeek V4 Flash | IQ3_XXS-AS & IQ2_S Bench | mainline b10064 vs fairydreaming | 1xRTX 3090 + 128GB DDR4 | 250PP/11TG on 50K CTX Hey all! Wanted to see how DeepSeek V4 Flash GGUFs in two different quants perform on my hardware and share the results. Tested two quants on the fairydreaming/llama.cpp dsv4 fork . As a bonus, I also ran the same model (IQ3_XXS-AS) on mainline llama.cpp b10064 just to check.… 22 Hugging Face official-blog 27d ago Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers Back to Articles a]:hidden"> Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers Enterprise + Article Published July 17, 2026 Upvote 1 Pranav Prashant Thombre pthombre nvidia linnan wang linnanwang nvidia Alexandros Koumparoulis akoumpa nvidia… 32 llama.cpp releases dev-tools 27d ago b10064 opencl: transpose q4_K noshuffle scales for coalesced reads ( #25805 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu… 4 llama.cpp releases dev-tools 28d ago b10063 sync : ggml Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu… 37 llama.cpp releases dev-tools 28d ago b10061 tests : initialize all tensors in test_dsv4_hc to avoid NaNs in sentinel tensors ( #25822 ) Co-authored-by: Stanisław Szymczyk [email protected] Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel… 26 llama.cpp releases dev-tools 28d ago b10059 ggml-blas: default hadamard mul_mat to cpu routine ( #25710 ) Signed-off-by: Aaron Teo [email protected] Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64… 8 TechCrunch — AI news-outlet 28d ago Why the first GPU financiers are turning to inference chips in a $400 million deal A $400 million chip-backed loan points to the next wave of AI infrastructure deals. 27 llama.cpp releases dev-tools 28d ago b10058 vulkan: Support Q2_0 ( #25430 ) vulkan: Support Q2_0 The backend perf tests for mat-vec-mul weren't very good at first (worse than q2_k), doubling the rows per workgroup made a big difference. reorder resolve merge conflict, adjust err threshold for f16->q2_0 set_rows Website:… 37 llama.cpp releases dev-tools 28d ago b10057 sycl: fix row calculation when K_QUANTS_PER_ITERATION is 1 ( #25690 ) sycl: fix incorrect row calculation when K_QUANTS_PER_ITERATION=1 Signed-off-by: Todd Malsbary [email protected] sycl: use K_QUANTS_PER_ITERATION for non-reordered Q5_K kernel This is the only Q5_K… 24 Hugging Face Daily Papers research 28d ago LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget Abstract A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization at deployment. The gap is especially… 16 Hugging Face Daily Papers research 28d ago Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models Abstract In-context learning is commonly interpreted as a form of conditional inference, in which the prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates… 31 llama.cpp releases dev-tools 28d ago b10056 opencl: add ABS op ( #25115 ) 31 r/LocalLLaMA community 28d ago GPU Power Efficiency Tips and Tricks Ran a quick test sweep across the frequency range of the P100s using Darwin-36B (Qwen 3.6 35B-A3B sort of). Not a comprehensive test. Feel free to share your power efficiency tips, maybe we make a bigger thread as we explore ways to improve our efficiency, not just raw tps. Try… 23 arXiv — NLP / Computation & Language research 28d ago Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation arXiv:2607.14895v1 Announce Type: cross Abstract: Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding. These domains permit reliable verification of model outputs, which is important for enabling the reinforcement… 35 arXiv — Machine Learning research 28d ago LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget arXiv:2607.14952v1 Announce Type: new Abstract: A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization… 14 arXiv — NLP / Computation & Language research 28d ago D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding arXiv:2607.14647v1 Announce Type: new Abstract: Speculative decoding accelerates large language model (LLM) inference without compromising output quality. Recent parallel drafting methods further improve single-request performance by decoupling draft length from drafting… 32 arXiv — NLP / Computation & Language research 28d ago Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models arXiv:2607.15277v1 Announce Type: new Abstract: In-context learning is commonly interpreted as a form of conditional inference, in which the prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this… 32 arXiv — NLP / Computation & Language research 28d ago MemoHarness: Agent Harnesses That Learn from Experience arXiv:2607.14159v1 Announce Type: cross Abstract: An agent harness is the external control layer that turns a base LLM into an executable agent by managing context, tools, orchestration, memory, decoding, and output handling. While harness design strongly affects agent behavior,… 14 arXiv — NLP / Computation & Language research 28d ago Scaling Evaluation-time Compute with Reasoning Models as Evaluators arXiv:2503.19877v3 Announce Type: replace Abstract: As language model (LM) outputs get more and more natural, it is becoming more difficult than ever to evaluate their quality. Simultaneously, increasing LMs' "thinking" time through scaling test-time compute has proven an… 8 llama.cpp releases dev-tools 28d ago b10054 docs: added a note about using OpenCl with Adreno 810 ( #25786 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x… 7 Vercel — AI dev-tools 28d ago Optimized CDN caching and deploying of immutable static assets Vercel now reuses static files across deployments for frameworks that output content-addressed assets. It's zero-config: Vercel uses Framework-defined infrastructure to manage these immutable files alongside your code changes, handling the hard parts like hash collisions, file… 7 r/LocalLLaMA community 28d ago What do I need to get Kimi 3 locally reliably, full context, 20 concurrent req? GPU and mainboard/cpu suggestions? Must be a single machine, not linked. Willing to run it Nvfp4 Edit: Oh deng. People are so salty when asking questions here. I will be talking to vendors but need some base info to activate the BS meter (yes, we’ve been burned before). Thanks… 18 llama.cpp releases dev-tools 28d ago b10052 hexagon: L2 cache handling rework (dirty bit tracking with lazy flushing) and more MUL_MAT updates ( #25762 ) hex-mm: fix artificial limit in the solver that restricted number of act-prep threads hex-mm: fix warning hex-prof: do not apply --top to the timeline report hmx-mm: add… 18 llama.cpp releases dev-tools 28d ago b10051 kleidiai: Add SME vs SME2 distinction in kernel dispatch ( #25478 ) The current integration treats SME as a single capability (CPU_FEATURE_SME) with no distinction between SME(v1) and SME2. The kernels dispatched under CPU_FEATURE_SME use SME2-specific instructions, making… 10 llama.cpp releases dev-tools 28d ago b10050 vulkan: when using transfer queue for async copies, sync on event_wait to avoid race ( #25229 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)… 27 r/LocalLLaMA community 28d ago KIMI K3 Beats Claude Fable and GPT 5.6 sol in arena.ai!!! Unbelievable to see kimi k3 beat frontier models that were 'too dangerous' for public use.   submitted by   /u/Gohab2001 [link]   [comments] 15 llama.cpp releases dev-tools 28d ago b10048 TP: fix Phi3, Bert, Plamo2/3, ChatGLM ( #25536 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64… 14 r/LocalLLaMA community 28d ago tried predicting which MoE experts get used next token to speed up cpu/gpu offload, got some real numbers, is this actually implementable or am i wasting my time (30tg/s -> 150-200tg/s) so ive been messing around with qwen3.6 35b a3b (MXFP4 gguf) on my 3060 12gb, doing the usual cpu/gpu offload thing where half the expert layers sit in ram and get pulled over pcie whenever needed. and like everyone whos done this knows the gpu just sits there idle waiting for… 28 llama.cpp releases dev-tools 28d ago b10047 vendor: update BoringSSL to 0.20260713.0 ( #25624 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64… 24 r/LocalLLaMA community 28d ago DFlash makes Qwen3.6 27B 2.2x faster with no quality loss We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks were: quicksort, write a Steam library in JSON, solve a logic puzzle and write a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98… 22 llama.cpp releases dev-tools 28d ago b10046 tests: actually exercise test-recurrent-state-rollback ( #25758 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x… 32 llama.cpp releases dev-tools 28d ago b10045 server : allow text-only slot save/restore with mtmd ( #25076 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x… 25 Hugging Face official-blog 28d ago NVIDIA Nemotron 3 Embed Ranks #1 Overall on RTEB, Advancing Agentic Retrieval Back to Articles a]:hidden"> NVIDIA Nemotron 3 Embed Ranks #1 Overall on RTEB, Advancing Agentic Retrieval Enterprise + Article Published July 16, 2026 Upvote 12 Yauhen Babakhin ybabakhin nvidia Ronay Ak ronay-nv nvidia Jiarui Cai jiaruic nvidia Vinay Raman viraman nvidia Radek… 24 NVIDIA Developer Blog official-blog 28d ago Scaling Agentic AI Factories Through Extreme Co-Design with NVIDIA BlueField Agentic AI changes the infrastructure pattern for AI factories. One request can trigger many model calls, tool calls, memory lookups, policy checks, storage... 16 r/LocalLLaMA community 29d ago Q2 DeepSeek V4 Flash on 2x 3080 20GB, 64GB DDR5 | 17 tk/s gen, 270 tk/s prefill Hey, it's my first time posting here and I thought I'd share my progress on getting Antirez's imatrix Q2 DeepSeek V4 Flash GGUF (86.7 GB) running on my build. I used this llama.cpp fork which fixed the model's output when KV cache is quantised to Q8. Specs: - Ryzen 7 7800X3D -… 28 llama.cpp releases dev-tools 29d ago b10043 CUDA: Support CUDA Virtual Devices ( #25228 ) support cuda virtual devices disable NCCL path when virtual devices are used label virtual devices in description; add GPUx2 server CI jobs code refactor Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple… 6 r/LocalLLaMA community 29d ago I just got my first GPU that can actually run an LLM (Laptop 5090 24 GB) what do I play with first? I figure I can run 30b or even 72b models on it, but this is my first time running my own local LLM. I want to play with prompt engineering and see the most complex things I can get it to do. Any hot tips? I know it all moves fast and this sub is wired in I have 32GB ram also… 24 r/LocalLLaMA community 29d ago NVIDIA H200 Disassembly & Liquid-Cooling Installation with EK-Pro H200 NVL Water Block This is a full teardown video of the NVIDIA H200 NVL and installation of an EK-Pro H200 NVL Water Block, covering the disassembly, prep and mount process, including: PCB separation from the stock cooler using Torx T6, T9, and T10 drivers plus a plastic spudger Complete removal… 38 llama.cpp releases dev-tools 29d ago b10042 Enable CUDA graphs on volta+turing ( #25749 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64… 35 llama.cpp releases dev-tools 29d ago b10041 server: Ignore empty / non-existing Origin headers ( #25756 ) Otherwise this gives lots of unnecessary warnings: W srv operator(): (CORS) skip non-localhost origin: Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled)… 8 llama.cpp releases dev-tools 29d ago b10040 ggml-cuda : restore prop.integrated on HIP builds ( #24233 ) PR #16308 set info.devices[id].integrated = false unconditionally for all CUDA/HIP devices as a workaround for corrupted output on Jetson Orin ( #15034 ). On HIP/ROCm the device's real hipDeviceProp_t.integrated flag… 37 llama.cpp releases dev-tools 29d ago b10038 ci : add official website link to release notes ( #25728 ) Assisted-by: pi:llama.cpp/Qwen3.6-27B Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU)… 4 llama.cpp releases dev-tools 29d ago b10037 quant : allow using manual tensor types with --pure ( #25716 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu… 32 r/LocalLLaMA community 29d ago NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B on 2x3090s I managed to get this model working on 2x 3090s with full 262k ctx and N=4, if anyone is interested to try it, thanks to this quant: https://huggingface.co/danielrmay/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-W4A16 Nemotron-Labs-3-Puzzle-75B-A9B (W4A16) on 2× RTX 3090 — vLLM, no CPU… 6 llama.cpp releases dev-tools 29d ago b10036 opencl: disable FA and MoE weights repack to work around compiler issues for Adreno 850 GPU ( #25745 ) opencl: workaround for A850 compiler compat opencl: fix DX compiler version parsing and cleanup Co-authored-by: Li He [email protected] macOS/iOS: macOS Apple Silicon… 7 llama.cpp releases dev-tools 29d ago b10035 cuda: extract Q1_0 elements via __byte_perm ( #25628 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64… 18 Page 9 of 10 · 500 articles ← Newer Older →
