News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow llama.cpp releases dev-tools 2d ago b10359 ggml-webgpu: fix CI errors from #25025 and #25262 ( #26566 ) test new flash_attn test rebase and fix to disable subgrou matrices when max_kv_tile == 0 delete log output Add i32 support to cpy and enables the all ops test restore the non target ci tests comment out of TODO of… 26 r/LocalLLaMA community 2d ago Tested in Coding: BF16 Muse Glimmer vs BF16 Qwen3.6 27B I'm guessing that many people have been waiting for this comparison. For clarity, both models are running at full FP16 KV-cache. Due to VRAM limitations, Muse Glimmer is running full 262,144 context, whilst Qwen3.6 27B can only run at 147,500 context - full GPU offload in both… 28 Stratechery (Ben Thompson) community 2d ago Nvidia’s Risky Business Nvidia is finding new ways for its customers to raise money, and it's expanding the risk of the AI buildout significantly. 33 r/LocalLLaMA community 3d ago Nvidia reportedly testing lower memory configs of Rubin Ultra as memory shortage bites back — designs tested include as little as 192 GB and step back to HBM4   submitted by   /u/fallingdowndizzyvr [link]   [comments] 16 llama.cpp releases dev-tools 3d ago b10362: tests : disable backend sampler hip multi output (#26878) test-backend-sampler: skip multi_output_sampling_chain on HIP The new multi_output_sampling_chain test uses top_k, whose backend probs path needs CUB (unavailable on HIP), so sampled_probs is null and the test aborts. Add it to the existing HIP skip list alongside the other… 29 arXiv — Machine Learning research 3d ago A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs arXiv:2608.08294v1 Announce Type: new Abstract: Knowledge distillation trains a smaller student to match the outputs of a larger teacher. Feature-based methods also align intermediate representations, but this extra constraint may affect students differently. We study this… 23 arXiv — Machine Learning research 3d ago Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure arXiv:2608.08722v1 Announce Type: new Abstract: Benchmarks for systems that are optimized against the evaluation signal measure something different from what they claim. We document this concretely in two GPU-kernel-optimization suites with held-out generalization gates:… 24 arXiv — Machine Learning research 3d ago Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference arXiv:2608.08730v1 Announce Type: new Abstract: Large Language Models are deployed to multiple types of environments, from internet browsers to edge devices, and WebGPU serves as a modern cross-platform standard. The engines for browser-based LLM inference have proliferated, yet… 19 arXiv — Machine Learning research 3d ago Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts arXiv:2608.08853v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) routers commonly use the same scores both to select experts and to weight their already-computed outputs. We study whether these two roles, dispatch and aggregation, should be coupled. On pretrained… 11 arXiv — NLP / Computation & Language research 3d ago APEX-VW: A Document-Level English-Spanish Post-Editing Dataset in the Healthcare Domain arXiv:2608.08059v1 Announce Type: new Abstract: Post-Editing (PE) of Machine Translation (MT) output often involves repeating the same lexical and terminological corrections across many segments, especially in specialised and highly repetitive documents. Despite substantial work… 25 arXiv — NLP / Computation & Language research 3d ago Failure-Aware Long-Form Translation: Design and Implementation of a Recoverable LLM Translation System arXiv:2608.09187v1 Announce Type: new Abstract: A long-form translation request can succeed at the API layer and still produce an unusable result. The output may be empty, truncated, filtered, dominated by source or prompt material, or interrupted after producing text worth… 32 arXiv — NLP / Computation & Language research 3d ago Pragmatic Attack Surface: Vulnerabilities of Implicit Context in Large Language Models arXiv:2608.09551v1 Announce Type: new Abstract: In the era of large language models (LLMs), attackers often manipulate natural language to elicit unsafe or harmful outputs, creating a new natural language attack surface unique to LLM-based systems, where attacks directly exploit… 26 llama.cpp releases dev-tools 3d ago b10355 llama : support multi-output backend sampling ( #25532 ) Enable backend sampling with token speculation Clamp the mask sum before converting it into the sampled index Add a numeric context parameter declaring the maximum outputs one sequence More fixes Don't reuse memory for… 8 llama.cpp releases dev-tools 3d ago b10353 ggml : require contiguous src for ROLL on CUDA and Metal ( #25928 ) ggml_roll only asserts nb[0] == ggml_type_size, so a permuted src is a valid input, but the CUDA and Metal roll kernels index by ne alone and never read the nb strides. A non-contiguous src therefore produced… 25 Hugging Face official-blog 3d ago Build Low-Latency Multilingual Voice Agents: Open Weights & Full Deployment Control with NVIDIA Magpie TTS Back to Articles a]:hidden"> Build Low-Latency Multilingual Voice Agents: Open Weights & Full Deployment Control with NVIDIA Magpie TTS Enterprise + Article Published August 10, 2026 Upvote 4 Maryam Motamedi maryameee nvidia Mikyas Desta mdestanv nvidia Jason Li blisc nvidia… 20 r/LocalLLaMA community 3d ago DeepSeek V4 Flash 0731 is the ‘killer app’ that is going to sell A LOT of DGX Sparks Having a ‘Killer Application’ that everyone wants to use helps sell hardware, plain and simple. DeepSeek V4 Flash 0731 isn’t an app of course, but I think it’s going to be the major catalyst for getting a lot of people to buy a couple of NVIDIA GB10-based systems because: It is… 33 Hacker News — AI on Front Page community 3d ago Humanising LLM Outputs Is Dumb Article URL: https://kuber.studio/blog/Reflections/Humanising-LLM-Outputs-is-Actually-Dumb Comments URL: https://news.ycombinator.com/item?id=49243474 Points: 200 # Comments: 131 5 NVIDIA Developer Blog official-blog 3d ago Run Local Agentic AI Workflows with Meta’s Muse Glimmer on NVIDIA Meta returns to the open source ecosystem with the release of Muse Glimmer, a 30B open-weight dense model with a 120K+ context window built for local AI... 37 Ollama releases dev-tools 3d ago v0.32.7 Muse Glimmer Note: Muse Glimmer is currently available via initial support via Ollama's MLX engine on Apple Silicon. Support for NVIDIA, AMD, and other platforms will be available in the coming days. Muse Glimmer , Meta's newest open model and the first released by Meta… 33 llama.cpp releases dev-tools 4d ago b10336 ggml-webgpu : refactor several wgsl files and simplify flash_attn wgsl. ( #26134 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64… 15 vLLM releases dev-tools 4d ago v0.27.0: [Kimi][MM] disable kimi_vit's dynamic torch.compile for TPU (#51196) Signed-off-by: Linkun Chen [email protected] (cherry picked from commit 7f58e82 ) 11 r/LocalLLaMA community 4d ago Running Qwen 3.5 35B A3B-Q8_0 gguf on a cheap radeon 7600 at 18 token/s I also have 64 gb ddr4 ryzen 5600 Using llama.cpp Ubuntu distro Settings are as follows --n-gpu-layers 999 \ --n-cpu-moe 37 \ --no-mmap \ -ctk q8_0 \ -ctv q8_0 \ -fa 1 \ -c 9000 \   submitted by   /u/Sweaty_Perception655 [link]   [comments] 30 arXiv — Machine Learning research 4d ago The Sparsity Whisperer arXiv:2608.06630v1 Announce Type: new Abstract: Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs. We argue that this overlooks a key computation performed by particularly… 13 arXiv — Machine Learning research 4d ago CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights arXiv:2608.06763v1 Announce Type: new Abstract: Weight quantization for large-language-model inference must balance adaptive reconstruction levels with representations regular enough for efficient GPU execution. Uniform integers constrain each group to a linear grid. Low-bit… 9 arXiv — Machine Learning research 4d ago Faster Query-Key Learning Sharpens Attention in Self-Attention Models arXiv:2608.06776v1 Announce Type: new Abstract: A standard self-attention layer consists of two interacting circuits: the query-key circuit that governs attention allocation, and the output-value circuit that maps attended representations to predictions. Collapsed and factorized… 8 arXiv — Machine Learning research 4d ago How Molecular Generative Models Organize Molecular Identity arXiv:2608.06956v1 Announce Type: new Abstract: Generative models for matter are often evaluated as samplers over output representations, and their latent spaces are commonly used as proxies for navigating chemical space. Much less is known about how these models internally… 6 arXiv — Machine Learning research 4d ago Stochastic Autoregressive Learning arXiv:2608.07224v1 Announce Type: new Abstract: Motivated by LLMs, which generate outputs by iteratively sampling from next-token distributions, we introduce a PAC-learning model for binary stochastic autoregressive learning. This generalizes the deterministic autoregressive… 13 arXiv — NLP / Computation & Language research 4d ago NTDH: Complex Reasoning for Comprehensive Affective Analysis arXiv:2608.06425v1 Announce Type: new Abstract: Comprehensive affective analysis is challenging for two reasons: it spans heterogeneous prediction tasks with continuous, ordinal, and multi-label outputs, and affective meaning is context-dependent, requiring conflicting cues to… 9 arXiv — Machine Learning research 4d ago Density-Functional Excited-State Gradients and Nonadiabatic Couplings on a Consumer GPU from a Contraction-DAG arXiv:2608.06536v1 Announce Type: cross Abstract: Nonadiabatic dynamics needs an excited-state gradient and an interstate nonadiabatic coupling matrix element (NACME) at every nuclear geometry, and a double-hybrid functional's accuracy has been unavailable for the coupling. We… 7 arXiv — NLP / Computation & Language research 4d ago Beyond "AI Language": The case for the idiolectal nature of LLM output arXiv:2608.06589v1 Announce Type: new Abstract: While large language model outputs are frequently analysed as a collective super variety termed "AI language," this chapter argues that this perspective coexists with distinct, model-specific linguistic signatures akin to human… 10 arXiv — NLP / Computation & Language research 4d ago CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity arXiv:2608.07460v1 Announce Type: new Abstract: While post-training improves the capabilities of large language models (LLMs), it generally lowers their output diversity and creativity, negatively impacting tasks that explicitly require creativity (e.g., story generation) as… 27 r/LocalLLaMA community 4d ago [2606.05682] Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments. Quantization-aware distillation (QAD) helps recover accuracy lost under low bit… 32 r/LocalLLaMA community 4d ago Pathway's BDH(post-transformer arch) matches GPT2 scaling from 10M to 1B params trained from scratch. runs on Normal GPUs   submitted by   /u/Candid-Tackle-9061 [link]   [comments] 36 r/LocalLLaMA community 4d ago Add a amd 9700 ai pro to a 3x5090 system vs buy a 5070ti for general useage. Hey guys I have this OCD im trying to decide about, I was lucky enough to buy 3 5090 before all the crazy ai stuff started and while that system works fine. Id also bought a razor core egpu that stopped working a little ago and so got sent off for repair, well its come back now… 35 r/LocalLLaMA community 4d ago Underestimated budget solution: radeon 780m iGPU There are so many posts where people complaining about high prices and asking for solution <= 1000 EUR. So, there is one solution to consider: PC/mini PC/laptop on Ryzen 7 260 / Ryzen 9 8945HX /etc CPU with 780m iGPU and 64 Gb of DDR5 RAM. Barebone mini PC costs around 300-400,… 6 r/LocalLLaMA community 4d ago AMD llama.cpp: reducing MTP buffer overhead gave me 64K → 149K context for Qwen 27B Available context length with and without the patch: Model: QWEN 27B ROCm stock patched Vulkan stock patched IQ4_XS Pure, single 16GB GPU 19.456 76.032 68,352 78,592 Q6_K_L on 16GB + 12GB 64,256 149,248 68,864 151,296 The issue is that llama.cpp overestimates the memory needed… 6 r/LocalLLaMA community 5d ago Extremely slow DSpark draft model performance (1-2 t/s) with DeepSeek-V4-Flash on llama-server compared to MTP? Hey everyone, I could use some advice on setting up speculative decoding correctly with llama-server . My Hardware: GPUs: RTX 4090 + RTX 6000 Pro (120GB total VRAM) RAM: 32GB I am currently testing the DeepSeek-V4-Flash-0731 unsloth model using the 137GB Q4 UD-IQ4-NL quant. When… 17 r/LocalLLaMA community 5d ago enabling PCI-E p2p for consumer Nvidia cards will yield you more than you think Disclaimer - no LLM was used to write this post/note As larger post about my setup will come later, want to give heads-up to folks who use VLLM and >= 2 GPUs. So I have pretty meaty server (8 channel AMD EPYC, ~150GB/s RAM bw) and 4x5060Ti 16GB in PCI-E 4.0 8x mode. Considering… 29 r/LocalLLaMA community 5d ago Building a budget 32GB → 48GB VRAM home AI server: 2-3x RX 9060 XT 16GB vs RTX 5060 Ti 16GB, AM5 vs used EPYC? I’m planning a dedicated home AI server, mainly for local LLM inference, agents/tool use, Docker services, and eventually larger MoE models with CPU offload. My plan is to start with 2x 16GB GPUs = 32GB VRAM , but I want to build the platform from day one knowing that I’ll… 37 llama.cpp releases dev-tools 5d ago b10330 CUDA: fuse rms_norm + mul + rope (+ view + set_rows) ( #26767 ) CUDA: fuse rms_norm + mul + rope (+ view + set_rows) tests: add broadcast weight case to rms_norm_mul_rope CUDA: check memory ranges before rms_norm rope fusion CUDA: check memory ranges in rope set_rows fusion… 25 r/LocalLLaMA community 5d ago Building a zero-dependency C inference engine for BitNet (1.58-bit) - lessons from hitting 36 tok/s on a Xeon CPU Over the past few months I have been building a CPU-first inference engine from scratch in pure C99 (no Python, no CUDA, no BLAS, just GCC and make). The focus has been running 1.58-bit ternary models natively without heavy runtime overhead. Currently it hits 36.25 tok/s on… 30 r/LocalLLaMA community 5d ago My first run of Kimi K3 locally. Running across 2 clusters using llama.cpp over RPC too. Both clusters are not enough to hold everything in memory, so main cluster still partially offloads to run. Goal will be to get all the GPUs in one system and without RPC, I should probably see 2-3x faster speed. Running… 7 r/LocalLLaMA community 5d ago Tesla V100 Qwen3.6 27B Performance Looking for V100 users to share your config and it's performance. GPU: Tesla V100 PCIE 32Gb Qwen3.6 27B Q4_K_M + Q8_0 MTP 128K context length Pi coding agent llama.cpp model preset: [*] spec-default = 1 ctx-size = 131072 mmap = 1 kv-unified = 1 n-gpu-layers = 999 threads = 18… 14 r/LocalLLaMA community 5d ago MI25 for 80-100€ worth it? seems to be about as good as a vega 56 with 16Gb of VRAM, is it worth it? (don’t want to deal with NVIDIA drivers on Linux, already have an rx6650xt and might simply use vulkan for llamacpp inference)   submitted by   /u/Kidplayer_666 [link]   [comments] 8 r/LocalLLaMA community 5d ago Has anyone here fiddled with TPUs for inference ? I discovered recently that Google uses their own TPUs, like tiny ASIC cards like the toy ones that existed for bitcoin. And while it sounds inefficient the fact they use thousands of them because...they can...means at scale they aren't so bad. Has no one here given them a try? I… 26 llama.cpp releases dev-tools 6d ago b10327 CUDA: fix thread/block count in quantized cpy kernel launches ( #26731 ) CUDA: fix thread/block count in quantized cpy kernel launches tests: add uneven block count cpy case Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI… 31 r/LocalLLaMA community 6d ago PSA for anyone with multiple V620's or other gfx1030 cards having problems making llama.cpp tensor split work -- set "-ub 384" and -b to a multiple of that depending on number of GPUs Basically what the title says. For me, it would always crash and burn trying to use tensor split. Apparently, there's some bug where GPU memory gets corrupted with the default microbatch (512) or higher. I will be opening an issue report on the llama.cpp GitHub if there isn't… 6 llama.cpp releases dev-tools 6d ago b10326 tts: account for the vocoder pass in the timings line ( #26733 ) get_output runs the waveform work the pipeline defers to it, from a single trailing window to a full pass depending on the model. Measuring it keeps the reported total and the audio to process ratio honest.… 6 r/LocalLLaMA community 6d ago Why no good external multi gpu solutions? I guess we are a small market, but I would love a turnkey external dual and quad GPU box with built in pcie5 full 16 lane support for all GPUs so you get full p2p. Then I could just plug the thing into a mini pc instead of contemplating a giant eatx xeon or threadripper build.… 11 r/LocalLLaMA community 6d ago DeepSeek v4 Flash 0731 on H100 node Hi, I've been playing with recent DeepSeek Flash updated model and it seems excellent. Like talking to Claude 4.8. I'm currently deploying it on VLLm and I'm in progres of tuning performance. I seem to hit the point where I trade decode with prefill by juggling batching knobs.… 21 Page 2 of 10 · 500 articles ← Newer Older →
