r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 6d ago

model: support Longcat-Flash (need testing) by ngxson · Pull Request #19182 · ggml-org/llama.cpp

This PR should be ready for testing now . I tested with a very small (8B params) sub-model extracted from the original one. Appreciate if someone can test with the bigger model. GGUF(for testing) from PR : (Please check latest comments at bottom for updated GGUFs )…

33
r/LocalLLaMA community 6d ago

DeepSeek V4 Flash 0731 appreciation post

I’m running DSV4F 0731 on dual spark, and honestly… wow. It’s an absolute workhorse, and the benchmarks are real. Everyday tasks with Hermes agent? Effortless. Coding tasks with OpenCode? I’m genuinely amazed at what it can handle. I can throw a two-hour coding session at it,…

12
r/LocalLLaMA community 6d ago

Anyone else amped up over Qwen 3.8?

I’ve been using 3.6 27B Q4, and that quant is fast on an M5. The code has been average, but consistently “good enough.” And, after a year, I can see home LLMs being served at home much like streaming music was introduced. A simple browser extension and all your queries go…

14
r/LocalLLaMA community 6d ago

Qwen 35B-A3B MoE vs 27B dense in local coding tests: ~4× faster, much smaller quality gap than I expected

I compared Qwen 35B-A3B MoE against Qwen 27B dense on a series of local coding-maintenance tasks. On my R9700/llama.cpp setup, the MoE model generated about 3.9× faster (~116 vs ~30 tok/s) , but the coding-quality difference was much smaller than I expected. Both usually handled…

8
r/LocalLLaMA community 6d ago

4x2080Ti 22GB tensor parallel possible?

Hi there, I have a setup with 4x 2080Ti 22GB, but am not able to see much benefits in running models larger than 16-20GB in speed. I did some research and saw something like tensor parallels are possible, but also that because of flash attention only being possible with 30…

28
r/LocalLLaMA community 6d ago

PSA for anyone with multiple V620's or other gfx1030 cards having problems making llama.cpp tensor split work -- set "-ub 384" and -b to a multiple of that depending on number of GPUs

Basically what the title says. For me, it would always crash and burn trying to use tensor split. Apparently, there's some bug where GPU memory gets corrupted with the default microbatch (512) or higher. I will be opening an issue report on the llama.cpp GitHub if there isn't…

6
r/LocalLLaMA community 6d ago

I got tired of my 300GB model loads taking 5min on RPC. PR 26291 speeds it 300% to 1min30sec (4060ti+ddr4) + (4060ti+ddr5)

- On b10173 - "state":"loading" 4min54sec. - With this PR and GGML_RPC_LOAD_THREADS 12 - "state":"loading" 1min38sec The PR is close to ready, will need a docs change if they want to keep the new GGML_RPC_LOAD_THREADS variable.. and hopefully they take it with the client-side…

10
r/LocalLLaMA community 6d ago

BeeLLama issues

Tried to use Beellama , and using the kvarn6 flag, i notice that its in llama-server --help but its not working. I must be doing something wrong. trying to run the following: llama-server.exe ^ --model "C:\llamaROCM\models\Qwen3.6-35B-A3B-IQ4_XS-3.53bpw.gguf" ^ --spec-type…

11
r/LocalLLaMA community 6d ago

Is anyone else finding DeepSeek-V4-Flash unreliable for non-coding tasks?

(I am not a native speaker, written by myself, so please bear with me) I really want to like DeepSeek-V4-Flash-0731. But it has serious flaws that don't align with the high score on intelligence benchmarks. And those flaws render it useless unfortunetely for anything else…

21
r/LocalLLaMA community 6d ago

U.S. Department of Energy Launches the Genesis Open Models Initiative and, with Arcee, Unveils Genesis-Science-1 — Its First Open-Weight Model for Scientific Research

  submitted by   /u/johnnyApplePRNG [link]   [comments]

18
r/LocalLLaMA community 6d ago

Serving Deepseek v4 Flash 0731 on 2x DGX Spark — 5-7 GB OS headroom, what would you do to lower VRAM usage and increase OS available RAM?

Hey all, I'm serving DSv4Flash 0731 on a cluster of 2x DGX Sparks but am running into constant issues with having almost no RAM (unified memory) left for the OS/cache and I'd love to hear the community feedback on what I could do to get more RAM for headroom. The DGX has an…

32
r/LocalLLaMA community 6d ago

Qwen 3.6 27B flags/settings in llama.cpp

I run the following on a 5090 and have been okay with its performance, it does most things somewhere 80-100 t/s, though that can slow down at full 262k context - more like 40 t/s at times. I use it primarily in appdev tasks. This just barely fits in the 5090, no vision, with…

31
r/LocalLLaMA community 6d ago

Why no good external multi gpu solutions?

I guess we are a small market, but I would love a turnkey external dual and quad GPU box with built in pcie5 full 16 lane support for all GPUs so you get full p2p. Then I could just plug the thing into a mini pc instead of contemplating a giant eatx xeon or threadripper build.…

11
r/LocalLLaMA community 6d ago

Which model would be best for Delphi development?

Hey all, Are there any Delphi developers in the crowd? If so, which models would you say are best at doing development in Delphi? What are your thoughts/suggestions here, and is there a good GUI client/harnass you like for doing delphi specific things? Thanks   submitted by…

22
r/LocalLLaMA community 6d ago

DeepSeek V4 Flash 0731 - ARC-AGI Results

  submitted by   /u/johnnyApplePRNG [link]   [comments]

20
r/LocalLLaMA community 6d ago

How to even compare quants from various sources?

How do you guys deal with so many variables? So many publishers, each calling their quants "best", and then it's a mess to manage, download the weights, tweak the temperature etc. for each source? It is relatively simple if I'm comparing different quantization levels (like Q4 vs…

7
r/LocalLLaMA community 6d ago

DeepSeek v4 Flash 0731 on H100 node

Hi, I've been playing with recent DeepSeek Flash updated model and it seems excellent. Like talking to Claude 4.8. I'm currently deploying it on VLLm and I'm in progres of tuning performance. I seem to hit the point where I trade decode with prefill by juggling batching knobs.…

21
r/LocalLLaMA community 6d ago

Got job as Director of AI and Systems development self-taught

Hey everyone, I just wanted to share my journey here for some motivation. Three years ago, I saw the sudden spike in AI and realized it was the future of tech. My goal at the time was to be an indie game dev, and seeing that AI could write basic code, I told myself I needed to…

20
r/LocalLLaMA community 6d ago

parakeet.wgsl – Fast, accurate ASR in the browser, via raw WebGPU & SIMD WASM

High-performance inference of NVIDIA's Parakeet TDT 0.6B V2 English transcription model, in the browser. Check out the live demo: https://parakeet.narcotic.sh/ A fully custom, dependancy-free implementation with raw WebGPU compute shaders and SIMD WebAssembly audio frontend. 1…

19
r/LocalLLaMA community 6d ago

llama.cpp PR reports up to 169% faster quantized-KV decode at 118K context on Intel Battlemage from one SYCL kernel switch

A fresh llama.cpp PR (#26689) changes what looks like a tiny SYCL FlashAttention dispatch decision. With a quantized KV cache ("q4_0" / "q8_0"), decode was being sent through the VEC kernel. On the author's Battlemage test system, switching that path to TILE gets much faster as…

31
r/LocalLLaMA community 6d ago

A visualization of LLM API costs to ask for local resources

I have not been successful with management to get funding for local resources despite bringing forth solid arguments about data sovereignty and related architectures. What actually succeeded in getting a budget is showing them how you cannot forecast or negotiate or otherwise…

37
r/LocalLLaMA community 6d ago

Am I just hallucinating

Or is there any reason why I feel like model output quality seems to be better when I use higher micro-batch values (ub) in llama-cpp? I don't really have any hard numbers or anything (just running the same prompts), it's all just vibes. Some context, I'm running the latest…

27
r/LocalLLaMA community 6d ago

Wan-Animate-2: Pushing the Application Boundaries of Character Animation Models

📝 Introduction We present Wan-Animate-2, a novel end-to-end character animation framework that directly consumes driving videos in a redesigned Diffusion Transformer, which achieves high-fidelity motion generation and strong identity preservation by eliminating intermediate…

16
r/LocalLLaMA community 6d ago

LFM2.5-2.6B model+KV cache quantization report

LFM2.5-2.6B is a new tiny model by LiquidAI, with benchmarks that put it head to head with much larger models. I've run llama-perplexity on many model GGUF quants, crossed with many KV cache quants, to understand the model's best overall quantization for any given amount of…

32
r/LocalLLaMA community 7d ago

A llama.cpp PR makes Q2_0 3.0–3.6x faster on x86 CPUs, 8B decode goes 2.39 → 8.20 tok/s

I was going through the current llama.cpp CPU PRs and #26348 stood out because this isn't the usual +5% kernel optimization. It adds an x86 VNNI implementation for the Q2_0 × Q8_0 dot product, and the author's controlled CPU-only benchmarks show roughly 3–3.6x higher throughput…

24
r/LocalLLaMA community 7d ago

LabyrinthBench: a local-focused, judge-free LLM benchmark that measures context recall under interference for multi-step agentic tasks.

LabyrinthBench measures the thing that actually kills long agent runs — whether a model can still use what it learned twenty turns ago — deterministically, with no LLM judge, on your own hardware, with a swappable harness for testing whatever context-management strategy you…

36
r/LocalLLaMA community 7d ago

RTX 5090 Owner Built An Open-Source Tool That Shuts Down PC If It Detects The 12VHPWR Cable Drawing Too Much Power, But It Can Only Work On Specific GPUs

GitHub : https://github.com/humza-khalid/12vhpwr-guard Reddit thread : https://www.reddit.com/r/nvidia/comments/1vglua1/i_built_a_free_open_source_tool_that_shuts_your/   submitted by   /u/pmttyji [link]   [comments]

9
r/LocalLLaMA community 7d ago

Anyone running DeepSeek-V4-Flash-0731 on MI325X with vLLM? Mine is behaving completely broken

Is anyone here successfully running DeepSeek-V4-Flash-0731 locally with vLLM , especially on AMD MI325X? My setup: GPU: 1x AMD Instinct MI325X Model: deepseek-ai/DeepSeek-V4-Flash-0731 vLLM: 0.26.0 ROCm image --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4…

20
r/LocalLLaMA community 7d ago

An open-weight model too, Moonshot joins the race (gently this time)

From Sauers 𝕏: https://x.com/Sauers_/status/2085585414954312113 Wired: One of China’s Most Powerful AI Models Has Also Escaped Containment: https://www.wired.com/story/moonshot-kimi-k3-ai-model-escape-sandbox/   submitted by   /u/Nunki08 [link]   [comments]

17
r/LocalLLaMA community 7d ago

Gemma 4 QAT could be improved further by Google aligning the QAT model to modern q4_k instead of q4_0

Hello, For the past few days I have been benchmarking Gemma 4 26b QAT UD Q4_K_XL extensively versus Bartowski's Q4_K_L. While QAT is certainly very effective and reducing memory consumption versus the highest q4 quant from him, I also have noticed some regressions in my own…

4
r/LocalLLaMA community 7d ago

Black Hat USA 2026: The 'Breaking' News: The OpenAI–Hugging Face Incident

  submitted by   /u/SilentLennie [link]   [comments]

38
r/LocalLLaMA community 7d ago

New Unsloth KImi K3 drops! Q1_0 (466GB), TQ1_0(509GB), IQ1_M(649),TQ2_0(551GB)!!

The smallest UD-Q1_0 is 466GB, TQ2_0 551GB. Well done team Unsloth! https://huggingface.co/unsloth/Kimi-K3-GGUF   submitted by   /u/Hannibalj2ca [link]   [comments]

10
r/LocalLLaMA community 7d ago

DS4 Flash incoming price increase "we've been able to reproduce their current prices even on rented GPUs"

https://preview.redd.it/kvfk26z2uwhh1.png?width=598&format=png&auto=webp&s=356a8793a6c31bc563d552aaa5a73112ced7372e https://preview.redd.it/xthbu87auwhh1.png?width=598&format=png&auto=webp&s=08f686fee339905a33609a0346f13163aedc2671 Hello, I've seen these tweets from dax…

16
r/LocalLLaMA community 7d ago

IS GLM 5.2, Kimi 2.7 still worth it?

Since now we have kimi k3 and next week we are getting Qwen 3.8 Max and also soon V4 pro Deepseek. I am curious if the old power house like Kimi 2.6/7 code and GLM.5.2 are all that relevant. especially for long hours of coding   submitted by   /u/Hannibalj2ca [link]…

22
r/LocalLLaMA community 7d ago

what will be the future of LocalLLaMA?

For a long time now, the most popular posts on LocalLLaMA have been either about using LLM in the cloud or about politics. I suspect that people using local models are about 10% now. You can say that this is very good, because now it is an inclusive sub, without gatekeeping. But…

28
r/LocalLLaMA community 7d ago

This is not a conspiracy

  submitted by   /u/Neighbor_ [link]   [comments]

20
r/LocalLLaMA community 7d ago

Custom Water Cooled Quad 7900 XTX Build 96 GB VRAM

  submitted by   /u/tictacturkey [link]   [comments]

11
r/LocalLLaMA community 7d ago

AMD Acquires Taalas to Advance Compute Solutions for Rapidly Growing AI Inference Market

Press My earlier prediction that Tesla would buy them completely missed the mark. With AMD focusing heavily on the enterprise side, the idea of consumer-facing hot-swappable AI model chips looks pretty much dead. Fast forward ten years, you might find used model blade cards on…

18
r/LocalLLaMA community 7d ago

EschaLabs/Qwen3.6-35B-A3B-Escha-W2 · Hugging Face

Hey peeps. I know you're tired of low quants giving hard to believe numbers. I'm quite skeptical too and from what I tried I'm often left with the impression that the claims fall short. So this model popped up on Twitter for me. Tried it and was lowkey surprised it held its own.…

26
r/LocalLLaMA community 7d ago

cant wait for llms to be benchmaxxed on this /s

google facing yet another "innovator's dilemma" or some shi smh lgtb ig EDIT - WHEN THE FUCK WILL LE CHATON FAT COME RIGHT ON !!????   submitted by   /u/combo-user [link]   [comments]

36
r/LocalLLaMA community 7d ago

My issue with Artificial Analysis's 'intelligence index'

I swear AA is not the bipartisan they so claim. An open source mode (Qwen 3.8 max) was number 1 on the agentic index, then they just so happen to launch "v4.1.1" of their index in which they just adjusted the weights of the gdpval and t3 banking so that it would be lower than…

13
r/LocalLLaMA community 7d ago

Echo Dot 2 can run 28M LLM at decent speed

Code and instructions available here: https://github.com/albertoZurini/echo-dot-2-playground Hello there! After a few days of experimenting I was able to get a completely local voice pipeline running on an Amazon Echo Dot 2. The interesting part for this community is that the…

26
r/LocalLLaMA community 7d ago

BBC is running article titled "Artificial Intelligence used to design brand new viruses" ... cue the "We must regulate Open Weights Models to prevent the next Covid or worse" articles in 3... 2..

  submitted by   /u/johnnyApplePRNG [link]   [comments]

20
r/LocalLLaMA community 7d ago

Friday humor

  submitted by   /u/etherd0t [link]   [comments]

31
r/LocalLLaMA community 7d ago

Dual 3090 setup: 400 pp t/s to 1600 pp t/s on Qwen 3.6 27B... with slightly lower tps.

First of all, my setup: Ryzen 9 5950x DDR4 3200Mhz 64gb (2x32) Dual 3090s, no NVLINK Runtime: llama.cpp Nvidia Drivers 610 Windows 11 25H2 Qwen 3.6 27B Q8 I've been using llama-server with --split-mode tensor for a couple months now, since it gave a pretty nice 10%-20% boost in…

35
r/LocalLLaMA community 7d ago

🟩 NVIDIA's whole speech stack just went local. ASR + TTS + codec, quantized to GGUF, running on-device via NeMo-Speech.cpp

🐦‍⬛ Magpie-TTS Multilingual 🦜 Nemotron Speech Streaming EN 0.6B 🦜 Nemotron-3.5 ASR Streaming 🦜 Parakeet CTC 1.1B 🦜 Parakeet TDT 0.6B v3 🥦 NanoCodec Merged PR https://huggingface.co/nvidia/magpie_tts_multilingual_357m#run-magpietts-locally-with-nemo-speechcpp I run open…

9
r/LocalLLaMA community 7d ago

I thought Deepseek was the answer since I cannot afford GPU for local LLM

  submitted by   /u/HsSekhon [link]   [comments]

18
r/LocalLLaMA community 7d ago

Scotoma-2: Gemma4, but with less annoying slop and better writing.

GGUFs here: https://huggingface.co/ReadyArt/gemma-4-31B-it-scotoma-2-GGUF Disclaimer: By slop, we are specifically talking about specific tics with the model(sentence structures), but this doesn't include words such as "ozone". Summery of the model: Scotoma-2 is a model made by…

22
r/LocalLLaMA community 7d ago

Qwen 3.8 Max now ranked as best overall model ahead of Opus 5 by Artificial Analysis agentic index

  submitted by   /u/anderspitman [link]   [comments]

29
r/LocalLLaMA community 7d ago

New model release: Ling-3.0-tiny: 7.9B total parameters, with only 1.3B active per token- free for a week

A native hybrid reasoning model built for real-world tasks, math, instruction following, and resource-sensitive deployment.   submitted by   /u/niacolhealth [link]   [comments]

8

model: support Longcat-Flash (need testing) by ngxson · Pull Request #19182 · ggml-org/llama.cpp

DeepSeek V4 Flash 0731 appreciation post

Anyone else amped up over Qwen 3.8?

Qwen 35B-A3B MoE vs 27B dense in local coding tests: ~4× faster, much smaller quality gap than I expected

4x2080Ti 22GB tensor parallel possible?

PSA for anyone with multiple V620's or other gfx1030 cards having problems making llama.cpp tensor split work -- set "-ub 384" and -b to a multiple of that depending on number of GPUs

I got tired of my 300GB model loads taking 5min on RPC. PR 26291 speeds it 300% to 1min30sec (4060ti+ddr4) + (4060ti+ddr5)

BeeLLama issues

Is anyone else finding DeepSeek-V4-Flash unreliable for non-coding tasks?

U.S. Department of Energy Launches the Genesis Open Models Initiative and, with Arcee, Unveils Genesis-Science-1 — Its First Open-Weight Model for Scientific Research

Serving Deepseek v4 Flash 0731 on 2x DGX Spark — 5-7 GB OS headroom, what would you do to lower VRAM usage and increase OS available RAM?

Qwen 3.6 27B flags/settings in llama.cpp

Why no good external multi gpu solutions?

Which model would be best for Delphi development?

DeepSeek V4 Flash 0731 - ARC-AGI Results

How to even compare quants from various sources?

DeepSeek v4 Flash 0731 on H100 node

Got job as Director of AI and Systems development self-taught

parakeet.wgsl – Fast, accurate ASR in the browser, via raw WebGPU & SIMD WASM

llama.cpp PR reports up to 169% faster quantized-KV decode at 118K context on Intel Battlemage from one SYCL kernel switch

A visualization of LLM API costs to ask for local resources

Am I just hallucinating

Wan-Animate-2: Pushing the Application Boundaries of Character Animation Models

LFM2.5-2.6B model+KV cache quantization report

A llama.cpp PR makes Q2_0 3.0–3.6x faster on x86 CPUs, 8B decode goes 2.39 → 8.20 tok/s

LabyrinthBench: a local-focused, judge-free LLM benchmark that measures context recall under interference for multi-step agentic tasks.

RTX 5090 Owner Built An Open-Source Tool That Shuts Down PC If It Detects The 12VHPWR Cable Drawing Too Much Power, But It Can Only Work On Specific GPUs

Anyone running DeepSeek-V4-Flash-0731 on MI325X with vLLM? Mine is behaving completely broken

An open-weight model too, Moonshot joins the race (gently this time)

Gemma 4 QAT could be improved further by Google aligning the QAT model to modern q4_k instead of q4_0

Black Hat USA 2026: The 'Breaking' News: The OpenAI–Hugging Face Incident

New Unsloth KImi K3 drops! Q1_0 (466GB), TQ1_0(509GB), IQ1_M(649),TQ2_0(551GB)!!

DS4 Flash incoming price increase "we've been able to reproduce their current prices even on rented GPUs"

IS GLM 5.2, Kimi 2.7 still worth it?

what will be the future of LocalLLaMA?

This is not a conspiracy

Custom Water Cooled Quad 7900 XTX Build 96 GB VRAM

AMD Acquires Taalas to Advance Compute Solutions for Rapidly Growing AI Inference Market

EschaLabs/Qwen3.6-35B-A3B-Escha-W2 · Hugging Face

cant wait for llms to be benchmaxxed on this /s

My issue with Artificial Analysis's 'intelligence index'

Echo Dot 2 can run 28M LLM at decent speed

BBC is running article titled "Artificial Intelligence used to design brand new viruses" ... cue the "We must regulate Open Weights Models to prevent the next Covid or worse" articles in 3... 2..

Friday humor

Dual 3090 setup: 400 pp t/s to 1600 pp t/s on Qwen 3.6 27B... with slightly lower tps.

🟩 NVIDIA's whole speech stack just went local. ASR + TTS + codec, quantized to GGUF, running on-device via NeMo-Speech.cpp

I thought Deepseek was the answer since I cannot afford GPU for local LLM

Scotoma-2: Gemma4, but with less annoying slop and better writing.

Qwen 3.8 Max now ranked as best overall model ahead of Opus 5 by Artificial Analysis agentic index

New model release: Ling-3.0-tiny: 7.9B total parameters, with only 1.3B active per token- free for a week