r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 11d ago

Are you ready for Le Chaton FAT or still wasting money on GPUs?

According to rumors (spread by myself) Le Chaton FAT will be 26T-a3b and I AM READY for it. Let's be real, I can't afford that many 5060Ti, so I got 12x Gen 4 3.2 TB (two per card). This gives me about 60GBs bandwidth on 30TB. Added 256gb ddr4 just for kv cache, but I can also…

8
r/LocalLLaMA community 11d ago

Deepseek v4 flash - 100-150 faster t/s in prefill/pp.

You have two choices here (in order of pref): Downgrade CUDA from 13.3 to 13.1 (skip 13.2 due to bugs) <- prefer this (thanks to u/fairydreaming for pointing this out) Use this vibed fork that works with CUDA 13.3 https://github.com/vektorprime/working_ds4_speed I was…

34
r/LocalLLaMA community 12d ago

Deepseek-V4-Flash-0731 Dwarfstar on Mac

Here is the prefill performance in an M2 Ultra with 192GB of RAM. For decode, at the following depth: Start: 28 t/s 45k: 23.5 t/s 192k: 18 t/s That speed is maintained with 8k token output at those depths.   submitted by   /u/Badger-Purple [link]   [comments]

17
r/LocalLLaMA community 12d ago

Conclusion: r/LocalLLaMA still has brilliant open-weight research, but finding it requires wading through endless benchmark drama, non-local Discussion Points and repetitive hardware flexes.

I let Gemma4-31b run on my laptop for like almost a day using a heavily altered pi to do a deep dive on our beloved Llama tangentially related Subreddit, and this was the conclusion. Feels pretty accurate. Kind funny to let a small LLM loose and see what happens. Next target I'm…

30
r/LocalLLaMA community 12d ago

Has Qwen 3.8 has dropped yet? Day 90...

  submitted by   /u/mailto_devnull [link]   [comments]

32
r/LocalLLaMA community 12d ago

llama.cpp just added MTP / DSpark support for DeepSeek V4 Flash

  submitted by   /u/rmhubbert [link]   [comments]

21
r/LocalLLaMA community 12d ago

Vacuum 16T

https://huggingface.co/tsfrm/vacuum-16t A 16.5-trillion-parameter model that contains nothing. This model is just a ████ you to the labs and companies who say that "haha I have the biggest model out there!". We the people with shitty laptops want to get a record. And I now have…

4
r/LocalLLaMA community 12d ago

Xberg v1 is out

Hi all, I'm happy to announce that Xberg v1 is out. Xberg is the successor to Kreuzberg, equivalent to what would have been Kreuzberg v5. It's a content intelligence framework that handles a very wide range of inputs: documents (currently 101 formats), code and data formats…

16
r/LocalLLaMA community 12d ago

[Release] WinterMix — Qwen3.5-122B-A10B in native MLX: an 82 GiB build that beats 94–95 GiB quants, plus a 68 GiB build for agent swarms

TL;DR: I spent 9 days developing a new quantization method for MLX models and measured 18 variants against each other on a single M5 Max MacBook Pro (128 GB). The result is the best-measuring MLX quant of Qwen3.5-122B-A10B I'm aware of at any size — the 82 GiB build edges out…

8
r/LocalLLaMA community 12d ago

Setting up of a 16xGB10 (DGX Spark) cluster

Preparing this to be able to run locally frontier level open models. Deepseek v4 pro, Kimi K3, future ones like GLM 5.5 and Minimax M4. 16x Asus GX10 linked by mikrotik crs804-4ddq with 4 breakout cables of 400 to 100gbit. Most probable I will be running 2 models on 8x cluster…

13
r/LocalLLaMA community 12d ago

DeepSeek-V4-Flash 284B on 5.3GB of memory

Following up on my Qwen 3.6 port , I wanted to keep adding models and ended up fixing a bunch of things along the way, so it's its own engine now: Mference . Same core idea from TurboFieldfare , MoE models activate a few B params per token, so keep the shared core and KV cache…

8
r/LocalLLaMA community 12d ago

PSA for DeepSeek-V4-Flash-0731 users — don't blow out your prompt cache with system role messages mid-conversation

DSv4F doesn't ship a jinja, but for distributions that do and faithfully reconstruct what DS releases in their chat template python, every system message is hoisted into the system prompt at the top -- the format has no mid-conversation system turn. So, anything you stick at the…

13
r/LocalLLaMA community 12d ago

Real-world reality check on Qwen for autonomous coding agents

TLDR below 👇🏼 I’ve seen a lot of hype around Qwen 3.6 35B and 3.5 120B lately, especially regarding coding and tool-use capabilities. On this subreddit it is the defacto recommended model for everyone without a Datacenter at home. I’ve been running Qwen 3.5 120B…

9
r/LocalLLaMA community 12d ago

DeepSeek-V4-Flash-0731 UD-IQ3_XXS about 11t/s on 1x 7900 XTX 24GB + 3x MI60 32GB + 128GB DDR4

Hello, Also I want to join the hype of posting token specs. CPU: 2x Intel Xeon CPU E5-2650 v4 @ 2.20GHz RAM: 2x 4 Channel 2400MHz DDR4 GPU: 1x AMD Radeon 7900 XTX 24GB 3x AMD Instinct MI60 32GB Strange GPU combination, right? One of my AMD Instinct MI60 32GB failed, and I have…

35
r/LocalLLaMA community 12d ago

I pushed Kimi K3 onto one CPU with 8 GB of RAM

I deployed K3 on 32 H100s at work a couple of weeks ago and then got annoyed that there was no way to poke at it on my own machine . So I wrote an inference engine for it in C99. Nothing clever going on. 93% of that 1.56 TB checkpoint is routed experts, and only 16 of 896 fire…

14
r/LocalLLaMA community 12d ago

DeepSeek-V4-Flash-0731 UD-Q8_K_XL 17.20~ t/s on A6000 + 256GB DDR4

Hello everyone I want to join the hype of posting specs. CPU: AMD EPYC 74F3 24-Core RAM: 8 Channel 3200 DDR4 GPU: RTX A6000 48GB Prompt processing is in the high 70t/s (got down to mid 30t/s at 300k context). Inference is a steady 17.20t/s~ and the 48GB VRAM is enough to have…

10
r/LocalLLaMA community 12d ago

Ran DS V4-Flash-0731 Locally on 3xMI50 32GB @ ~15 t/s TG

Hey y'all. I'll be concise. TL;DR: DS V4-Flash-0731 @ UD-IQ2_M running fully in VRAM on 3xMI50s (90.9 GB model, 96 GB VRAM). Actual speed on llama-server is: - Text Generation: ~15-16 tokens/second stable. Never dipped below 14 tokens/second, even when the model was spitting out…

25
r/LocalLLaMA community 12d ago

Expert-only IQ3 requant of DeepSeek-V4-Flash-0731: better KLD than UD-IQ3_S, 1.4x decode on a CPU-spill rig

Hey all, tldr / who this helps: you run a mixed multi-GPU box where the experts spill to RAM, and you want to stay in the 3-bit tier instead of dropping to Q2 to make it fit. https://huggingface.co/TacoTakumi/DeepSeek-V4-Flash-0731-GGUF I requantized only the 129 routed expert…

19
r/LocalLLaMA community 12d ago

Why are almost all new benchmarks and leaderboards coding focused?

I know in in this community LLM's are generally used for coding but there are other usecases besides coding and those usecases should be tested too. I also know benchmarks can sometimes be benchmaxxed and the model can still turn out shit but it can give a good outline on how a…

35
r/LocalLLaMA community 12d ago

Koboldcpp v1.118 released

  submitted by   /u/Fcking_Chuck [link]   [comments]

12
r/LocalLLaMA community 12d ago

DeepSeek-V4-Flash-0731 on Bosgame M5 with RTX PRO 6000 Max-Q eGPU

Here are my numbers: Quant Size Layout Decode Prefill Draft acceptance UD-Q8_K_XL 150.8 GiB 20 layers CUDA0 / 23 ROCm0 + drafter 44.0 t/s 564 t/s 0.535 UD-Q4_K_XL 144.4 GiB 22 / 21 + drafter 48.4 t/s 585 t/s 0.532 UD-Q2_K_XL 90.2 GiB entirely on CUDA0, no drafter 59.5 t/s 1513…

17
r/LocalLLaMA community 12d ago

DeepSeek-V4-Flash-0731 UD-IQ3_S 12.5 tok/s on RTX 3090 +128GB DDR5

I managed to run DeepSeek-V4-Flash-0731 UD-IQ3_S in text-generation-webui with: RTX 3090 24 GB 128 GB DDR5 overclocked to 5600 MHz using AMD EXPO llama.cpp loader First, I had to use a rather brutal workaround: I replaced the llama.cpp binaries included with…

12
r/LocalLLaMA community 12d ago

A collection of small domain-specific benchmarks for local models (30+ and growing)

Hello fellow local AI people! I took "you must create your own benchmarks" literally, and built a website for this. How does the end result look like Let's say I want to know which model has most common sense in its responses, I did everything including evaluating responses (see…

34
r/LocalLLaMA community 12d ago

What’s next for Qwen open-source releases?

Been using Qwen 3.6 35B-A3B quite extensively lately and honestly, I’m pretty happy with it. Also tried a few community improvements like Ornith 1.0, which add some interesting tweaks. That said, I’m curious about what the community expects next from Qwen’s open-source roadmap.…

8
r/LocalLLaMA community 12d ago

Is there a point where models just cannot get any smaller without losing intelligence?

DeepSeek V4 Flash got me thinking... We keep seeing smaller models get way better. A model at a certain parameter count today can be much smarter than a model of the same size from a year or two ago. Better training, better data, better architectures, distillation, MoE, and all…

38
r/LocalLLaMA community 12d ago

Fix for Deep Seek v4 Flash 0731 tool calling has been added to llama cpp

This pull request was added to the main llama cpp about 12 hours ago. I was experiencing some looping and poor behavior yesterday but haven't had any problems since this fix. https://github.com/ggml-org/llama.cpp/pull/26269   submitted by   /u/kwizzle [link]  …

37
r/LocalLLaMA community 12d ago

I've had ling-3.0-flash and glm-5.2 both in my executor slot for a few weeks. They don't split the way the benchmarks predict

Same harness, same task set, same agent scaffold, the only thing I swapped was the executor. Not a proper benchmark, no clean tok/s numbers, this is a workflow read not a leaderboard. glm-5.2 is the better model and it shows on anything that needs an actual decision. When the…

22
r/LocalLLaMA community 12d ago

Deepseek v4 flash 0731 still not holding up.

The biggest issue with preview was its inability to follow rules prompts and skills. It seems like no matter what you do it ignores them. I've tried first person and second person. I've tried Chinese and English. It does not follow them. That's the only problem with these models…

14
r/LocalLLaMA community 12d ago

DeepSeek V4 Flash 0731 IQ2_M benchmark for Dual 3060 and 96GB RAM ≈ 3.5 tok/s.

Thanks to the community help I finally launched this llm. LM Studio refused to load weight onto second GPU but Unsloth Studio did so everything was done in there. Not a proper benchmark (used PC in parallel as well) but it gives an idea of the performance from dual 3060 with…

16
r/LocalLLaMA community 13d ago

EU AI Act takes effect tomorrow, August 2, 2026. 🤡

Basically you now have to mark all AI generated images, audio, video and text as AI generated. :P   submitted by   /u/xoxaxo [link]   [comments]

32
r/LocalLLaMA community 13d ago

Tomte - super fast harness for Gemma 4

I think people are sleeping on Gemma and local models so I built a free, very fast harness for Gemma 4 that I call Tomte. https://tomteapp.com Works on Macs with M processors, will have a companion app you can connect to anywhere. So far does everything I ever needed chatGPT…

5
r/LocalLLaMA community 13d ago

LongCat-Flash-Lite-Sparse Is Now Available for Download

The weights have now been added to the repo an hour ago. This model is built upon LongCat-Flash-Lite , the differences are that LongCat-Flash-Lite-Sparse : Replaces dense MLA with LongCat Sparse Attention (LSA) Natively supports context lengths of up to 1M tokens (vs 256k for…

13
r/LocalLLaMA community 13d ago

I'm kinda tired of obsession for one-shot tests in coding, there are good tests for multi-step debugging with analyzing output/images/videos?

Personally, i think good coding model shouldn't be focused on one-shot "everything in one html-file" tests, but should be really good on debugging, fixing and modifying its own output. Anyone know such simple tests that i would able to run with local models? May be some kind of…

15
r/LocalLLaMA community 13d ago

DSv4 Flash 0731 Running on Unoptimized Single 3090 System

https://preview.redd.it/eqqebec92sgh1.png?width=873&format=png&auto=webp&s=4e9a120e439dc0a64ee4aef87d84b45c51f6bf02 - 3x32 DDR4 2400 Mhz ECC. CPU itself supports quad channel, and we already know why am i haven't filling those slot yet. - E5 2690v4. - 3090 Running on 250W. -…

15
r/LocalLLaMA community 13d ago

New official weights for Laguna S 2.1 FP8 & NVFP4 are now available

Poolside have updated the FP8 and NVFP4 checkpoints for Laguna S 2.1, increasing the default context size to 1 million, and updating the configs. Here's hoping they fixed the looping issue, this model has been great in my development workflows, when not looping.   submitted…

6
r/LocalLLaMA community 13d ago

Gemma4 (31B, bf16) constantly fails to edit files due to mismatches in original text - just me?

I've been trying to find a good model to run locally, and in the benchmarks I can Gemma4 does well. However, whenever I give it anything that involves writing any code, it sits in loops trying to edit files sending the wrong original content (usually it messes up something like…

37
r/LocalLLaMA community 13d ago

DeepSeek-V4-Flash-0731: Oneshot evals, surprisingly not token efficient??

I ran the newly released DeepSeek-V4-Flash-0731 in my oneshot eval harness across 34 prompts and here are the results. https://oneshotlm.com/model/deepseek-deepseek-v4-flash-0731/ The providers on openrouter were unstable and I had to retry generation multiple times.…

10
r/LocalLLaMA community 13d ago

DS4 flash 0731 - Acquarium Panel Failure - Q3_K_XL Unsloth

https://preview.redd.it/1a39x4zivqgh1.png?width=1550&format=png&auto=webp&s=de591c039cc18782a6b5d8e402fdc1594be05132 start C:\llm\llamam5\build\bin\llama-server.exe --model "H:\UD-Q3_K_XL\DeepSeek-V4-Flash-0731-UD-Q3_K_XL-00001-of-00004.gguf" --host 127.0.0.1 --port 8080 -c…

33
r/LocalLLaMA community 13d ago

…3.8…

Like when?   submitted by   /u/JLeonsarmiento [link]   [comments]

32
r/LocalLLaMA community 13d ago

New DeepSeek V4 Flash 0731 vs ChatGPT Luna comparison

  submitted by   /u/perelmanych [link]   [comments]

23
r/LocalLLaMA community 13d ago

Deepseek V4 Flash 0731. LM Studio loading only into RAM.

The model refuses to load into VRAM and uses only RAM. What can be an issue? Q2_K_XL from Unsloth if that changes something.   submitted by   /u/esw123 [link]   [comments]

29
r/LocalLLaMA community 13d ago

DeepSeek-V4-Flash-0731: Models you can run locally now have the intelligence score of the top frontier model from March 2026

March 6th, 2026 the highest intelligence index score was 51 for frontier models. deepseek-ai/DeepSeek-V4-Flash-0731 that has an intelligence score of 50. If these benchmarks are accurate, models available to run locally on <8K USD (us prices - just guestimating/not exact)…

19
r/LocalLLaMA community 13d ago

Are 1B LLMs Going Away in 2026?

I don't know much about llms aside from downloading them through a frontend and running them on my laptop or potato phone. Google released gemma 4, but unlike gemma 3, there isn't a 1b model this time. Llama also had a 1b model before, but there doesn't seem to be a new one.…

21
r/LocalLLaMA community 13d ago

Weight-Aware Streaming Tensor Engine: run Kimi K3 using 29 GB of RAM at 0.50 tok/s

  submitted by   /u/galapag0 [link]   [comments]

29
r/LocalLLaMA community 13d ago

Qwen 3.6 27B Q5 on 3x2080ti: 55tps with llama.cpp. Can I squeeze out more?

CPU: Threadripper 3970X RAM: 128GB DDR4 GPUs: 3x2080ti 11GB The current best parameters to run it: llama-server \ --model Qwen3.6-27B-Q5_K_S.gguf \ --n-gpu-layers 999 \ --split-mode tensor \ --flash-attn on \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --ctx-size 16384 \…

8
r/LocalLLaMA community 13d ago

What speeds are everyone getting with deepseek v4 flash 0731?

What speeds are everyone getting with deepseek v4 flash 0731? I’m getting~200 tps prompt processing / ~11 tps token gen, on 4x5060ti16gb with ddr4 3200 ram at 4-channel, via llamacpp, with context window of 128000, -ub/-b at 4096, “q8” unsloth’s lossless quant   submitted by…

36
r/LocalLLaMA community 13d ago

Me: Worn out from all the new model drops this week, but still hyped for all the great new releases.

I mean seriously y’all, what an amazing past few days. So many awesome new models to test out in the mid range model sizes.   submitted by   /u/Porespellar [link]   [comments]

15
r/LocalLLaMA community 13d ago

DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731.gguf

Antirez stealthily uploaded the new weights in the old folder... and there we were tapping our fingers. https://huggingface.co/antirez/deepseek-v4-gguf/tree/main   submitted by   /u/challis88ocarina [link]   [comments]

14
r/LocalLLaMA community 13d ago

[audio.cpp] Release 0.5: DramaBox expressive TTS, Confucius4 cross-lingual voice transfer, plus 7 more models and ROCm/HIP

audio.cpp 0.5 is out :) The most fun new model in 0.5 is DramaBox . It is closer to prompt-directed voice acting. DramaBox is built on the LTX-2.3 audio architecture, and prompts can control emotion, delivery, laughs, sighs, pauses, transitions, and speaker behavior. Example…

15
r/LocalLLaMA community 13d ago

DeepSeek v4 Flash for DS4 (DwarfStar) GGUF w/ DSpark MTP Head

I'm an avid user of Deepseek v4 Flash via antirez's DS4 DwarfStar inference engine , and so when the new checkpoint dropped, the first thing I did was rent a cloud box and spin up a quantization for use in my DS4 deployment. Props to Unsloth for getting their GGUFs out so…

7

Are you ready for Le Chaton FAT or still wasting money on GPUs?

Deepseek v4 flash - 100-150 faster t/s in prefill/pp.

Deepseek-V4-Flash-0731 Dwarfstar on Mac

Conclusion: r/LocalLLaMA still has brilliant open-weight research, but finding it requires wading through endless benchmark drama, non-local Discussion Points and repetitive hardware flexes.

Has Qwen 3.8 has dropped yet? Day 90...

llama.cpp just added MTP / DSpark support for DeepSeek V4 Flash

Vacuum 16T

Xberg v1 is out

[Release] WinterMix — Qwen3.5-122B-A10B in native MLX: an 82 GiB build that beats 94–95 GiB quants, plus a 68 GiB build for agent swarms

Setting up of a 16xGB10 (DGX Spark) cluster

DeepSeek-V4-Flash 284B on 5.3GB of memory

PSA for DeepSeek-V4-Flash-0731 users — don't blow out your prompt cache with system role messages mid-conversation

Real-world reality check on Qwen for autonomous coding agents

DeepSeek-V4-Flash-0731 UD-IQ3_XXS about 11t/s on 1x 7900 XTX 24GB + 3x MI60 32GB + 128GB DDR4

I pushed Kimi K3 onto one CPU with 8 GB of RAM

DeepSeek-V4-Flash-0731 UD-Q8_K_XL 17.20~ t/s on A6000 + 256GB DDR4

Ran DS V4-Flash-0731 Locally on 3xMI50 32GB @ ~15 t/s TG

Expert-only IQ3 requant of DeepSeek-V4-Flash-0731: better KLD than UD-IQ3_S, 1.4x decode on a CPU-spill rig

Why are almost all new benchmarks and leaderboards coding focused?

Koboldcpp v1.118 released

DeepSeek-V4-Flash-0731 on Bosgame M5 with RTX PRO 6000 Max-Q eGPU

DeepSeek-V4-Flash-0731 UD-IQ3_S 12.5 tok/s on RTX 3090 +128GB DDR5

A collection of small domain-specific benchmarks for local models (30+ and growing)

What’s next for Qwen open-source releases?

Is there a point where models just cannot get any smaller without losing intelligence?

Fix for Deep Seek v4 Flash 0731 tool calling has been added to llama cpp

I've had ling-3.0-flash and glm-5.2 both in my executor slot for a few weeks. They don't split the way the benchmarks predict

Deepseek v4 flash 0731 still not holding up.

DeepSeek V4 Flash 0731 IQ2_M benchmark for Dual 3060 and 96GB RAM ≈ 3.5 tok/s.

EU AI Act takes effect tomorrow, August 2, 2026. 🤡

Tomte - super fast harness for Gemma 4

LongCat-Flash-Lite-Sparse Is Now Available for Download

I'm kinda tired of obsession for one-shot tests in coding, there are good tests for multi-step debugging with analyzing output/images/videos?

DSv4 Flash 0731 Running on Unoptimized Single 3090 System

New official weights for Laguna S 2.1 FP8 & NVFP4 are now available

Gemma4 (31B, bf16) constantly fails to edit files due to mismatches in original text - just me?

DeepSeek-V4-Flash-0731: Oneshot evals, surprisingly not token efficient??

DS4 flash 0731 - Acquarium Panel Failure - Q3_K_XL Unsloth

…3.8…

New DeepSeek V4 Flash 0731 vs ChatGPT Luna comparison

Deepseek V4 Flash 0731. LM Studio loading only into RAM.

DeepSeek-V4-Flash-0731: Models you can run locally now have the intelligence score of the top frontier model from March 2026

Are 1B LLMs Going Away in 2026?

Weight-Aware Streaming Tensor Engine: run Kimi K3 using 29 GB of RAM at 0.50 tok/s

Qwen 3.6 27B Q5 on 3x2080ti: 55tps with llama.cpp. Can I squeeze out more?

What speeds are everyone getting with deepseek v4 flash 0731?

Me: Worn out from all the new model drops this week, but still hyped for all the great new releases.

DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731.gguf

[audio.cpp] Release 0.5: DramaBox expressive TTS, Confucius4 cross-lingual voice transfer, plus 7 more models and ROCm/HIP

DeepSeek v4 Flash for DS4 (DwarfStar) GGUF w/ DSpark MTP Head