r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 10d ago

DeepSeek V4-Flash (284B MoE) at 33 tok/s single / 68 tok/s aggregate on 2× RTX 3090 + a used quad-Xeon DDR4 server — full config

Ran DeepSeek V4-Flash-0731 — the full official checkpoint, not a re-quant — on commodity used hardware. Sharing because I couldn't find anyone else publishing Ampere results for this engine. Why bother with a 2018 server The model is 156 GB . That number decides everything…

34
r/LocalLLaMA community 10d ago

[Paper] Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression

In this paper, the authors tackle continued pretraining without the risk of catastrophic forgetting, by identifying parameters which can safely be changed without risking identified concepts, and freezing the rest: https://arxiv.org/abs/2604.19089v1 Current practice is to mix…

8
r/LocalLLaMA community 10d ago

'I ran my own benchmarks on it' seems to be pretty common comment around here. How about dedicating a thread for this and sharing?

Of course, the concern is that in the end, this thread will be fed into the models' training data, but I feel benchmarking isn't so open and very fragmented.   submitted by   /u/jinnyjuice [link]   [comments]

38
r/LocalLLaMA community 10d ago

[NEW MODELS!] Supra2-100M Base and Instruct - go check them out!

Hey guys! After a LOT of good feedback on our previous models like Supra-50M-Instruct and -Reasoning, many community likes, follows and upvotes we saw many community requests asking for new models. We've inspired a lot of people with our work - and now we're presenting the all…

14
r/LocalLLaMA community 10d ago

Qwen3.8-Max matches Kimi K3 and DeepSeek V4 Flash

Qwen3.8-Max (2.4T) is another massive contribution to the open weight community. On benchmarks, it performs closely to Kimi K3 and DeepSeek V4 flash across all categories and is better at coding and software tasks. Qwen3.8-27B will also be open weight soon too. Weights are being…

11
r/LocalLLaMA community 10d ago

Thermal paste PSA for old GPUs

I know many of us are using older GPUs like the 3090 because they work great. I just replaced the thermal paste and am seeing consistently 10 C lower temperatures. The old paste was cracking and like dry dust when I removed it. This made the difference between super loud fans…

8
r/LocalLLaMA community 10d ago

The Chinese labs everyone lumps together are making four pretty different bets. I work at one of them.

Every time a model drops from a Chinese lab the thread fills with people who already know who made it, and the guess is usually Alibaba. There was a thread here recently asking what separates the open source labs from the frontier labs. It ran to nearly sixty comments and hardly…

30
r/LocalLLaMA community 10d ago

I CANNOT believe I've got DeepSeek-V4-Flash-0731, a frontier model, running on my home PC. Insane!

So this is the stuff of absolute insanity. In less than 20 months we've gone from super expensive cloud models only, to being able to run a Q3 quant of DeepSeek on an Intel Windows PC with a very average 24GB of VRAM. No wonder the big boys are panicking (and yes it's slow as…

10
r/LocalLLaMA community 10d ago

an espresso Q/A model running fully offline on an ESP32S3

i already had an esp32 generating stories, but generating text is not the same as receiving a question and giving a useful answer. barista v0.1, a small model trained for espresso troubleshooting and running on an esp32s3 n16r8 witout cloud. you type a question(over usb for…

24
r/LocalLLaMA community 11d ago

All DeepSeek model oneshots: 242 outputs to look at and compare!

Continuing my weekend of oneshotting the cheap OpenRouter models, here are all 10 DeepSeek models across the same 35 prompts. DeepSeek had a rougher time (more provider errors / empty completions), so only 242 made it out of the 10*35 matrix. Here they are…

31
r/LocalLLaMA community 11d ago

"Data center in a Box (on Wheels)" 256Gb VRAM/512Gb RAM AI Server 6-8 Month Operational Review, Stability Write Up, Benchmarks

I've been out of these forums for awhile but I figured I would provide a formal update on how this has been going now that it has some operation time under its belt, just to put the information out there and share knowledge if there is any interest. I also wasn't satisfied with…

32
r/LocalLLaMA community 11d ago

Quantization hurts knowledge nonlinearly - Qwen3.6 27B case study

  submitted by   /u/pmigdal [link]   [comments]

35
r/LocalLLaMA community 11d ago

V4-Flash-0731 - vibes after first weekend of use

Spent way too much time with V4-Flash-0731 this weekend and wanted to share my vibes as briefly as possible. I sent it through a bit of real-work and some of my personal benchmarks. My quick thoughts are: Quantization hits this thing like a truck - I've tried a bunch of the Q2…

14
r/LocalLLaMA community 11d ago

70-class VRAM stagnation

been thinking about how the desktop 70-class has sat at 12GB for two generations now, 4070, 4070 super, 5070, all 12GB. the 1070 gave you 8GB back in 2016 and it felt generous for the price. ten years later the jump is... 4GB. and the thing is these chips arent even weak. the…

32
r/LocalLLaMA community 11d ago

I compared MinerU, Granite-Docling, and PaddleOCR-VL on 12 PDF-parsing capabilities using 6 document types

I tested them by sending the 6 documents, each meant to represent a different document type, through my own webapp and comparing every output against the source. All ran on the same L4 GPU. The documents: Financial statements with merged multi-level headers (A typical annual…

37
r/LocalLLaMA community 11d ago

model: MTP support for Qwen3-Next by yomaytk · Pull Request #25589 · ggml-org/llama.cpp

Now we can run Qwen3-Next at “full speed” :) Do you still remember this model?   submitted by   /u/jacek2023 [link]   [comments]

26
r/LocalLLaMA community 11d ago

ds v4 flash preview vs official

is all about post training? idk much about that spesific thing, whats matter most to achieve more performance, optimized results i wonder   submitted by   /u/Initial-Carry1038 [link]   [comments]

26
r/LocalLLaMA community 11d ago

KAT Coder 2.5 dev: Do yourself a favor and try it!

It is so good! I don't know why there aren't more people talking about it. Fewer tokens, faster and more accurate than Qwen 3.6 35b a3b. On my setup it's nearly as good as 27b, but 5x faster. And it completely trashes the Gemma 4 models. At least for my use case, it feels…

31
r/LocalLLaMA community 11d ago

GLM 5.3 Spotted

https://github.com/zai-org/z-ai-sdk-java/commits/glm-5.3   submitted by   /u/Few_Painter_5588 [link]   [comments]

10
r/LocalLLaMA community 11d ago

AI9Stars released G9v3-39A5B

AI9Stars has released G9v3-39A5B an open weights language model designed to deliver even stronger reasoning capabilities than ai9stars/G9v3-3B with its 39B and 5 active experts. It is released under the Apache 2.0 license making it fully open for personal and commercial use It…

10
r/LocalLLaMA community 11d ago

[RELEASE] SupraBrain-50M-v0.1

Hey there! So today we're releasing SupraBrain-50M, a hybrid language model that combines Gated DeltaNet linear recurrence with Sliding-Window Attention and Surprise-Gated update mechanisms to deliver very strong performance. Here are the benchmarks:…

19
r/LocalLLaMA community 11d ago

Döner Bench DeepSeek-V4-Flash IQ2_XS running on a single RTX 3090

https://preview.redd.it/3zcvpbds14hh1.png?width=1911&format=png&auto=webp&s=a79aafb71eeca97638da93d2591902631e897fd5 I tried a test similar to the recent model-quant comparisons, but this time I focused only on: DeepSeek-V4-Flash-0731-IQ2_XS-Experts-Q8_0 model link…

16
r/LocalLLaMA community 11d ago

I benchmarked classic vector RAG vs Google's new OKF format vs both combined — same corpus, same 7 questions, all local (Ollama + ChromaDB)

Google Cloud published OKF (Open Knowledge Format) on June 12th — a spec for storing curated knowledge as a directory of markdown files with YAML frontmatter. One concept per file, linked to each other, with an index.md for progressive disclosure. The only required field is…

7
r/LocalLLaMA community 11d ago

Daniel Han of Unsloth validates Qwen3.8-27B will run only 17GB VRAM

Super excited about this release for the new 27B. Who else is with me. Only 17GB VRAM needed 😍😍   submitted by   /u/quantier [link]   [comments]

7
r/LocalLLaMA community 11d ago

Can't wait to see Qwen3.8-27B

Qwen announced Qwen3.8 a few hours ago, and it looks like we’re getting a new 27B model! Really excited to try this one locally.   submitted by   /u/FormOne2615 [link]   [comments]

34
r/LocalLLaMA community 11d ago

Seedance 2.5 Vs Minimax H3 (Open Weight). Excellent Output Comparison!

  submitted by   /u/Hannibalj2ca [link]   [comments]

23
r/LocalLLaMA community 11d ago

Open weight has made to frontier

Am looking forward to this! Open weight has come near frontier for 5x less the cost per/M tokens on task completion Open weight ranking Kimi k3 Qwen 3.8 GLM 5.2 Deepseek v4 flash 07/31   submitted by   /u/Specialized-Trap404 [link]   [comments]

4
r/LocalLLaMA community 11d ago

MiniMax-H3 now on huggingface

MiniMax H3 is a general-purpose, omni-modal generative system. It supports unified understanding of multimodal contexts composed of text, images, video, and audio, and can generate video with native stereo audio at resolutions up to 2K and durations of up to 15 seconds. Thanks…

16
r/LocalLLaMA community 11d ago

Qwen3.8-27B announced alongside Qwen3.8-Max

https://preview.redd.it/gy0tgokdl2hh1.png?width=540&format=png&auto=webp&s=7db9e034613a915cb33d378b99ad72c31c7cc18f source: https://x.com/Alibaba_Qwen/status/2084100707423289643   submitted by   /u/TKGaming_11 [link]   [comments]

29
r/LocalLLaMA community 11d ago

Qwen 3.8 is live now.

Update: And yes, Qwen3.8-27B is coming too. Next week! 2.4-trillion-parameter MoE flagship delivering a comprehensive leap in coding and professional work. Autonomously codes and delivers complete projects spanning 10+ days. Open weights coming soon! It is live at…

19
r/LocalLLaMA community 11d ago

GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache.   submitted by   /u/ab2377 [link]  …

29
r/LocalLLaMA community 11d ago

Anyone Used MiniMAx H3 yet? Open Weights are out today!

I am curious if anyone have used it. I would love to feed it key frames and test if it can create in-between frames between my keys. Anyone have tried it, any thoughts?   submitted by   /u/Hannibalj2ca [link]   [comments]

14
r/LocalLLaMA community 11d ago

You really should not quantize KV Cache for DeepSeek V4 Flash

I don't think anyone should quantize the KV with DS4F. I checked the the quality impact (PPL, KLD, Same TopP) for swhitching from BF16 KV to Q8 KV, and it appears significant. Very much in contrast to Qwen 397B. Here are the results for DS4F: ====== Perplexity statistics ======…

11
r/LocalLLaMA community 11d ago

China’s DFSX Offers 2x The Memory Bandwidth Of NVIDIA’s GB200

  submitted by   /u/MundanePercentage674 [link]   [comments]

8
r/LocalLLaMA community 11d ago

PSA: llama.app, Mac app and llama serve from llama.cpp

https://llama.app/ Been using llama.cpp for years now and im on here all the time (im a mod..), but somehow I totally missed that llama.app exists and its official from the HF/llama.cpp team. So posting this as I'm quite sure I'm not the only one in this boat. The llama.cpp team…

5
r/LocalLLaMA community 11d ago

https://huggingface.co/poolside/Laguna-S-2.1-NVFP4

Updated release (August 2026). This is a new checkpoint that supersedes the earlier version of this repository. The weights have changed, not only the config, so if you downloaded a previous copy please re-download to pick up the current checkpoint.   submitted by  …

23
r/LocalLLaMA community 11d ago

DeepSeek V4 @ IQ3XXS on M1 Ultra 128GB- 16 tok/s in LM Studio after patch

M1 Ultra 128GB, Unsloth UD-IQ3_XXS, wired limit at 120GB. I was at 5-6 tok/s before the patch. Getting 15-16 tok/s now with the patched engine, and the output seems to have improved. Big thanks to this guy.   submitted by   /u/mil_phickelson [link]   [comments]

9
r/LocalLLaMA community 11d ago

Parlor v2: best-effort fully local GPT-Live clone on an M3 Pro

GPT-Live is so good that I use it almost every day. I've been wanting to replicate it since it was released. My first attempt was to fine-tune Gemma 4 12B to behave like a full-duplex model. Something like grafting a decision tick + speech head to the model. It failed after…

24
r/LocalLLaMA community 11d ago

DeepSeek-V4-Flash-0731: When Low is higher than High

I decided to test a few questions against DeepSeek-V4-Flash-0731. Locally, I was running Unsloth's UD-Q2_K_XL quant. After I saw the surprising shape of the results, I tested against DeepSeek's official API to confirm that I didn't do anything wrong. For anyone using OpenRouter,…

12
r/LocalLLaMA community 11d ago

DeepSeek-V4-Flash-0731: surpasses Fable-5, Sol & Kimi-K3 on Chess Benchmark

  submitted by   /u/mrwang89 [link]   [comments]

4
r/LocalLLaMA community 11d ago

DSpark Benchmark Result on Deepseek v4 Flash 0731

TensorSharp supports DSpark on Deepseek v4 Flash 0731 now. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8 with/without DSpark: Model: DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF DSpark draft model from:…

31
r/LocalLLaMA community 11d ago

All Qwen model oneshots: 1109 outputs to look at and compare!

I've been busy this weekend generating oneshots for all the cheapest models on the openrouter and ended up going through all 33 qwen models across 35 prompts (there were some failures and only 1109 made out of 33*35 matrix). Here they are https://oneshotlm.com/model/?q=qwen Qwen…

35
r/LocalLLaMA community 11d ago

Are you ready for Le Chaton FAT or still wasting money on GPUs?

According to rumors (spread by myself) Le Chaton FAT will be 26T-a3b and I AM READY for it. Let's be real, I can't afford that many 5060Ti, so I got 12x Gen 4 3.2 TB (two per card). This gives me about 60GBs bandwidth on 30TB. Added 256gb ddr4 just for kv cache, but I can also…

8
r/LocalLLaMA community 11d ago

Deepseek v4 flash - 100-150 faster t/s in prefill/pp.

You have two choices here (in order of pref): Downgrade CUDA from 13.3 to 13.1 (skip 13.2 due to bugs) <- prefer this (thanks to u/fairydreaming for pointing this out) Use this vibed fork that works with CUDA 13.3 https://github.com/vektorprime/working_ds4_speed I was…

34
r/LocalLLaMA community 11d ago

Deepseek-V4-Flash-0731 Dwarfstar on Mac

Here is the prefill performance in an M2 Ultra with 192GB of RAM. For decode, at the following depth: Start: 28 t/s 45k: 23.5 t/s 192k: 18 t/s That speed is maintained with 8k token output at those depths.   submitted by   /u/Badger-Purple [link]   [comments]

17
r/LocalLLaMA community 12d ago

Conclusion: r/LocalLLaMA still has brilliant open-weight research, but finding it requires wading through endless benchmark drama, non-local Discussion Points and repetitive hardware flexes.

I let Gemma4-31b run on my laptop for like almost a day using a heavily altered pi to do a deep dive on our beloved Llama tangentially related Subreddit, and this was the conclusion. Feels pretty accurate. Kind funny to let a small LLM loose and see what happens. Next target I'm…

30
r/LocalLLaMA community 12d ago

Has Qwen 3.8 has dropped yet? Day 90...

  submitted by   /u/mailto_devnull [link]   [comments]

32
r/LocalLLaMA community 12d ago

llama.cpp just added MTP / DSpark support for DeepSeek V4 Flash

  submitted by   /u/rmhubbert [link]   [comments]

21
r/LocalLLaMA community 12d ago

Vacuum 16T

https://huggingface.co/tsfrm/vacuum-16t A 16.5-trillion-parameter model that contains nothing. This model is just a ████ you to the labs and companies who say that "haha I have the biggest model out there!". We the people with shitty laptops want to get a record. And I now have…

4
r/LocalLLaMA community 12d ago

Xberg v1 is out

Hi all, I'm happy to announce that Xberg v1 is out. Xberg is the successor to Kreuzberg, equivalent to what would have been Kreuzberg v5. It's a content intelligence framework that handles a very wide range of inputs: documents (currently 101 formats), code and data formats…

16

DeepSeek V4-Flash (284B MoE) at 33 tok/s single / 68 tok/s aggregate on 2× RTX 3090 + a used quad-Xeon DDR4 server — full config

[Paper] Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression

'I ran my own benchmarks on it' seems to be pretty common comment around here. How about dedicating a thread for this and sharing?

[NEW MODELS!] Supra2-100M Base and Instruct - go check them out!

Qwen3.8-Max matches Kimi K3 and DeepSeek V4 Flash

Thermal paste PSA for old GPUs

The Chinese labs everyone lumps together are making four pretty different bets. I work at one of them.

I CANNOT believe I've got DeepSeek-V4-Flash-0731, a frontier model, running on my home PC. Insane!

an espresso Q/A model running fully offline on an ESP32S3

All DeepSeek model oneshots: 242 outputs to look at and compare!

"Data center in a Box (on Wheels)" 256Gb VRAM/512Gb RAM AI Server 6-8 Month Operational Review, Stability Write Up, Benchmarks

Quantization hurts knowledge nonlinearly - Qwen3.6 27B case study

V4-Flash-0731 - vibes after first weekend of use

70-class VRAM stagnation

I compared MinerU, Granite-Docling, and PaddleOCR-VL on 12 PDF-parsing capabilities using 6 document types

model: MTP support for Qwen3-Next by yomaytk · Pull Request #25589 · ggml-org/llama.cpp

ds v4 flash preview vs official

KAT Coder 2.5 dev: Do yourself a favor and try it!

GLM 5.3 Spotted

AI9Stars released G9v3-39A5B

[RELEASE] SupraBrain-50M-v0.1

Döner Bench DeepSeek-V4-Flash IQ2_XS running on a single RTX 3090

I benchmarked classic vector RAG vs Google's new OKF format vs both combined — same corpus, same 7 questions, all local (Ollama + ChromaDB)

Daniel Han of Unsloth validates Qwen3.8-27B will run only 17GB VRAM

Can't wait to see Qwen3.8-27B

Seedance 2.5 Vs Minimax H3 (Open Weight). Excellent Output Comparison!

Open weight has made to frontier

MiniMax-H3 now on huggingface

Qwen3.8-27B announced alongside Qwen3.8-Max

Qwen 3.8 is live now.

GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

Anyone Used MiniMAx H3 yet? Open Weights are out today!

You really should not quantize KV Cache for DeepSeek V4 Flash

China’s DFSX Offers 2x The Memory Bandwidth Of NVIDIA’s GB200

PSA: llama.app, Mac app and llama serve from llama.cpp

https://huggingface.co/poolside/Laguna-S-2.1-NVFP4

DeepSeek V4 @ IQ3XXS on M1 Ultra 128GB- 16 tok/s in LM Studio after patch

Parlor v2: best-effort fully local GPT-Live clone on an M3 Pro

DeepSeek-V4-Flash-0731: When Low is higher than High

DeepSeek-V4-Flash-0731: surpasses Fable-5, Sol & Kimi-K3 on Chess Benchmark

DSpark Benchmark Result on Deepseek v4 Flash 0731

All Qwen model oneshots: 1109 outputs to look at and compare!

Are you ready for Le Chaton FAT or still wasting money on GPUs?

Deepseek v4 flash - 100-150 faster t/s in prefill/pp.

Deepseek-V4-Flash-0731 Dwarfstar on Mac

Conclusion: r/LocalLLaMA still has brilliant open-weight research, but finding it requires wading through endless benchmark drama, non-local Discussion Points and repetitive hardware flexes.

Has Qwen 3.8 has dropped yet? Day 90...

llama.cpp just added MTP / DSpark support for DeepSeek V4 Flash

Vacuum 16T

Xberg v1 is out