r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 9d ago

MiniMax issues

https://www.reddit.com/r/StableDiffusion/s/HrU7odaJe6 I think this is more important that all the political stuff you share here   submitted by   /u/jacek2023 [link]   [comments]

12
r/LocalLLaMA community 9d ago

Qwen Developers' responses from their recent Twitter/X AMA

Questions & Responses(in BOLD ) below. Favorite question(s) moved to end of the thread with combined responses(removed duplicates). Be optimistic folks. I'm sure we're getting other models too apart from 27B. And 27B gonna make massive noise on release. (Based on their…

11
r/LocalLLaMA community 9d ago

I updated my localy run benchmark with DeepSeek V4 Flash 0731

It's the purple cluster on the top left (the good corner...) I'm running the MXFP4 version from Bartoswski with Dspark at 1K t/s prefill and 90 t/s gen (average). I tried different sampling params, you can check the detail. It's very efficient while scoring the best yet. Too bad…

12
r/LocalLLaMA community 9d ago

Anyone interested in building a harness-only benchmark?

There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one. End goal: a leaderboard of harness performance (multiple axis) on a set of diverse real world tasks [1] , grouped by underlying models…

12
r/LocalLLaMA community 9d ago

Building a Fully Local PDF Read-Aloud & PDF-to-Audiobook Desktop App with Kokoro 82M, Qwen, and llama.cpp

Hey everyone, I’ve been building Speechfony - a desktop app for reading PDFs (and EPUBs) with offline text-to-speech. Open a document, listen sentence-by-sentence with highlighting, or export selected pages to an MP3. Everything runs locally: Kokoro for speech, and an on-device…

5
r/LocalLLaMA community 9d ago

Qwen3-TTS voice cloning is now in mainline llama.cpp — the old demo finally became real support

People may remember the Qwen3-TTS llama.cpp demo from a few months ago. That PR said it probably wouldn’t be merged because llama.cpp was missing some of the graph and API pieces it needed. A new implementation was merged into master yesterday. What works now: -…

16
r/LocalLLaMA community 9d ago

Intern S2 Mobius

A Qwen3.5-35B derived model with an interesting architectural difference that results in larger throughput and less token consumption (allegedly): https://huggingface.co/internlm/Intern-S2-Mobius   submitted by   /u/Miserable-Dare5090 [link]   [comments]

7
r/LocalLLaMA community 9d ago

GPT-X2.5-135M scores 3rd place on Open SLM Leaderboard on Huggingface, Beating Facebook's MobileLLM-R1-140M

  submitted by   /u/Megneous [link]   [comments]

28
r/LocalLLaMA community 9d ago

China’s Open-Weight Models Will Be Spared US Safety Tests

  submitted by   /u/fallingdowndizzyvr [link]   [comments]

36
r/LocalLLaMA community 9d ago

VibeVoice 1.5B Running Locally...On an iPhone! Only ~2.2 GB of Memory and Up to 1.28× Real-Time Speed

I speed up the generation part of the demo in case you get bored 😄 I also tested another long-form generation, and the VRAM usage looks stable. The demo is about a minute long, and I posted it on X. This started as a random idea and somehow turned into a full detour from…

26
r/LocalLLaMA community 9d ago

Thinking of buying more DRAM right now...

So I'm looking at https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF and I realize my 128GB of DRAM just isn't cutting it for this (incredibly powerful) model. If only I had another 64GB, I thought... EVERYBODY is probably thinking that right this second... I hate to say…

4
r/LocalLLaMA community 9d ago

PSA Update CUDA from 13.2 to 13.3 to solve DeepSeek V4 Flash 0731 Looping Problem!

So one of yall mentioned that cuda 13.1 or 13.2 is broken for unsloth so I looked in to it, and they were right. I had 13.2 installed, after I switched to 13.3 no more looping!!! Before the cuda update, the model was literally unusable. A few minutes into the run it would start…

20
r/LocalLLaMA community 9d ago

Maple-Preview: 20B-A1B ternary-weight reasoning open-weight LLM

  submitted by   /u/cafedude [link]   [comments]

30
r/LocalLLaMA community 9d ago

DeepSeek-V4-Flash on SM89 4x48gb 4090s with DSpark

https://github.com/yhfgyyf/vllm-deepseek-v4-sm89 I couldn't believe that someone actually got vLLM working with this particular set of GPUs, but here it is. The video is from right after I got it working with 64k context, but it is now running with 256k.   submitted by  …

26
r/LocalLLaMA community 9d ago

DeepSeek-v4-Flash-Mini 54GB GGUF running at ~20.5 t/s

Took the REAP adaptation of DeepSeek-V4-Flash ( 0xSero/DeepSeek-V4-Flash-0731-REAP ) along with antirez/deepseek-v4-gguf as inspiration, and decided to see how aggressive we could get with standard quant tricks to create a budget-friendly "Mini" build. For the lulz, naturally.…

11
r/LocalLLaMA community 9d ago

White House AI Guidelines Exempt U.S. Open Models From Government Review

  submitted by   /u/realmvp77 [link]   [comments]

11
r/LocalLLaMA community 9d ago

GPT-OSS has turned one year old today!

It is one of the best local models ever released, in both 20B and 120B versions. I always come back to it, especially the 120B version. Its only competition is, in my opinion, Qwen 3.5 122B, but that model is much slower (A10B) and has not been released in a local-friendly QAT…

9
r/LocalLLaMA community 9d ago

A 2.6B model with tool calling and 128K context now runs at 30 tok/s on a phone

Liquid AI released LFM2.5-2.6B today, and this might be more relevant to local AI than another massive model most people cannot run. The model is only 2.69B parameters, has 128K context, supports tool calling and was post-trained specifically for multi-step agent workflows. The…

23
r/LocalLLaMA community 9d ago

Local LLM 35B MoE — Real-world coding benchmarks (Qwen vs Ornith vs KAT)

I’ve been running a fairly opinionated evaluation loop on ~35B A3B/MoE-class models for coding over the past few months. Not synthetic benchmarks: actual dev workflows, iterative debugging, refactoring passes, and failure recovery. Here’s where things stand for me: Qwen 3.6 (35B…

9
r/LocalLLaMA community 9d ago

Kimi K3 full model running on 16x GB10 cluster at 20+tps

Kimi K3 full model running on 16x GB10 cluster at 20+tps average (llama-benchy coherent corpus) 38tps peak, 750tps prefill. This is the first run of full k3 with dspark on my cluster. I will be doing some tests and try tp speed this up. As soon as it looks ready I'll publish the…

13
r/LocalLLaMA community 9d ago

Hugging Face CEO says China is winning the AI race and dominating on open models

This is something that was spoken here and there, and now it is like writing on the wall. The main additional point is that China has created an independent supply chain. Starting from raw materials and home-made lithography equipment, through their own GPU manufacturing, and to…

28
r/LocalLLaMA community 9d ago

Introducing Shieldstral. | Mistral AI

  submitted by   /u/tengo_harambe [link]   [comments]

28
r/LocalLLaMA community 9d ago

Has anyone tried Mach-1 Additive? 95% of performance of Qwen 3.6 35B while being 10x smaller

Why nobody is talking about this? Seems pretty significant to the community   submitted by   /u/MuzafferMahi [link]   [comments]

6
r/LocalLLaMA community 9d ago

A llama.cpp PR caches “hot” MoE experts on the GPU — 33 → 56 tok/s reported with 8GB VRAM

A new llama.cpp PR (#26563) adds a heatmap that tracks which MoE experts are used most often. Instead of keeping every expert on the GPU or offloading all of them, it caches the frequently selected experts in VRAM while the cold experts continue running on the CPU. The author’s…

29
r/LocalLLaMA community 9d ago

No more SLM open-source??

https://x.com/xiong_hui_chen/status/2084695353346117760?s=20   submitted by   /u/Capital-Remove-6150 [link]   [comments]

38
r/LocalLLaMA community 9d ago

LFM2.5-2.6B is out

Released today, with emphasis on agentic capabilities. I really like their models for simple, high volume tasks ("summarize these gazillion documents") and their 8b-a1b was my go-to for certain tasks so I'm excited to see how this one performs. There's not enough love for tiny…

18
r/LocalLLaMA community 9d ago

Deepseek V4 flash 0731 ranks #21 on Agent Arena

https://preview.redd.it/522fsdwvtdhh1.png?width=1200&format=png&auto=webp&s=6a6cf7a467514167a8193029dbd20fb3a9ba4f6c It ranks lower than both Sonnet 4.6 and Luna. I'd wager Luna costs in the same ballpark as DS4F considering Luna’s token efficiency. DeepSeek being open source is…

30
r/LocalLLaMA community 9d ago

Design systems from code alone - Without external images, Ling-3.0-flash generated webpages across Bauhaus, Bohemian, acid design, and more—using CSS gradients, SVG paths, typography, and layout to preserve each visual language.

Weights went up today so this is downloadable now, MIT, ~128GB for the official FP8. I ran these on the API before that landed, so treat it as a preview of what you'd be pulling rather than a local benchmark   submitted by   /u/AcanthisittaOk1699 [link]   [comments]

32
r/LocalLLaMA community 9d ago

Gemma 4 on 500MB

https://x.com/i/status/2084656348617392261 https://www.reddit.com/r/LLMDevs/s/9oL5ogmE6s   submitted by   /u/jacek2023 [link]   [comments]

12
r/LocalLLaMA community 9d ago

Decrease the power limit of your 5090 to at least 480W - the performance penalty for inference is negligible.

I run my inference machine in the living room, so noise and heat output are a significant concern. Ran a quick test using my daily driver model (Qwen 3.6-27b) and at 480W, the card outputs only 2.1% less t/s in decode and 8.8% in prefill (which is already very fast). Well worth…

6
r/LocalLLaMA community 9d ago

DeepSeek V4 Flash 0731 (Q4) now reaches 1,328 tok/s prefill and ~29 tok/s decode on one RTX PRO 6000

I've been working on speeding up DeepSeek-V4-Flash-0731 in Krasis and have now got the long-prompt prefill quite a bit faster on a single RTX PRO 6000 96GB. These are timing-disabled internal Krasis results using INT4 experts. They aren't HTTP round-trip speeds: Prompt size…

15
r/LocalLLaMA community 9d ago

inclusionAI/Ling-3.0-flash weights are up on Hugging Face — MIT, BF16 plus an official FP8

Went public in the last few minutes, both repos ungated. Ling-3.0-flash, BF16, 24 shards, ~255GB Ling-3.0-flash-fp8, official FP8, ~128GB 127.5B total, they quote 5.1B active. What jumped out at me in config.json is 512 experts with 8 active per token, which is a lot…

17
r/LocalLLaMA community 9d ago

inclusionAI/Ling-3.0-flash · Hugging Face

The Ling-3.0-flash MoE is now open-weighted at 124B A5B params. I know the original announcements were before the Kimi K3, DeepSeek-V4-Flash and Qwen3.8 hype, but this model might still have a good niche for itself due to its sizing. Discussion on the benchmarks are here:…

12
r/LocalLLaMA community 9d ago

Deepseek V4 Flash 2-bit quant is the first model I can run locally that achieves 100% in this SQL benchmark

I really like to use this one SQL benchmark when testing new models. I had another post some time ago with my benchmarks, but I decided to post a new one because of how well Deepseek did. I like the benchmark because it's quick to run, is pretty "real-world" and requires good…

37
r/LocalLLaMA community 10d ago

[Deepseek-V4-Flash-0731] Full 1M context on a single RTX5090 + DDR5 Desktop Setup with VLLM CPU/Ram Offloading, ~800 tps pp & 15+ tps decode [Agentic Coding]

First of all, obviously I took some help from AI to type this post and this is the topic that enabled me to accomplish all that: https://old.reddit.com/r/LocalLLaMA/comments/1veow4b/deepseek_v4flash_284b_moe_at_33_toks_single_68/ This post of mine is based on the link above. My…

11
r/LocalLLaMA community 10d ago

I built a DwarfStar-inspired Vulkan/Metal inference engine for Qwen3.6-35B-A3B on 16 GB machines

Disclosure: I’m the author and maintainer of QuarkStar. I built QuarkStar , a small native inference engine inspired by Antirez’s DwarfStar. QuarkStar currently supports: Qwen3.6-35B-A3B , using the same Antirez-inspired Q2 and Q2/Q4 quantization recipes KAT-Coder-V2.5-Dev , the…

6
r/LocalLLaMA community 10d ago

SK hynix, In Collaboration With SanDisk, Unveils The New High Bandwidth Flash (HBF) Standard, Helping To Resolve AI Inference Bottlenecks, Targeting Up To 3TB/s Bandwidth

Hopefully this would let us have faster local models....but it will probably be out of our price range.   submitted by   /u/giveen [link]   [comments]

12
r/LocalLLaMA community 10d ago

Llama.cpp PR 8% speed boost

Llama.cpp currently uses cpu based sampling for user with mtp enabled. The PR moves sampling to the gpu, which on a 5090 boasts an 8% increase in tok/s for qwen3.6:35b. I tested it on my P40 and observed a 4% increase inference speed boost. Pretty exciting to see 84 tok/s max on…

19
r/LocalLLaMA community 10d ago

DeepSeek V4 Flash 0731GGUFs with updated template (supports reasoning levels)

  submitted by   /u/tarruda [link]   [comments]

10
r/LocalLLaMA community 10d ago

Why are Chinese models better* at Frontend than the western top labs?

I use A LOT both openAI and Anthropic products. When I need some frontend work (pure web dev) (or answer that feel less verbose and more to the point) I use Anthropic. For multimodality openAI feels better (understanding audio, screenshots, generating images, etc). But openAI…

17
r/LocalLLaMA community 10d ago

Optimised DSv4-Flash for 2x GH200: 10,000 tok/s PP, >300 tok/s TG on SGLang

There are some PRs to use and a nice trick to speed up PP on really longs contexts in my write up. Hope it helps! TL;DR: On this dual GH200 box, you build vLLM v0.26.0 from source, add the merged DSV4 cache-layout patch (PR #48993), disable async scheduling, and run DSpark at 6…

25
r/LocalLLaMA community 10d ago

Time to finally migrate from LM Studio -> llama.cpp, your experience?

Has anyone moved from LM Studio to llama.cpp? What was your experience like? What did you have to learn in order to recreate your experience? Which harness/GUI did you switch to? Thanks in advance!   submitted by   /u/CSEliot [link]   [comments]

19
r/LocalLLaMA community 10d ago

Is LM Studio abandoning their core product?

Some of you may be aware that a few weeks ago, LM Studio announced a new agent, Bionic. This is pretty much an agentic harness for both local models and paid cloud models. But most aren't aware that LM Studio replaced almost every link to the original app that built their brand…

7
r/LocalLLaMA community 10d ago

can someone create a website where people share specific hardware specs with specific llama cpp flags so we see what works?

title   submitted by   /u/Rank201AltAccount [link]   [comments]

20
r/LocalLLaMA community 10d ago

More Qwen 3.8 sizes coming

  submitted by   /u/appakaradi [link]   [comments]

14
r/LocalLLaMA community 10d ago

NousResearch keeps doing things on hermes

Has anyone followed nousresearch work on Hermes? I mean we are Q3 2026. We have some crazy models trickling down from HGX territory to multi gpu workstation. And we have nousresearch deploying the 0.20 of its hermes agent while starting releasing the project with a 0.2 mid…

13
r/LocalLLaMA community 10d ago

nvidia/NVIDIA-NemotronLabs-VoiceChat-11B · Hugging Face (full duplex)

  submitted by   /u/adefa [link]   [comments]

9
r/LocalLLaMA community 10d ago

Only 3 days ago...

  submitted by   /u/Fun_Librarian_7699 [link]   [comments]

22
r/LocalLLaMA community 10d ago

G9v3-39A5B: Agentic heavy MOE with low hallucination

Hugging Face Artificial Analysis Should be a sweet spot for general work. Seems like coding is the only part that is inferior to Qwen.   submitted by   /u/axseem [link]   [comments]

16
r/LocalLLaMA community 10d ago

Ling-3.0-flash is another potential model to test before qwen3.8 27b

I tested Ling-3.0-flash with hard bugs and it fixed bugs that qwen3.6-27b could not. This models speed faster than deepseek v4 flash but almost the same level as (old) deepseek v4 flash. Note: hard bugs mean they don't have "error messages" but they are unexpected behaviors of a…

13

MiniMax issues

Qwen Developers' responses from their recent Twitter/X AMA

I updated my localy run benchmark with DeepSeek V4 Flash 0731

Anyone interested in building a harness-only benchmark?

Building a Fully Local PDF Read-Aloud & PDF-to-Audiobook Desktop App with Kokoro 82M, Qwen, and llama.cpp

Qwen3-TTS voice cloning is now in mainline llama.cpp — the old demo finally became real support

Intern S2 Mobius

GPT-X2.5-135M scores 3rd place on Open SLM Leaderboard on Huggingface, Beating Facebook's MobileLLM-R1-140M

China’s Open-Weight Models Will Be Spared US Safety Tests

VibeVoice 1.5B Running Locally...On an iPhone! Only ~2.2 GB of Memory and Up to 1.28× Real-Time Speed

Thinking of buying more DRAM right now...

PSA Update CUDA from 13.2 to 13.3 to solve DeepSeek V4 Flash 0731 Looping Problem!

Maple-Preview: 20B-A1B ternary-weight reasoning open-weight LLM

DeepSeek-V4-Flash on SM89 4x48gb 4090s with DSpark

DeepSeek-v4-Flash-Mini 54GB GGUF running at ~20.5 t/s

White House AI Guidelines Exempt U.S. Open Models From Government Review

GPT-OSS has turned one year old today!

A 2.6B model with tool calling and 128K context now runs at 30 tok/s on a phone

Local LLM 35B MoE — Real-world coding benchmarks (Qwen vs Ornith vs KAT)

Kimi K3 full model running on 16x GB10 cluster at 20+tps

Hugging Face CEO says China is winning the AI race and dominating on open models

Introducing Shieldstral. | Mistral AI

Has anyone tried Mach-1 Additive? 95% of performance of Qwen 3.6 35B while being 10x smaller

A llama.cpp PR caches “hot” MoE experts on the GPU — 33 → 56 tok/s reported with 8GB VRAM

No more SLM open-source??

LFM2.5-2.6B is out

Deepseek V4 flash 0731 ranks #21 on Agent Arena

Design systems from code alone - Without external images, Ling-3.0-flash generated webpages across Bauhaus, Bohemian, acid design, and more—using CSS gradients, SVG paths, typography, and layout to preserve each visual language.

Gemma 4 on 500MB

Decrease the power limit of your 5090 to at least 480W - the performance penalty for inference is negligible.

DeepSeek V4 Flash 0731 (Q4) now reaches 1,328 tok/s prefill and ~29 tok/s decode on one RTX PRO 6000

inclusionAI/Ling-3.0-flash weights are up on Hugging Face — MIT, BF16 plus an official FP8

inclusionAI/Ling-3.0-flash · Hugging Face

Deepseek V4 Flash 2-bit quant is the first model I can run locally that achieves 100% in this SQL benchmark

[Deepseek-V4-Flash-0731] Full 1M context on a single RTX5090 + DDR5 Desktop Setup with VLLM CPU/Ram Offloading, ~800 tps pp & 15+ tps decode [Agentic Coding]

I built a DwarfStar-inspired Vulkan/Metal inference engine for Qwen3.6-35B-A3B on 16 GB machines

SK hynix, In Collaboration With SanDisk, Unveils The New High Bandwidth Flash (HBF) Standard, Helping To Resolve AI Inference Bottlenecks, Targeting Up To 3TB/s Bandwidth

Llama.cpp PR 8% speed boost

DeepSeek V4 Flash 0731GGUFs with updated template (supports reasoning levels)

Why are Chinese models better* at Frontend than the western top labs?

Optimised DSv4-Flash for 2x GH200: 10,000 tok/s PP, >300 tok/s TG on SGLang

Time to finally migrate from LM Studio -> llama.cpp, your experience?

Is LM Studio abandoning their core product?

can someone create a website where people share specific hardware specs with specific llama cpp flags so we see what works?

More Qwen 3.8 sizes coming

NousResearch keeps doing things on hermes

nvidia/NVIDIA-NemotronLabs-VoiceChat-11B · Hugging Face (full duplex)

Only 3 days ago...

G9v3-39A5B: Agentic heavy MOE with low hallucination

Ling-3.0-flash is another potential model to test before qwen3.8 27b