r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 7d ago

nvidias nemotron omni only loads its text half on a mac, so i wrote the vision and audio towers in mlx

nvidias nemotron omni is open weights and it sees, hears and reasons. theres already a 4bit mlx quant on hugging face but only the text backbone loads with standard mlx tooling. the model card says it plainly, the vision and audio towers need a runtime that implements the…

20
r/LocalLLaMA community 7d ago

Best open-source harnesses for combining cloud and local AI model orchestration?

Looking for best current solutions for combining cloud models and local models seamlessly inside a harness' orchestration Edit: Right now, we don't have harnesses (that I'm aware of) that are blending local and cloud models to work together simultaneously to accomplish tasks set…

32
r/LocalLLaMA community 7d ago

KV cache quantization benchmarks: 413 pairs tested on Qwen 3.6 27B, Gemma 4 31B. KLD with BeeLlama.cpp v0.4.0: KVarN 6-bit beats q8_0, precision tail 1024 dominates

Link to the article: KV Cache Quantization Benchmarks: KVarN, Precision Tail KLD benchmarks with BeeLlama.cpp v0.4.0 , fork of llama.cpp with more KV cache quantization options. Models: Qwen 3.6 27B Q5_K_S 64k context, Gemma 4 31B Q5_K_S 16k context Standard quants, extended:…

9
r/LocalLLaMA community 7d ago

I ported vLLM's serving stack to C++20: 66 MiB binary, no Python at inference, output checked token-for-token against vLLM

I'm the author, so discount the enthusiasm accordingly. This is an unaffiliated community port, not endorsed by the vLLM project, which it uses to verify its correctness. What started it: I love vLLM, but a vLLM install here is 9.1 GiB of virtualenv, and I wanted to embed…

36
r/LocalLLaMA community 7d ago

AI clickbait

Reading through this subreddit and many more I keep running into what I am calling "AI click bait". Either projects that seems interesting in the description/title but when you open them they're the same AI vide coded slop that does not solve the problem; or apparent discussions…

11
r/LocalLLaMA community 7d ago

2 x 5070ti Qwen 27B full config / stats

Following up on yesterday's post about running everyone's faves on 2 x 16gb cards while maximizing performance and KV. Previous post data used abandoned Cu130 VLLM image. Stats here are done on cu129-nightly. Which has the KV cache connector fixes and performance improvements.…

34
r/LocalLLaMA community 7d ago

nvidia/NVIDIA-Nemotron-Parse-2.0 · Hugging Face

NVIDIA Nemotron Parse 2.0 transforms document images into structured, machine-readable representations with text, layout classes, bounding boxes, and reading-order information. Given a Red, Green, Blue (RGB) document image and a task prompt, the model produces formatted text and…

38
r/LocalLLaMA community 7d ago

I compared even more parsers on 14 PDF-parsing capabilities using different types

In a previous post , I compared MinerU, Granite-Docling, and PaddleOCR-VL. Many commentors suggested I added their favorite parsers. So I did. And also added some new capabilities to differentiate the top models. Here is the full list of parser compared: MinerU 2.5 (1.2B VLM)…

9
r/LocalLLaMA community 7d ago

endless-frontier/BigBang-v1 (35B-A3B)

*Not affiliated with the project, just sharing new model I found   submitted by   /u/AaronFeng47 [link]   [comments]

22
r/LocalLLaMA community 7d ago

32 total local models tested head to head

I ran 32 local models head to head on one fact-extraction corpus, 1,001 notes, paired bootstrap on every adjacent pair. Several weeks of compute time, all on consumer grade cards. Most of the field does not separate. Six consecutive steps from 2B to 31B, and the bootstrap cannot…

37
r/LocalLLaMA community 8d ago

How come artificialanalysis.ai ranks Gemma4 above Qwen3.6 27b in SciCode

Just came across this coding benchmark: SciCode Artificialanalysis.ai reports a ranking which contradicts the feeling we've towards those models in real life coding. Is Gemma 4 really that good, or a benchmarking issue? EDIT: The contribution of this benchmark to the…

28
r/LocalLLaMA community 8d ago

The death of SLMs?

I love to see these impressive models coming out that compete with the giants from companies like Z.ai, Moonshot, Alibaba, etc. A win for the open source/weight community is always welcome. While I am grateful, I worry we might be seeing the slow death of models smaller than…

29
r/LocalLLaMA community 8d ago

Knowledge vs. hallucination rate: what is your favorite model?

https://preview.redd.it/h7dwy09z2rhh1.png?width=1240&format=png&auto=webp&s=7285448ac2d9c895c9a11062557044fc97504fa3 I was evaluating which model to test for text summarization and rewriting in a scenario that also requires general world knowledge. Taking into account the…

33
r/LocalLLaMA community 8d ago

They almost catched up on Frontier performance, so now catching up on prices

Users also report that the free version was significantly downgraded after the release of the new models this is very important for us when considering local hosting. A lot of people decided not to buy expensive hardware because DeepSeek’s prices made it very difficult to break…

25
r/LocalLLaMA community 8d ago

Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090)

TL;DR: On a Qwen3.6-35B-A3B Q6 setup sized for 64K context on a 24GB RTX 3090, spilling eight MoE expert layers to CPU freed enough VRAM to increase -b from 512 to 1024 and -ub from 128 to 512. Prompt processing improved by 2.36×, while generation speed remained unchanged within…

9
r/LocalLLaMA community 8d ago

Final optimization: from ~10 tok/s to ~15 tok/s on DeepSeek-V4-Flash-0731 at 128K ctx - 1 RTX 3090

J'ai consacré beaucoup de temps à l'optimisation de DeepSeek-V4-Flash-0731 GGUF sur une seule RTX 3090. Mon exigence absolue pour chaque configuration était la suivante : Le modèle doit rester utilisable avec une fenêtre de contexte de 128 000 jetons. J'ai testé les différentes…

4
r/LocalLLaMA community 8d ago

Best llama cpp flags to run Deepseek-flash 0731

Hi all. These are my system specs: dual xeon e5 2696 v2 , 160gb DDR3 ram ECC(1600mhz), 3 gpus: 3060 12gb, p100 16gb, 3050 6gb. And a 400gb nvme sdd RAID0, 3000 mb/s. The model is Deepseek-flash-0731 UD_8_X_XL, loseless, 161gb. Now, I'm not too knowledgeable about llama cpp…

12
r/LocalLLaMA community 8d ago

i just spent weeks rewriting my webUI from scratch, getting rid of all AI slop within the codebase and switching it over to a proper lightweight framework (alpine.js). i am now comfortable suggesting it as an alternative to openwebUI, librechat and the like! it is made for local…

[Fully open source under GPL3, made from the ground up for use with local models, no subscriptions, no corporate backing] When i first started this, it was meant to be a fully lightweight, extremely modular alternative to openclaw, hermes and the like , and it still is! But i…

19
r/LocalLLaMA community 8d ago

Qwen3.8-2.4T-A95B (aka Qwen3.8-Max) open release time: next wednesday

https://preview.redd.it/9zwlpcushphh1.png?width=972&format=png&auto=webp&s=18cb49c738caba9799d22d4916337c772baa830f https://modelscope.cn/models/Qwen/Qwen3.8-2.4T-A95B   submitted by   /u/HugeConsideration211 [link]   [comments]

17
r/LocalLLaMA community 8d ago

I get that AI labs need to make money, but zero-warning price spikes are a nightmare for production builds

Seen a ton of posts today about the DeepSeek API price hike. Half the feed is doom-posting, the other half is explaining basic GPU economics. Honestly, I get the cost side. Sub-cent tokens were never gonna last forever. But what actually sucks is the zero-day notice. Dropping a…

23
r/LocalLLaMA community 8d ago

GLM/Qwen Appreciation Post

https://preview.redd.it/o6ik6qboeohh1.png?width=1134&format=png&auto=webp&s=4016f26c50c1d93bd3d0c7e880e9b55a2d75310f I have been running Qwen3.6 27b for a little while (mostly coding tasks) and recently trying out V4 flash 0731 in it's place. It was very apparent the new v4…

29
r/LocalLLaMA community 8d ago

How many people in this sub try to train their own AI from scratch on their systems just for fun and to test out techniques from research papers?

As for me, I own a system with an RTX 5090, Ryzen 9 9950X3D2, and 64 GB of DDR5. Every time I see research come out with a new way to train AI, I immediately think to try it on my system to see the results I get. Applying things like Titans, that one Deepseek paper on engrams,…

26
r/LocalLLaMA community 8d ago

Introducing BetterBench - more accurate PP and TPS measurement

I built this because the existing benchmarks were using random data and with MTP content types can vary a lot on what performance you see. 5% or more with content types. BetterBench is designed to have content consistency within 1% and also measures across different content…

8
r/LocalLLaMA community 8d ago

Get AI max+ 395 laptop or wait for rtx spark?

So I can either pull the trigger on a 128gb AI max+ 395 laptop or wait for RTX Spark for LLMs. Maybe I get it now and the price of the spark is super high so it's a good purchase or maybe the Spark shocks everyone with a low price and I forever regret my purchasing decision.…

37
r/LocalLLaMA community 8d ago

Zuck will "share more on open source" soon

  submitted by   /u/realmvp77 [link]   [comments]

16
r/LocalLLaMA community 8d ago

Prime Agent - a new coding harness surpassing Codex/CC/PI

Prime Agent is an open-source coding and research agent for general and long-running work. A self-improving RLM harness for coding and long-running autonomous tasks. Designed to be both token-efficient and expressive through programmatic tool calling, context as a variable,…

18
r/LocalLLaMA community 8d ago

I remember a time when 'flash' meant 32B

I mean, Deepseek V4 Flash is an absolutely fantastic model, even though I can't run it on my machine it's so fascinating to see how it performs. Knowing that potentially it could be run at home is really motivating and makes me hopeful that those capabilities will trickle down…

6
r/LocalLLaMA community 8d ago

Meta Model, Muse Spark 1.1 Hacked Another Company During Cybersecurity Testing, Breaching Systems and Making Changes to Internal Systems - The Information

  submitted by   /u/pscoutou [link]   [comments]

5
r/LocalLLaMA community 8d ago

Could we have a --disk-moe or --n-disk-moe like --cpu-moe or --n-cpu-moe so we can use disk/cpu/gpu ?

Explicit title, It would be nice to have the ability to have 3 tiers moe offload :(   submitted by   /u/storm1er [link]   [comments]

38
r/LocalLLaMA community 8d ago

Deepseek V4 Flash just hit Colibri, does anyone have numbers?

I'm mosty interested in 128-192GB VRAM with 128-256GB RAM to spare, so SSD streaming is basically not even necessary. Seems only FP4 is supported, so older hardware will likely be slow - no Unsloth GGUF supported either. I'd be curious what people are getting with V100s, R9700s,…

8
r/LocalLLaMA community 8d ago

you can now buy llm's at your local supermarket

  submitted by   /u/ECrispy [link]   [comments]

14
r/LocalLLaMA community 8d ago

Xiaomi-Robotics-1: New robotics model released

Xiaomi-Robotics-1 is a robot foundation model trained on over 100K hours of real-world manipulation trajectories. It is a Vision-Language-Action (VLA) model engineered for out-of-the-box mobile manipulation in unseen environments and efficient adaptation to new tasks. XR-1…

14
r/LocalLLaMA community 8d ago

40% speedup of MoE training with faster megakernel, by cursor, of all people (for B200s)

daily reminder not to trust benchmarks and run it yourself. claimed e2e speedup is ~40%, forwards are ~140% faster I would wager that compared to a naive kernel anyone can write it's more in the range of 10-20% faster e2e in reality, if at all, but hey, it's free and open!…

8
r/LocalLLaMA community 8d ago

bootai

I was on here a while ago showcasing it. I've stoped playing with it so I'm open sourcing it. I figure I'll let other people play now.   submitted by   /u/Electrical_Ninja3805 [link]   [comments]

27
r/LocalLLaMA community 8d ago

Inkling-Small 276B-A12B at ~2.9 tok/s on <10gb memory

A follow up to the launch of Mference , it now supports and runs Inkling-Small 276B-A12B . Inkling-Small (Thinking Machines, Apache 2.0), from the pipenetwork/Inkling-Small-MLX-4bit conversion: 276B total, ~12B active, 3.4 GB resident set , ~148 GB on disk. Measured on my M5,…

6
r/LocalLLaMA community 8d ago

Scenema Audio Comes to ComfyUI, Runs on 8GB VRAM

Hey everyone! Scenema Audio is now a native ComfyUI custom node. Same model that powers scenema.ai now quantized so it fits on 8GB VRAM. When we first released it a few months ago as an API and Docker stack, the full precision transformers were too heavy for most people to…

6
r/LocalLLaMA community 8d ago

Gemma 4 31b AttnRes Project

I had Claude re-draft this for me, thus it has Em Dashes. It's correct with lots of "Claude" simplifications. --- Hey all. It's been a while since I posted about the AttnRes architecture so I figured I'd give an update on where things are. Short version: it's alive. Longer…

11
r/LocalLLaMA community 8d ago

Mistral Releases premier Not-Hotdog model

https://huggingface.co/mistralai/Shieldstral-1.0-3B   submitted by   /u/rpiguy9907 [link]   [comments]

35
r/LocalLLaMA community 8d ago

DeepSeek V4 Flash 0731 at 10–17 t/s (nothink) on MacBook M5 Pro **64GB***, partly via SSD streaming

Inspired by a post from u/giveen I motivated claude (no patinence on my side to work through everything myself) to help me get DS running on my MacBook M5 Pro 64GB and it exceeded my expectations.. because it worked, and at a quite usable generation speed! background: antirez…

12
r/LocalLLaMA community 8d ago

Agent memory layers don't need an LLM deciding what to remember

Most agent memory setups run a model call on the way in. Something reads the turn, decides whether it's worth keeping, rewrites it into a "memory", tags it with a type and an importance score. That's a second inference pass on every interaction and I think it's the wrong place…

26
r/LocalLLaMA community 8d ago

Ling-3.0-flash MXFP4 released and running locally on one DGX Spark.

In tests: ~80 tok/s decoding 2,500–3,500 tok/s long-input prefilling Smooth use by 3–4 concurrent users Private, on-device inference for coding, agents, and offline batch jobs   submitted by   /u/niacolhealth [link]   [comments]

30
r/LocalLLaMA community 8d ago

The Session You Cannot Take With You | EARENDIL

  submitted by   /u/MoneyPowerNexis [link]   [comments]

12
r/LocalLLaMA community 8d ago

LFM2.5-2.6B on a OnePlus 13 at 17 tok/s ~ Pure CPU

As you all know the model is 2.69B parameters with a 128K context window and purpose-built for multi-step agent workflows. What you are seeing is the Q4_K_M GGUF running on my own inference engine built from scratch. The TUI is my own device probe suite running through ADB…

27
r/LocalLLaMA community 8d ago

DGX Spark now sells for 6000-8000 euros. I still remember when it was just 4000.

  submitted by   /u/Afraid-Yoghurt6731 [link]   [comments]

7
r/LocalLLaMA community 8d ago

Given the MiniMax H3 LoRAs Debacle - Some Important Context for Censorship enforcement and laws in China

*I felt the need to write this post because it seems like very few people on this sub are aware of Chinese laws and how they're enforced, so here's an explainer coming from a Chinese person (myself). I know that this post isn't directly about local models per se, but I'm seeing…

12
r/LocalLLaMA community 9d ago

jabbatheduck/DeepSeek-v4-flash-mini · Hugging Face

Because why not? How far can we go and make DeepSeek work?   submitted by   /u/giveen [link]   [comments]

6
r/LocalLLaMA community 9d ago

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature: Mixture-of-Experts CPU offload: --n-cpu-moe <N> | -ncmoe <N> Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on the CPU;…

29
r/LocalLLaMA community 9d ago

MiniMax issues

https://www.reddit.com/r/StableDiffusion/s/HrU7odaJe6 I think this is more important that all the political stuff you share here   submitted by   /u/jacek2023 [link]   [comments]

12
r/LocalLLaMA community 9d ago

Qwen Developers' responses from their recent Twitter/X AMA

Questions & Responses(in BOLD ) below. Favorite question(s) moved to end of the thread with combined responses(removed duplicates). Be optimistic folks. I'm sure we're getting other models too apart from 27B. And 27B gonna make massive noise on release. (Based on their…

11
r/LocalLLaMA community 9d ago

I updated my localy run benchmark with DeepSeek V4 Flash 0731

It's the purple cluster on the top left (the good corner...) I'm running the MXFP4 version from Bartoswski with Dspark at 1K t/s prefill and 90 t/s gen (average). I tried different sampling params, you can check the detail. It's very efficient while scoring the best yet. Too bad…

12

nvidias nemotron omni only loads its text half on a mac, so i wrote the vision and audio towers in mlx

Best open-source harnesses for combining cloud and local AI model orchestration?

KV cache quantization benchmarks: 413 pairs tested on Qwen 3.6 27B, Gemma 4 31B. KLD with BeeLlama.cpp v0.4.0: KVarN 6-bit beats q8_0, precision tail 1024 dominates

I ported vLLM's serving stack to C++20: 66 MiB binary, no Python at inference, output checked token-for-token against vLLM

AI clickbait

2 x 5070ti Qwen 27B full config / stats

nvidia/NVIDIA-Nemotron-Parse-2.0 · Hugging Face

I compared even more parsers on 14 PDF-parsing capabilities using different types

endless-frontier/BigBang-v1 (35B-A3B)

32 total local models tested head to head

How come artificialanalysis.ai ranks Gemma4 above Qwen3.6 27b in SciCode

The death of SLMs?

Knowledge vs. hallucination rate: what is your favorite model?

They almost catched up on Frontier performance, so now catching up on prices

Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090)

Final optimization: from ~10 tok/s to ~15 tok/s on DeepSeek-V4-Flash-0731 at 128K ctx - 1 RTX 3090

Best llama cpp flags to run Deepseek-flash 0731

i just spent weeks rewriting my webUI from scratch, getting rid of all AI slop within the codebase and switching it over to a proper lightweight framework (alpine.js). i am now comfortable suggesting it as an alternative to openwebUI, librechat and the like! it is made for local…

Qwen3.8-2.4T-A95B (aka Qwen3.8-Max) open release time: next wednesday

I get that AI labs need to make money, but zero-warning price spikes are a nightmare for production builds

GLM/Qwen Appreciation Post

How many people in this sub try to train their own AI from scratch on their systems just for fun and to test out techniques from research papers?

Introducing BetterBench - more accurate PP and TPS measurement

Get AI max+ 395 laptop or wait for rtx spark?

Zuck will "share more on open source" soon

Prime Agent - a new coding harness surpassing Codex/CC/PI

I remember a time when 'flash' meant 32B

Meta Model, Muse Spark 1.1 Hacked Another Company During Cybersecurity Testing, Breaching Systems and Making Changes to Internal Systems - The Information

Could we have a --disk-moe or --n-disk-moe like --cpu-moe or --n-cpu-moe so we can use disk/cpu/gpu ?

Deepseek V4 Flash just hit Colibri, does anyone have numbers?

you can now buy llm's at your local supermarket

Xiaomi-Robotics-1: New robotics model released

40% speedup of MoE training with faster megakernel, by cursor, of all people (for B200s)

bootai

Inkling-Small 276B-A12B at ~2.9 tok/s on <10gb memory

Scenema Audio Comes to ComfyUI, Runs on 8GB VRAM

Gemma 4 31b AttnRes Project

Mistral Releases premier Not-Hotdog model

DeepSeek V4 Flash 0731 at 10–17 t/s (nothink) on MacBook M5 Pro **64GB***, partly via SSD streaming

Agent memory layers don't need an LLM deciding what to remember

Ling-3.0-flash MXFP4 released and running locally on one DGX Spark.

The Session You Cannot Take With You | EARENDIL

LFM2.5-2.6B on a OnePlus 13 at 17 tok/s ~ Pure CPU

DGX Spark now sells for 6000-8000 euros. I still remember when it was just 4000.

Given the MiniMax H3 LoRAs Debacle - Some Important Context for Censorship enforcement and laws in China

jabbatheduck/DeepSeek-v4-flash-mini · Hugging Face

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

MiniMax issues

Qwen Developers' responses from their recent Twitter/X AMA

I updated my localy run benchmark with DeepSeek V4 Flash 0731

DeepSeek V4 Flash 0731 at 10–17 t/s (nothink) on MacBook M5 Pro 64GB*, partly via SSD streaming