r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 6d ago

Tmax-27b - a Qwen3.6-27b terminal agent for small GPUs trained with DPPO (RL)

Hey everyone, wanted to share some work on making the new Tmax-27B terminal agent actually runnable on consumer hardware. What is Tmax-27B? Ai2 just released Tmax, a family of terminal-agent LLMs trained with DPPO (RL) on top of Qwen3.6. The 27B model hits ~43% on Terminal Bench…

32
r/LocalLLaMA community 6d ago

OpenMythos benchmarks

Hey everyone! OpenMythos benchmarks are finally here sorry it took about a week to post these. The delay was mainly because SWE-bench results weren't matching up with Qwen 3.6 27B official numbers. Turns out Qwen used a different eval harness and also refined/filtered the…

12
r/LocalLLaMA community 6d ago

I'm eager for a 15x speedup on my strix halo

Nvidia says 15x speed up possible with diffusion model. Entire block of text generated at once. https://x.com/NVIDIAAI/status/2069465510790545761   submitted by   /u/Terminator857 [link]   [comments]

22
r/LocalLLaMA community 6d ago

650+ Apache-2.0 biomedical NER/de-id models that run on-device in MLX. Same fp32 weights, identical outputs: the clinical NER models run 30-40x faster than PyTorch-CPU on a 3-year-old M3 Max. Repro inside.

Disclosure first: I maintain OpenMed, so read this with that bias. I'm posting the numbers with the full methodology and a runnable script so you can reproduce or tear it apart. I'm here for the next couple of hours to answer methodology questions. What it is: an open-source…

25
r/LocalLLaMA community 6d ago

UPDATE: Qwen-27B-IQ4_KS and Qwen-27B-IQ_KS_KT for ik_llama.cpp, especially for NVIDIA with 16GB VRAM

Continuing 16GB VRAM Optimizations: New Qwen3.6-27B GGUF Quants (Experimental Trellis/iq4_kt & MTP) Hi everyone, I'm continuing my optimization efforts for 16GB VRAM and Nvidia GPUs from this post:…

7
r/LocalLLaMA community 6d ago

Not ironclad confirmation, but..

Over here: https://huggingface.co/papers/2606.21906 Kudos to xyzblaz for asking.   submitted by   /u/Kodix [link]   [comments]

6
r/LocalLLaMA community 6d ago

Is it possible to run a giant model like GLM5.2 on this cluster (4x servers with 512GB RAM + dual AMD Epyc)? 16 channel memory should hit 409GB/s per node.

Hey all, I have a piece of hardware laying around which is pretty fast from a traditional (non-GPU) server viewpoint. The hardware is the following: Dell C6525 Server with Quad Node (4x server blades) with the following: 2x AMD EPYC 7702 64-Core Processors 8 memory channels per…

30
r/LocalLLaMA community 6d ago

Openrouter model prices implying heavier quantization?

Theres been a lot of talk about quiet quantization of models and what access to guaranteed model quality would look like. I’ve been trying to sanity check the economics of running large open models, and I’m having trouble making the numbers work. Take GLM-5.2 as an example. Even…

18
r/LocalLLaMA community 6d ago

GLM 5.2 on Mac Studio Speedup PR

Just a heads up for the lucky few 512 gb mac owners: GLM 5.2 is a game changer because prefill speeds stay above 100 t/s at much higher context, and also take less space, so we can run 4 bit quants well above 100k context. See this PR by the oMLX creator:…

5
r/LocalLLaMA community 6d ago

I benchmarked 8 LLMs for medical scribing. Hallucinations were rare; omissions need attention.

I ran a small benchmark on LLMs for medical scribing. Reason: most discussion around AI scribe safety focuses on hallucinations. That matters, but in notes I kept seeing another problem: models often leave out clinically relevant details from the conversation. So I evaluated 8…

10
r/LocalLLaMA community 6d ago

7 Chinese companies are already shipping H100/H200-class AI chips, most IPO'd in the last 6 months. I mapped all of them.

https://preview.redd.it/8e85oakbz19h1.png?width=3000&format=png&auto=webp&s=258039cb277daa2572d1ba4eec8cc488357c62d0 I run Chinese open models on a 4×3090 rig every day. The more I watched these models get tuned for domestic hardware, the more I wanted to know what that hardware…

17
r/LocalLLaMA community 6d ago

Krea 2 released on Hugging Face

+ turbo: https://huggingface.co/krea/Krea-2-Turbo   submitted by   /u/paf1138 [link]   [comments]

15
r/LocalLLaMA community 6d ago

I mapped the KLD of KV cache quantization for Qwen3.6-35B-A3B and Gemma4-E2B QAT

TL;DR version q8/q8 is nearly free on both models q4/q4 is useable on Qwen and catastrophic on Gemma turbo4 is sometimes slightly better, sometimes slightly worse, than q4_0 turbo3 and turbo2 allow compressing the cache to unprecedented levels - but you'll pay dearly for it K is…

9
r/LocalLLaMA community 6d ago

Baidu: One-shot Long-horizon Parsing

  submitted by   /u/zxyzyxz [link]   [comments]

34
r/LocalLLaMA community 6d ago

CPU-only TTS benchmark: Kokoro 82M vs Supertonic 3 vs Inflect-Nano-v1 (4.6M params), with UTMOS scoring on every sample

Ran three open-weight TTS models head to head on CPU. Intel Xeon, 4 cores, 15.6GB RAM, no GPU. Five configs, six text lengths from 12 to 1712 chars, 5 timed reps per cell after warmup, 150 timed runs total. Every audio output scored with UTMOS (utmos22_strong) so quality isn't…

19
r/LocalLLaMA community 6d ago

Pooled round robin hardware with friends?

I have a rig Friend1 has rig Friend2 has a rig Each rig idle 90% of the time With agentic, how could we round robin as a group? So when I load up, it checks if friends rigs are idle (vpn etc) and if idle farms out tasks. If I understand right, agents work in parallel, so this…

29
r/LocalLLaMA community 6d ago

V100 4-card AI large model, Tesla 128G server

using google translate edit that will cost for USD 3687.76 V100 128G Liquid-Cooled Graphics Card Dock, 360° Liquid Cooling for the Entire System.   submitted by   /u/MundanePercentage674 [link]   [comments]

37
r/LocalLLaMA community 6d ago

I love GLM 5.2's attitude! It is a nice refresher from those bootlicker doormats they are feeding us. Does that come from training datasets related to the local culture?

I have realised one thing I really like about GLM 5.2, apart from its capabilites and huge consistent context, is its attitude : It is direct, concise, no fluff (as one infamous model likes to say) It won't take shit It won't sugar coat its answers, and will not blindly agree…

7
r/LocalLLaMA community 7d ago

My local server idling 99% of the time!

Guys what you running to make agents busy? Like some crazy 24/7 tasks, or maybe some useful ideas on how to utilize local llm with some purpose/use? I personally running Qwen3.6-27B with owu and with pi for coding (little-coder) but as in title - it’s idling all the time…  …

33
r/LocalLLaMA community 7d ago

Human Evaluation of GLM-5.2

I've seen plenty of benchmarks that put GLM-5.2 below many of the closed source alternatives but at their heels. I thought to myself, next version GLM will totally be where the best frontiers are at now. The last few days I've been testing it on a real world project, and it's…

6
r/LocalLLaMA community 7d ago

Multi Tier MoE Caching

I've never seen much discussion around this, but it feels like where MoE inference is heading. The bulk of big models we use, GLM 5.2, Deepseek V4, Stepfun, Minimix are MoE meaning inference is run on a small subsection of the experts. Currently we scatter these experts over a…

9
r/LocalLLaMA community 7d ago

How do I prove that I don't collect data from my llm app?

Building an incognito llm chat app for hobby and fun. I don't want users to trust me that I don't log prompts. I want them to be able to verify it. I can't really go the TEE route as that is very hardware leaning and I don't have the resources I'm not sure if open-sourcing the…

10
r/LocalLLaMA community 7d ago

Training a Qwen 3.5 4B/9B agent for multi-tool use: SFT first or go directly to RL?

To train Qwen 3.5 4B or 9B for a custom multi-tool agent workflow and would appreciate guidance from people who have done this successfully. A few questions: SFT → RL or RL-only? - Is it still recommended to first do supervised fine-tuning (tool-calling traces, reasoning…

15
r/LocalLLaMA community 7d ago

Is there any reason for a lack of love for Gemma 4 26b?

The answer to most questions on here is Qwen3.6 27b or 35b and then Gemma4 31b (but lesser so as it doesn’t fit well on a solo 3090). Is there any reason why Gemma 4 26b moe isn’t mentioned more? I plan on using Qwen for my coding agents. But I’ve been building a Jarvis for…

20
r/LocalLLaMA community 7d ago

MiniMax-M3-EAGLE3-GGUF - Llama.cpp compatible MiniMax M3 EAGLE draft model!

Hi all! With a new PR for llama.cpp, MiniMax M3's EAGLE decoder by Inferact/MiniMax-M3-EAGLE3 has successfully been converted to GGUF and runs without issue! The HF repo has instructions for both merging in the PR and running the model. I tested this on a 2x3090 and 128GB DDR4…

13
r/LocalLLaMA community 7d ago

100+ t/s on Qwen3.6-27B Q8 across a 5090 + 3090 Ti — switching to tensor split-mode got me from 70 to 100+

Wanted to share a setup that's been working great for me. Running Qwen3.6-27B at Q8_0 across two GPUs (RTX 5090 + RTX 3090 Ti) and getting ~100 t/s. The big jump came from switching --split-mode to tensor . I was sitting at 70+ t/s on layer split before that. Tensor split keeps…

22
r/LocalLLaMA community 7d ago

Boogu Base, Turbo, Edit - open-source unified image generation and editing model series

Boogu-Image-0.1 is a competitive Apache-2.0 open-source unified image generation and editing model family , including Base , Turbo , Edit , and other variants that provide stable, practical capabilities for high-quality text-to-image generation, fast generation, image editing,…

22
r/LocalLLaMA community 7d ago

Eff U, Arc / B70 Customers. We got ours! -Your Sugar Baby, Intel

Since we're due a "Should I buy a B70" post, no. Just no. Am I hoping someone who doesn't read this will buy mine on eBay in the next few days? Hell, yes. Here's the deal... A B70 is a decent buy if you want 32GB of VRAM over nearly all other considerations. They run reasonably…

19
r/LocalLLaMA community 7d ago

EU AI Act requires TEXT from models and providers to be watermarked 2nd August onwards. Everyone here is affected, regardless where you live.

Anyone hate the cookie banners ? Those are absolutely nothing in comparison to what is about to come. The AI Act requires lots of things, many people know it requires every AI modified or generated audiofile to be metadata tagged and fingerprint-watermarked from August on (32M$…

9
r/LocalLLaMA community 7d ago

sk hynix reallocating some hbm production to dram

SK hynix is reportedly delaying the transition of some fifth-generation HBM (HBM3E) production lines that were originally scheduled to be converted to HBM4. It is intended to increase the general-purpose DRAM market responsiveness, which currently has a higher operating profit…

37
r/LocalLLaMA community 7d ago

Why is NO one talking about Microsoft's open source Fast Context!!!

https://huggingface.co/microsoft/FastContext-1.0-4B-SFT https://github.com/microsoft/fastcontext FastContext-1.0 is a lightweight repository-exploration subagent for LLM coding agents. Instead of letting a single model both explore the repository and solve the task, FastContext…

38
r/LocalLLaMA community 7d ago

Is Gemma 4 going to be the next Mistral (or Qwen3.6) one day? Concerning the lack of finetunes

https://eqbench.com/creative_writing.html#:~:text=gemma%2D4%2D31B,Sample From what I've seen Gemma 4 has better everything (especially long-context adherence) EXCEPT for the raw prosing performance of Mistral... finetunes . Comparing bases only, Mistral Small 3.2 (the…

5
r/LocalLLaMA community 7d ago

New ablation operator. (apostate)

Today I added a new operator to apostate. This new operator is a contrastive co-vector edit E = I − R Dᵀ . Removing the refusal direction outright disturbs benign behavior, while naively preserving all harmless variance along it leaves the refusal that is entangled with general…

34
r/LocalLLaMA community 7d ago

DeepSeek raises $7.4B USD at $60B valuation. Remarkably, Liang Wenfeng invests $3B in DeepSeek himself.

  submitted by   /u/FullOf_Bad_Ideas [link]   [comments]

35
r/LocalLLaMA community 7d ago

Qt Creator 20 and local AI

  submitted by   /u/cristianadam [link]   [comments]

12
r/LocalLLaMA community 7d ago

GLM-5.2 vs Claude Opus

  submitted by   /u/johnnyApplePRNG [link]   [comments]

28
r/LocalLLaMA community 7d ago

Add Laguna M.1 GGUF support by empty-quiver · Pull Request #2003 · ikawrakow/ik_llama.cpp

Laguna M.1 225B-A23B GGUF : https://huggingface.co/sigargv/Laguna-M.1-GGUF FYI ik_llama.cpp already supports Laguna XS.2 too Laguna XS.2 33B-A3B GGUF : https://huggingface.co/ji-farthing/Laguna-XS.2-ik-llama-GGUF-v2 Try with latest ik_llama.cpp version & share your feedbacks on…

38
r/LocalLLaMA community 7d ago

Top-N-Sigma: Remove unconditional softmax+sort by TimNN · Pull Request #22645 · ggml-org/llama.cpp

Overview Currently, the Top-N-Sigma sampler does an unconditional softmax+sort at the end. In the (common, I believe) case of Top-N-Sigma being followed by Dist, this expensive work is completely wasted. Additional information On my M3 Max MacBook Pro, this PR increases the t/s…

32
r/LocalLLaMA community 7d ago

Moebius: 0.2B Lightweight Image Inpainting Framework with 10B-Level Performance

  submitted by   /u/Diablo-D3 [link]   [comments]

20
r/LocalLLaMA community 7d ago

Chinese Hackers Latest Masterpiece with NVIDIA

They reverse-engineered the Tesla v100's pinout definition, soldered it onto a half height PCB, then naming it Tesla v100 v4. Price (with 3 years warranty): 16G version: 1499 rmb (220 usd) 32G version: 3999 rmb (590 usd) The hacker's op:…

10
r/LocalLLaMA community 7d ago

TMax: A Simple Recipe for Terminal Agents

TMax is the strongest open RL recipe for terminal agents to date, bringing open data recipes closer to the frontier. We release two things. The first is TMax-15k , a dataset of 14,600 RL environments built from a compositional pipeline with explicit control over difficulty and…

22
r/LocalLLaMA community 7d ago

NEX-N2-mini: "There is no Pareto frontier. I am Pareto". This Qwen3.5-MoE fine tune fixed 3.5 and 3.6 overthinking apparently on my tests.

I have been testing all popular MoE for my Mac and it seems I just found gold: 3.5/3.6 level of reasoning (if not slightly superior) at a fraction of the reasoning tokens used (wasted). Dynamic plot with other benchmarks here: https://benchmark-yourself.streamlit.app/…

4
r/LocalLLaMA community 7d ago

Same model, same prompt, 4 different agents

Setup: one self-hosted Qwen3.6-27B (Q4) on llama.cpp, identical prompt, identical hardware. The only variable is the agent scaffolding. Agents tested: pi, opencode, hermes, qwen code . Task: a single-file 2D canvas solar system with scripted orbits and gravity that acts only on…

14
r/LocalLLaMA community 7d ago

GLM-5.2 UD-IQ1_M on llama.cpp — 5090 + 3090 Ti speed test (~ 579 t/s prefill @ 8k ctx, ~324 t/s prefill @ 57k ctx, ~10.6 t/s decode)

Just sharing some speed test numbers for GLM-5.2 running on llama.cpp. Setup: Model: unsloth/GLM-5.2-GGUF, UD-IQ1_M quant GPUs: RTX 5090 + RTX 3090 Ti 186 GB DDR5 used Debian 13 CUDA 13.3 128k context, q8_0 KV cache Prefill (prompt processing): n_tokens tokens/s 8,201 579.75…

4
r/LocalLLaMA community 7d ago

GLM5.2 @7tg on 4x3090 + 192GB on budget motherboard + cpu

I finally finished by home lab computer I started working on in May. I carefully waited and bought the 3090s in three local transactions. Every single seller was a gamer who was upgrading to 4090 or 5090 and none had any interest in AI. I bought the 192GB of 5200MHz of DDR5 and…

9
r/LocalLLaMA community 7d ago

Qwen3.6-35B-A3B APEX on a Single RTX 3090 - Getting the Most Out of It

Resources I used: - https://github.com/ikawrakow/ik_llama.cpp - as the reference llama.cpp fork - https://github.com/spiritbuun/buun-llama-cpp - to test the TurboQuant feature - https://huggingface.co/mudler - for the models - https://github.com/noonghunna/club-3090 - for speed…

19
r/LocalLLaMA community 7d ago

been tracking EU DDR5 data for 25 days: Prices are dropping, and the DE vs. NL gap is wild (good news for local LLM builders in EU)

hey again! been tracking DDR5 prices across 4 EU countries (DE, NL, ES, BE) for the past month. some findings relevant to local LLM builders: prices are falling: G.Skill DDR5 Aegis 2x16GB 6000: -28% in 25 days (€579 → €419) Kingston FURY Beast RGB 2x16GB 6000: -26% (€499 → €369)…

37
r/LocalLLaMA community 7d ago

Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale

arXiv : https://arxiv.org/abs/2606.15079 Full Paper : https://arxiv.org/pdf/2606.15079 HuggingFace : https://huggingface.co/inclusionAI/models?sort=created (This month they released base models for both Ling-2.6-1T & Ling-2.6-flash ) -------------------------- Wish they released…

11
r/LocalLLaMA community 7d ago

European inference providers for GLM 5.2, DeepSeek V4 Flash?

So I am using Openrouter and I see that for GLM 5.2 it lists 16 providers. Most of them in the US, 1 or 2 in Singapore or China. Are there seriously no European inference providers for open-weight models? (No I don't mean Mistral, I mean a provider running especially the Chinese…

12
r/LocalLLaMA community 7d ago

Gemma 4 QAT 31B responds better to KV cache quantization too

I've run benchmark from this post and got even better results on Gemma 4 31B   submitted by   /u/justicecurcian [link]   [comments]

29

Tmax-27b - a Qwen3.6-27b terminal agent for small GPUs trained with DPPO (RL)

OpenMythos benchmarks

I'm eager for a 15x speedup on my strix halo

650+ Apache-2.0 biomedical NER/de-id models that run on-device in MLX. Same fp32 weights, identical outputs: the clinical NER models run 30-40x faster than PyTorch-CPU on a 3-year-old M3 Max. Repro inside.

UPDATE: Qwen-27B-IQ4_KS and Qwen-27B-IQ_KS_KT for ik_llama.cpp, especially for NVIDIA with 16GB VRAM

Not ironclad confirmation, but..

Is it possible to run a giant model like GLM5.2 on this cluster (4x servers with 512GB RAM + dual AMD Epyc)? 16 channel memory should hit 409GB/s per node.

Openrouter model prices implying heavier quantization?

GLM 5.2 on Mac Studio Speedup PR

I benchmarked 8 LLMs for medical scribing. Hallucinations were rare; omissions need attention.

7 Chinese companies are already shipping H100/H200-class AI chips, most IPO'd in the last 6 months. I mapped all of them.

Krea 2 released on Hugging Face

I mapped the KLD of KV cache quantization for Qwen3.6-35B-A3B and Gemma4-E2B QAT

Baidu: One-shot Long-horizon Parsing

CPU-only TTS benchmark: Kokoro 82M vs Supertonic 3 vs Inflect-Nano-v1 (4.6M params), with UTMOS scoring on every sample

Pooled round robin hardware with friends?

V100 4-card AI large model, Tesla 128G server

I love GLM 5.2's attitude! It is a nice refresher from those bootlicker doormats they are feeding us. Does that come from training datasets related to the local culture?

My local server idling 99% of the time!

Human Evaluation of GLM-5.2

Multi Tier MoE Caching

How do I prove that I don't collect data from my llm app?

Training a Qwen 3.5 4B/9B agent for multi-tool use: SFT first or go directly to RL?

Is there any reason for a lack of love for Gemma 4 26b?

MiniMax-M3-EAGLE3-GGUF - Llama.cpp compatible MiniMax M3 EAGLE draft model!

100+ t/s on Qwen3.6-27B Q8 across a 5090 + 3090 Ti — switching to tensor split-mode got me from 70 to 100+

Boogu Base, Turbo, Edit - open-source unified image generation and editing model series

Eff U, Arc / B70 Customers. We got ours! -Your Sugar Baby, Intel

EU AI Act requires TEXT from models and providers to be watermarked 2nd August onwards. Everyone here is affected, regardless where you live.

sk hynix reallocating some hbm production to dram

Why is NO one talking about Microsoft's open source Fast Context!!!

Is Gemma 4 going to be the next Mistral (or Qwen3.6) one day? Concerning the lack of finetunes

New ablation operator. (apostate)

DeepSeek raises $7.4B USD at $60B valuation. Remarkably, Liang Wenfeng invests $3B in DeepSeek himself.

Qt Creator 20 and local AI

GLM-5.2 vs Claude Opus

Add Laguna M.1 GGUF support by empty-quiver · Pull Request #2003 · ikawrakow/ik_llama.cpp

Top-N-Sigma: Remove unconditional softmax+sort by TimNN · Pull Request #22645 · ggml-org/llama.cpp

Moebius: 0.2B Lightweight Image Inpainting Framework with 10B-Level Performance

Chinese Hackers Latest Masterpiece with NVIDIA

TMax: A Simple Recipe for Terminal Agents

NEX-N2-mini: "There is no Pareto frontier. I am Pareto". This Qwen3.5-MoE fine tune fixed 3.5 and 3.6 overthinking apparently on my tests.

Same model, same prompt, 4 different agents

GLM-5.2 UD-IQ1_M on llama.cpp — 5090 + 3090 Ti speed test (~ 579 t/s prefill @ 8k ctx, ~324 t/s prefill @ 57k ctx, ~10.6 t/s decode)

GLM5.2 @7tg on 4x3090 + 192GB on budget motherboard + cpu

Qwen3.6-35B-A3B APEX on a Single RTX 3090 - Getting the Most Out of It

been tracking EU DDR5 data for 25 days: Prices are dropping, and the DE vs. NL gap is wild (good news for local LLM builders in EU)

Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale

European inference providers for GLM 5.2, DeepSeek V4 Flash?

Gemma 4 QAT 31B responds better to KV cache quantization too