r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 1d ago

How accessible is local AI actually, and what happens if affordable access to frontier models doesn’t last?

Sometimes it’s easy to forget that this sub and others like it are probably the extreme minority when it comes to this hobby. Most people, I would think, don’t use or can’t afford one good GPU, let alone multiple GPUs, Mac Studios, Sparks, Strix Halos, etc. Is the average tech…

14
r/LocalLLaMA community 1d ago

2400cc Inference Racer: Dual RTX 3090 motors, NVLink turbo, naked 7840U ThinkPad ECU, VW Golf radiator

Today I present a fine piece of engineering, carefully assembled inside a custom chipboard chassis: the 2400cc Inference Racer , a.k.a. my winter heater. Power comes from two second-hand AORUS RTX 3090 XTREME WATERFORCE cards. One glows a beautiful teal, the other red. I have no…

37
r/LocalLLaMA community 1d ago

Where is MiMo V2.6 9B Distill RL?

Was gonna test it out but I can't find the RL checkpoint.   submitted by   /u/Aggravating-Push-207 [link]   [comments]

13
r/LocalLLaMA community 1d ago

For the longest time I’ve felt this sub should have a pinned section where a detailed post about each model should get featured.

For instance whenever a model comes out, what’s the best engine to run it, the best harness and absolute minimum you need to get same or near same re results that the benchmark of that model claims. And whenever a quant from Unsloth guys comes out the guide can either be updated…

30
r/LocalLLaMA community 1d ago

Qwen3.8-27B IQ3_XXS vs Qwen3.6-35B-A3B Q4_K_M

Which one is better for difficult tasks like web scrapping, coding, using tools? Looking for any benchmarks because i couldn't actually find one after quite some digging   submitted by   /u/Loose_Doubt367 [link]   [comments]

30
r/LocalLLaMA community 1d ago

My Reading Library: Evaluating LLMs on Android Tasks

Can LLM agents actually get through a day in the life of a normal user? That question got me reading papers on Android agents and mobile benchmarks over the past few months. A few patterns kept showing up: Most benchmarks run on emulators, making real-device metrics difficult to…

9
r/LocalLLaMA community 1d ago

The future of local AI

For those of us who have been around for a while, we witnessed the huge demand for desktops, servers, and specialized appliances in the early 2000s. Everything was hosted in house. Of course, the bottleneck was the Internet. Then the cloud came along, and everything moved from…

6
r/LocalLLaMA community 2d ago

Qwen3.8 flash next + exllamav3 + hermes is amazing

I know there is nothing new with what I am saying but I recently started with hermes agent (it’s been a while I wanted to but did not have the time). Qwen3.8fn 6bpw exl3 (from turboderp) on a 6x3090 (I assume lower quants on lower number of gpus work same) gives me around…

28
r/LocalLLaMA community 2d ago

I built a tiny (332MB) CPU-friendly model for document sorting that actually knows when to say "none fits" (BeeNara)

Hey r/LocalLLaMA ! I wanted to share a small project I’ve been working on called BeeNara Why I built this: I was looking for a way to automatically sort my local documents (invoices, letters, contracts) into my personal folders. While local LLMs are amazing, I noticed that…

35
r/LocalLLaMA community 2d ago

Introducing KoboldCpp Agent (and a plea for help)

Hello r/localllama once again, it's me your kobold concedo Been a few months since I last posted here, and today I have something new I'd like to share. Specifically, KoboldCpp now ships with a built-in integrated KoboldCpp Agent Harness ! I know it's a little late to the game,…

29
r/LocalLLaMA community 2d ago

internlm/Intern-Decision 4B and 0.8B

https://huggingface.co/internlm/Intern-Decision-0.8B Intern-Decision-4B Demo | Model Weights | GitHub Intern-Decision-4B is a multimodal structured decision model fine-tuned from Qwen3.5-4B . It accepts a shared state, a schema of named questions, and optional images, and…

6
r/LocalLLaMA community 2d ago

Best current Qwen Flash Next Q4-ish? + worth using?

Im running a 5090 and 64gb of ram, so im limited on what I can run. I have currently been able to fit the following - Atomic Q4_k_m 4.27bpw @ 31 layers offload Swift IQ4_xs @ 32 layers offload. Im about to try the Unsloth IQ4_xs as well. I could get a "bigger" (non IQ) quant for…

19
r/LocalLLaMA community 2d ago

Swift 1.5 27b: Swift Qwen just got faster

Enjoy! Fucking loving it.   submitted by   /u/sleight42 [link]   [comments]

11
r/LocalLLaMA community 2d ago

Qwen3.8-27B Q4_K_M on 2x3060

For the last few days, I've been using two computers to run multiple agents. My 4x3090 machine is running Qwen 3.8 27B with parallel=2, so I can run two agents at the same time. My pi instances are running on a machine with 2x3060, running/managing smaller models such as Gemma…

31
r/LocalLLaMA community 2d ago

ggml-cpu: tiled mul_mat for k-quants by jbooth · Pull Request #27851 · ggml-org/llama.cpp

faster CPU prompt processing: "TL;DR: 3-7x faster CPU mul_mat using VNNI with IMO minimal complexity"   submitted by   /u/jacek2023 [link]   [comments]

18
r/LocalLLaMA community 2d ago

What are you doing between prompts?

My local prompts take twenty minutes to three hours as of now. Curious for others what are you doing while it’s working?   submitted by   /u/xiraov [link]   [comments]

26
r/LocalLLaMA community 2d ago

Ling Tiny 3.0 is a glimpse of the future

I've been playing around with Ling 3.0 Tiny, which is an 8 billion parameter model (MoE, 1B active). And I've had a lot of poignant thoughts as a result. Just for fun, I got it running with llama.cpp on an old laptop. This is a laptop from 2017 with a 7th gen i5 and 8 gigs of…

27
r/LocalLLaMA community 2d ago

Custom Models in Oh My Pi: vLLM, llama.cpp, SGLang and More

  submitted by   /u/bolts98 [link]   [comments]

27
r/LocalLLaMA community 2d ago

Blabbermouth AI coding agents hate this one weird trick!

The conceited little fuckers love to inundate you with unnecessary details, noisy caveats, what's 'load bearing' and what's not, waste your time with a wall of text every time it reports back to you. They think human PP times are as fast as theirs, but they're not! It takes time…

35
r/LocalLLaMA community 2d ago

Are there still any hidden gem gpu's left?

you read about gpu's like the Tesla P40, V100, AMD M150 etc that people pick up for cheap. probably many others as well. Most are either server gpu's being phased out or mining discards, right? The problem of course is that whenever someone discovers these, they then make a…

7
r/LocalLLaMA community 2d ago

Kev 4B topped out in every Tetris game I ran. Mica v0.1 4B cleared about 4x more lines and survived two of them to the end

I had Mica (my 4B decision model) and Kev 4B play the same Tetris games, same seed and same piece order, one RTX 3090. For context, this is a side project. Training and all the experiments ran on rented 3090s, about $30 in total. Every turn both get the board and 4 possible…

19
r/LocalLLaMA community 2d ago

Mica v0.1 4B got an iron pickaxe in real Minecraft without generating a single token

Mica v0.1 4B playing a real Minecraft 1.20.4 server. Video attached. How it works - Each step the bot's live game state (inventory, nearby blocks, entities, last result) is written out as text. - Mica scores the candidate commands and picks the next one. It never generates text.…

31
r/LocalLLaMA community 2d ago

Mica v0.1 4B: open Jev-style decision model (yes/no, choice, score) that runs on an 8 GB GPU — trained for under $30 of GPU time

I've been building a small decision model for agent loops: gates, routers, "should I ask the user or just act" checks. It's out now as Mica v0.1 4B (Apache-2.0). What it does You give it a state, a question and the allowed answers, and it returns a calibrated probability for…

18
r/LocalLLaMA community 2d ago

What IDE to use for local models

Hi people, I am looking for a lightweight IDE or plugin that won't inject large context at initiation. I tried Cline and native VS Code but they inject such heavy initial context that it fills up my gpu and either goes oom or spend most of my time compacting. The only one I…

6
r/LocalLLaMA community 2d ago

How long can I expect to wait until the local ~30B A3B frontier catches up to GLM 5.3 Flash quality?

The jump from Qwen3 Coder 30B A3B to current-day Qwen 3.6 35B A3B is crazy, especially with all the fine-tunes, and that was around 6 months (I didn't care for local AI back then, or AI at all, apart from as a toy so I don't know). Is around a year until I will never need cloud…

16
r/LocalLLaMA community 2d ago

4-5 days replacing Claude w Qwen 3.8 Next

Hi, human here with rambling thoughts to share. Feel free to skip Overall, I don’t feel like I’m missing much; if anything. On my hardware(M1 ultra w 128Gb) it’s probably not as fast as Claude but I’ve been using opencode for research and other business related tasks and it’s…

37
r/LocalLLaMA community 2d ago

Qwen3.8-27B: Using KV Cache Transplants to Boost Output Quality

Since my last post , I've been thinking about different options for dynamic performance degradation, trying to squeeze as much high-quality inference out of my GPU as I can. Over the weekend I read this really interesting paper: Cache-to-Cache: Direct Semantic Communication…

18
r/LocalLLaMA community 2d ago

I ran the actual break-even math on buying vs renting an H200 box, and it is not where I expected

Every rent-vs-buy thread I read has confident people on both sides, but not many actually show the numbers. So I finally ran the numbers for our own decision. Posting the working here in case it is useful, or feel free to point it out in case someone thinks it's wrong. An 8-GPU…

28
r/LocalLLaMA community 2d ago

Swift1.5-Qwen3.8-Flash-Next is phenomenal vs. base 3.8-Flash!

TL;DR - Swift Flash is a killer model that massively reduces excess reasoning. Try it out! If you haven't seen from my previous comparison posts , I'm a huge fan of the Swift Qwen3.8 models. I've been using 27B since it dropped, and I'm really impressed with the performance and…

15
r/LocalLLaMA community 2d ago

Is there a lightweight version of Hermes agent?

I have limited Context (usually around 64k) For local use I don’t only do coding But also want like a personal assistant with memory and such. What is the best option?   submitted by   /u/Adventurous-Gold6413 [link]   [comments]

35
r/LocalLLaMA community 2d ago

What I learned letting a local 27B run overnight long-horizon coding on my own rig

Hi reddit, i know you hate AI slop so i indeed write the intro myself! iam dev and curios about local inference and long hoirzon coding on my own box. last day-ish i let my local model (qwen 27b on llama.cpp, 2x 16GB cards) go on a long coding tour inside deepseek harness while…

35
r/LocalLLaMA community 2d ago

Trained my first small language model

I have a tool that uses Gemini Flash with the lowest thinking budget to do summarization work. It's very fast, 0.9-1.2s in most cases. But I have a user experience problem where people make the wrong choice when using an internal app for the team. Gemini Flash can figure out…

12
r/LocalLLaMA community 2d ago

Ion v0.2.0 — No install. No backend. Just one HTML file.

The new Ion is available, a harness that run directly from a single HTML file, no install or backend required: is now more capable, more customizable, and has better tools!   submitted by   /u/fredconex [link]   [comments]

26
r/LocalLLaMA community 2d ago

Jev vs. Kev: open-source Jev alternative tested side by side

We hosted Kev 4B (Jared Palmer's Apache-2.0 fine-tune of Qwen3.5-4B) and ran it side by side with Jev on the same endpoint to see how it compares. We built a fresh set of 362 items published after both models shipped (new arXiv papers, Stack Exchange questions, GitHub issues),…

9
r/LocalLLaMA community 2d ago

Make Volta Fast Again

For those who have V100 cards, I wanted to point you to 1Cat-vLLM, a vLLM fork that enables optimized serving for these cards. Showing stats for Qwen3.6-35b comparing a Strix Halo with a hughly optimized llama.cpp fork (pwilkin) and the V100 with 1Cat. It’s not apples to apples,…

22
r/LocalLLaMA community 2d ago

I compared diarization models on 15 clinical conversations: Nemotron 3, Pyannote, Sortformer and VibeVoice

I've been working on clinical speaker attribution at Omi and wanted to compare the current diarization models on the same audio. I used 15 mock doctor–patient consultations from PriMock57, about 2.4 hours. Full recordings, automatic speaker counts, without telling the models…

35
r/LocalLLaMA community 2d ago

How do you use subagents & multiple agent with local models, and how many?

Running qwen3.8 27b nvfp4 on vllm at max context only gives around 8 agents with 32k context each. That doesnt seem like much; what use cases do people use multi-agent frameworks and find it helpful for?   submitted by   /u/Ambitious_Fold_2874 [link]   [comments]

15
r/LocalLLaMA community 2d ago

muse charm but local and free

  submitted by   /u/llo7d [link]   [comments]

38
r/LocalLLaMA community 2d ago

Splash on a 40-core M5 Max: +20% decode by tuning the kernels for your own chip

FYI the engine's default kernel rules were measured on smaller chips (16/20-core M5s and a 32-core M4 Max), so a 40-core M5 Max runs guesses. Splash's repo includes a developer tool, “make tune-kernels” that tests every available way of running each quantised matrix-multiply on…

16
r/LocalLLaMA community 2d ago

Qwengram-0.8B: I transferred Qwen3.8 Flash-Next’s n-gram memory into Qwen3.5-0.8B — 5.05% lower validation perplexity

I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model. I trained the 0.8B setup with limited resources, mostly using free Kaggle notebook GPUs. The setup keeps both the Qwen3.5-0.8B backbone and the…

23
r/LocalLLaMA community 3d ago

PSA for M5Ultra owners running LLMs: set your prefill step to 8192

Prefill step size affects both PP performance and your drafter fetching logits for MTP (affects dflash as well, mlx-vlm needs to be patched slightly to support chunked prefill for dflash). It also needs to be reasonably large to be able to fill all your cores but not overfill -…

12
r/LocalLLaMA community 3d ago

VLLM 4x rtx 3060 vs 8x rtx 3060 performance loss

Hello! I am currently building my local AI server, I have the Huananzhi H12D-8D EPYC Motherboard with 8x16GB memory sticks at 2666 mhz (waiting for the other components at the moment) I currently have four RTX 3060 12gb gpus and I plan running those at PCIe4 x16 in VLLM. However…

17
r/LocalLLaMA community 3d ago

Has anyone benchmarked AI agents against the SOLIDWORKS CSWA exam?

Would be interesting right? Models are starting to score higher and higher on benchmarks like Parametric CAD Bench , but can they pass an actual exam? The Certified SOLIDWORKS Associate (CSWA) exam might be an interesting place to start. They have an sample exam on their…

6
r/LocalLLaMA community 3d ago

Gemma 4 Developer Agent Competition

Just saw this pop up. This might be a fun one for the folks in here!   submitted by   /u/lakySK [link]   [comments]

5
r/LocalLLaMA community 3d ago

LLM on a budget part 2, from P102-100 to CMP 50HX.

I finally got around to upgrade the GPU's. First a word of warning, when upgrading GPU's on P520 you have to be extra careful not to slot the card on any angle other than straight when installing or when pulling the card out, the reason for that is that about a 1/4 an inch from…

34
r/LocalLLaMA community 3d ago

Former Intel CEO: "HBM is lousy". High Bandwidth Flash Is Coming

Irrational Analysis:"HBM is a mistake" Former Intel CEO: "HBM is lousy" SK Hynix VP:"HBM is not the final answer to the memory wall problem" "If the stacks get high enough...each core die operates slower than plain old commodity memory" Hot chips 2026 Q&A, Irrational Analysis…

6
r/LocalLLaMA community 3d ago

Did anyone do a full bench of e.g. Qwen Flash Next IQ4 and Qwen 27b FP8? Here are some

I let Codex do some eval on Qwen 3.8 Flash Next IQ4_XS (served via vllm and r9v) and Qwen 3.8 FP8 (served via vllm and radiance). Here are the results: Benchmark Flash-Next IQ4_XS Qwen3.8 27B FP8 Result MMLU-Pro 83.8% 75.0% Flash-Next GPQA Diamond 42.5% 27.5% Flash-Next GSM8K…

8
r/LocalLLaMA community 3d ago

M5 Ultra 80Core GLM-5.3-Flash on DwarfStar Speeds

I've been playing around with various models on the M5 Ultra 256GB 80-core Mac Studio. These are the results over many rounds of agentic inferencing. I'm happy with the performance. Glad to have the large amount of RAM. But it does feel like the GPU is underpowered for this…

6
r/LocalLLaMA community 3d ago

normalize benchmarks from different time period

LiveBench has benchmark snapshots from different points in time. Could someone run an agent to normalize the values across these snapshots so we can compare model strength consistently from 2024 through 2026? Right now, it’s difficult to make meaningful comparisons across the…

18
r/LocalLLaMA community 3d ago

How are you guys thinking about context now, and building around it?

Not asking for anyone’s secrets of the trade, I’m more curious how people are thinking about context now that newer models chew through huge amounts of it for reasoning. the TLDR: I’m starting to think of context less as working memory and more as a temp scratchpad to start each…

13

How accessible is local AI actually, and what happens if affordable access to frontier models doesn’t last?

2400cc Inference Racer: Dual RTX 3090 motors, NVLink turbo, naked 7840U ThinkPad ECU, VW Golf radiator

Where is MiMo V2.6 9B Distill RL?

For the longest time I’ve felt this sub should have a pinned section where a detailed post about each model should get featured.

Qwen3.8-27B IQ3_XXS vs Qwen3.6-35B-A3B Q4_K_M

My Reading Library: Evaluating LLMs on Android Tasks

The future of local AI

Qwen3.8 flash next + exllamav3 + hermes is amazing

I built a tiny (332MB) CPU-friendly model for document sorting that actually knows when to say "none fits" (BeeNara)

Introducing KoboldCpp Agent (and a plea for help)

internlm/Intern-Decision 4B and 0.8B

Best current Qwen Flash Next Q4-ish? + worth using?

Swift 1.5 27b: Swift Qwen just got faster

Qwen3.8-27B Q4_K_M on 2x3060

ggml-cpu: tiled mul_mat for k-quants by jbooth · Pull Request #27851 · ggml-org/llama.cpp

What are you doing between prompts?

Ling Tiny 3.0 is a glimpse of the future

Custom Models in Oh My Pi: vLLM, llama.cpp, SGLang and More

Blabbermouth AI coding agents hate this one weird trick!

Are there still any hidden gem gpu's left?

Kev 4B topped out in every Tetris game I ran. Mica v0.1 4B cleared about 4x more lines and survived two of them to the end

Mica v0.1 4B got an iron pickaxe in real Minecraft without generating a single token

Mica v0.1 4B: open Jev-style decision model (yes/no, choice, score) that runs on an 8 GB GPU — trained for under $30 of GPU time

What IDE to use for local models

How long can I expect to wait until the local ~30B A3B frontier catches up to GLM 5.3 Flash quality?

4-5 days replacing Claude w Qwen 3.8 Next

Qwen3.8-27B: Using KV Cache Transplants to Boost Output Quality

I ran the actual break-even math on buying vs renting an H200 box, and it is not where I expected

Swift1.5-Qwen3.8-Flash-Next is phenomenal vs. base 3.8-Flash!

Is there a lightweight version of Hermes agent?

What I learned letting a local 27B run overnight long-horizon coding on my own rig

Trained my first small language model

Ion v0.2.0 — No install. No backend. Just one HTML file.

Jev vs. Kev: open-source Jev alternative tested side by side

Make Volta Fast Again

I compared diarization models on 15 clinical conversations: Nemotron 3, Pyannote, Sortformer and VibeVoice

How do you use subagents & multiple agent with local models, and how many?

muse charm but local and free

Splash on a 40-core M5 Max: +20% decode by tuning the kernels for your own chip

Qwengram-0.8B: I transferred Qwen3.8 Flash-Next’s n-gram memory into Qwen3.5-0.8B — 5.05% lower validation perplexity

PSA for M5Ultra owners running LLMs: set your prefill step to 8192

VLLM 4x rtx 3060 vs 8x rtx 3060 performance loss

Has anyone benchmarked AI agents against the SOLIDWORKS CSWA exam?

Gemma 4 Developer Agent Competition

LLM on a budget part 2, from P102-100 to CMP 50HX.

Former Intel CEO: "HBM is lousy". High Bandwidth Flash Is Coming

Did anyone do a full bench of e.g. Qwen Flash Next IQ4 and Qwen 27b FP8? Here are some

M5 Ultra 80Core GLM-5.3-Flash on DwarfStar Speeds

normalize benchmarks from different time period

How are you guys thinking about context now, and building around it?