r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 1d ago

DFlash support merged into llama.cpp

  submitted by   /u/sammcj [link]   [comments]

36
r/LocalLLaMA community 1d ago

Sell ddr5 for vram?

Hi, I have 768gb ddr5 6400 ecc ram, given current ram prices should I sell half and buy rtx 6000 pros? Edit: currently have an epyc 9255 and 2 x rtx 6000 pro max q but could add another three Blackwell potentially then and run something like glm 5.2 in q4 at faster speeds then…

6
r/LocalLLaMA community 1d ago

Qwen3.6 27B local vs Opus 4.8, voxel engine in raw C with zero frameworks

Sunday experiment. Same prompt to both. Build a voxel world in plain C. No engine, no game library, no framework, just the compiler. The model does its own chunk meshing, render loop and memory management by hand. Left is Claude Code on Opus 4.8. Right is Qwen3.6 27B local on…

37
r/LocalLLaMA community 1d ago

Best case for dual RTX 3090 (250W each) on Crosshair VIII Hero?

I'm building a local LLM workstation and would appreciate some advice from people already running 2×3090s. Current hardware: ASUS Crosshair VIII Hero (X570) One Gainward Phoenix RTX 3090 Looking for a second used 3090 (not necessarily the same model) Both GPUs will be…

9
r/LocalLLaMA community 1d ago

Are there good closed vs open LLM rankings? Also, are 70B–350B models actually worth it?

hey, I’m currently getting enough VRAM to run something in the GLM-5.2 range, but I’m wondering: do we actually have a solid ranking that compares closed-source and open-weight LLMs side by side? I’ve been trying to find a clear “closed vs open” leaderboard, but most benchmarks…

26
r/LocalLLaMA community 1d ago

How many of you do use Q1 or Q2 of Big models(100-250B)? How's it?

Sharing popular(also recent) models for reference: 151-250B : DeepSeek-V4-Flash Step-3.X-Flash Command-a-plus-05-2026 Laguna-M.1 MiniMax-M2.X Qwen3-235B-A22B 100-150B : GLM-4.5-Air Qwen3.5-122B-A10B NVIDIA-Nemotron-3-Super-120B-A12B Mistral-Small-4-119B-2603…

34
r/LocalLLaMA community 1d ago

"DuckDuckGo is blocking with a CAPTCHA. Let me try other approaches:"

My local llama.cpp-based LLM just started reporting this this morning: "DuckDuckGo is blocking with a CAPTCHA. Let me try other approaches:" Is anyone else seeing this with DuckDuckGo?   submitted by   /u/SensitiveCranberry00 [link]   [comments]

34
r/LocalLLaMA community 1d ago

This application to join the GPT 5.6 Sol preview is wild

+ face scanner, fingerprint checker and passport swiper   submitted by   /u/Complete-Sea6655 [link]   [comments]

16
r/LocalLLaMA community 1d ago

A barebones CPU-only inference engine for Qwen 3, written from scratch in pure C

TL;DR: The (very messy) code and writeups can be found at https://github.com/jakint0sh/qwen3-engine Read the README for instructions on how to get started. And for those who just want a bulleted list: - Inference engine for Qwen 3 sizes 4B and below - Written from scratch in…

37
r/LocalLLaMA community 1d ago

We're probably going to need that soon.

From: Vladik on 𝕏: https://x.com/Kostoglodov/status/2071144065857679631 Shaw (spirit/acc) on 𝕏: https://x.com/shawmakesmagic/status/2070918006033817867   submitted by   /u/Nunki08 [link]   [comments]

8
r/LocalLLaMA community 1d ago

Whisperian: It is one of the best applications for Android, if you want to use Mic with some local ASR models. And it is also available on Play Store.

  submitted by   /u/9r4n4y [link]   [comments]

29
r/LocalLLaMA community 1d ago

Is Qwen3-VL-2B the only viable VLM for JSON extraction on a "potato"?

After spending countless hours testing on 3 "potato" laptops (Intel i3, 8GB RAM, Win11, integrated GPU), that's my conclusion. For reliably extracting data from images to JSON on low-end hardware, nothing else even comes close. Yet, it’s completely missing from major benchmarks…

23
r/LocalLLaMA community 2d ago

clark-labs/clark-air-sana-1.6b-1.58bit · Hugging Face

A Sana 1.6B text-to-image transformer compressed to ternary (~1.85 bits/weight): 8.6× smaller than FP16, near-FP16 quality. Footprint (measured) Artifact Size vs FP16 What it is FP16 transformer 3.21 GB 1× (100%) reference Clark Air (packed) 374 MB 8.6× (≈12%) packed ternary (…

36
r/LocalLLaMA community 2d ago

A Blind Visual Paradigm for Testing Skill Transfer in Small Models Without Fine-Tuning

TL;DR: Small models aren't dumb, they're shallow. I designed a cross-domain, blind, visual experiment to see if a large model can compress its "planning discipline" into a reusable scaffold that makes a small model deeper — with zero fine-tuning. Three.js is the testbed because…

28
r/LocalLLaMA community 2d ago

Finally.. my rig is maxed out

Got all the parts before the crazy price increase except for the rtx pro 5k! Was saving up to order rtx pro 6000 in US and i did, but wanted to join nvidia inception program for the discount. It was around $8.5k during that time, less 1k if I succeeded. It took around 3 months…

7
r/LocalLLaMA community 2d ago

US Ban Benchmark Updated: Toe-to-toe Between Two Big Names!

OpenAI ties with Anthropic in this benchmark following the preview of GPT 5.6 just yesterday. Chinese models have no hope of catching up forever, while Gemini's figure is yet to be updated.   submitted by   /u/Complete-Sea6655 [link]   [comments]

30
r/LocalLLaMA community 2d ago

10x Kaioken SSJ1 4th grade, worth it in 2026? Can it run Qwen3.6?

  submitted by   /u/Ice94k [link]   [comments]

20
r/LocalLLaMA community 2d ago

Koboldcpp v1.116 released

  submitted by   /u/Fcking_Chuck [link]   [comments]

19
r/LocalLLaMA community 2d ago

I had 55 LLMs blind-grade each other (22k judgments, all open). Every model family with enough data is biased toward its own siblings. Qwen judges favor Qwen by ~0.9 points. Mistral penalizes its own by ~1.0.

I have been running an open evaluation setup where N models answer the same prompt, then blind-grade each other in an N x N matrix with self-judgments excluded. No single privileged judge. So far: 286 evaluations, 198 hand-written questions, 22,254 valid judgments across 55…

35
r/LocalLLaMA community 2d ago

2x RX 9060xt 16gb, is it worth it?

I'm planning to buy 2x RX 9060xt with 16gb each to run Qwen 3.6 27B and alike. Would it be a good investment? How much tk/s should i expect in generation and prefill? I'm planning to use this as a coding agent in a large codebase. Currently I'm running this on my i7 64gb laptop…

35
r/LocalLLaMA community 2d ago

I built a tool to turn your Claude Code sessions into fine-tuning data for local models

If you use Claude Code, every session is already sitting on disk as a .jsonl file under ~/.claude/projects/ . It has real coding conversations: multi-turn edits, tool calls, reasoning traces. That's training data you already generated for free. The problem is the format is not…

36
r/LocalLLaMA community 2d ago

Full document redaction with Qwen 3.6 27B with a Pi agent harness

Link to full blog post with all method details, results, and links to all relevant code/skills/prompts at the bottom of this post. Apologies for not having more links throughout, it seems this subreddit restricts too many links in posts. Document redaction tasks are complex…

10
r/LocalLLaMA community 2d ago

Will Chinese Open Source Models be the only option soon?

US techbros do not just want to make money. They want total global control of everything. Releasing any more advanced AI interferes with that plan.   submitted by   /u/GeographHero [link]   [comments]

38
r/LocalLLaMA community 2d ago

Model Registry: Torrents for open models using Hugging Face as a fallback web seed.

Hi, I created a repo/site for publishing/sharing .torrent files for popular open models, added web seed support and a few scripts to automate it. Repo: https://github.com/marella/modelregistry Site: https://modelregistry.io Web seed will be used as a fallback to download files…

18
r/LocalLLaMA community 2d ago

If it doesn't make my PP better, I don't want it

Highlights: 4 x 48GB modded 4090s - 192GB VRAM 128GB DDR5 Pro WS WRX90E-SAGE SE 3000w PSU 240V/30A dryer line Q. Is putting a server on a dryer line a good idea? A. No, or emphatically yes. Splitters on this line are not code compliant, so I have to turn off the server to use…

33
r/LocalLLaMA community 2d ago

Could AI game upscalers (like DLSS/FSR) benefit from lightweight game-specific adapters, especially for handheld gaming?

This is more of a thought experiment than a proposal. I couldn't find much discussion of this in publicly available papers, so I'm wondering whether this has already been explored or whether I'm missing a fundamental limitation. One thing that surprised me after using my Legion…

27
r/LocalLLaMA community 2d ago

Does quantizing change the MTP draft rate?

Speculative decoding speeds up LLM generation by using a small "drafter" model to predict several tokens ahead of the main model. The main model then verifies these predictions in a single forward pass. If the main model is heavily quantized (low bit-rate), it becomes less…

32
r/LocalLLaMA community 2d ago

Ornith 35B is great so far

Tried creating a quick 3d game with it, after 3 prompts, it got me this(checkvideo). If I compare this with qwen3.5-35b-a3b, it was not able to successfully generate this and was failing even after multiple prompts. Harness: Claude Code How is your experience so far ?…

4
r/LocalLLaMA community 2d ago

Running GLM5.2 on budget hardware < $2500.

Too many times I hear people whine about not being ble to run SOTA models or claim it would require $50k, or $100k. https://www.ebay.com/itm/398079051468 Epcy Motherboard & CPU - $460 https://www.ebay.com/itm/206374955959 P40 24gb - $230 get 2 - $460…

19
r/LocalLLaMA community 2d ago

Even Google still believes in small models for coding.

I've been meaning to post about this. The community has been pretty vocal in criticizing "vibe-coded" projects. I used to think the backlash was the real problem, but I've started getting annoyed by a lot of these posts myself — many are just tiny, hyper-specific tools with…

37
r/LocalLLaMA community 2d ago

Anyone still doing fine-tunes on consumer grade hardware?

Felt like there used to be a thriving fine-tuning community a few years back - and then once we started getting models that were smart enough and generalist enough (i.e. post Llama-3-8b era) things kind of dropped off a little. Less need for fine-tunes when prompt-tweaking can…

22
r/LocalLLaMA community 2d ago

Mythos was the first, now GPT-5.6

https://techcrunch.com/2026/06/26/openai-limits-gpt-5-6-rollout-after-government-request-says-restrictions-shouldnt-be-the-norm/ Either a hype before IPO, or they have just shot themselves in a foot. This is pretty much it for more advanced online models. Local LLM is one of the…

17
r/LocalLLaMA community 2d ago

A Cory Doctorow Interview with Thoughts on AI and Some Advocacy of Local AI

On ArsTechnica: https://arstechnica.com/gadgets/2026/06/how-to-burst-the-ai-bubble-strike-at-its-roots/ I liked it so much, giving whiplashes to those known companies trying to IPO, among other things. I never knew the guy and don't know what else he said, but all this resonated…

13
r/LocalLLaMA community 2d ago

[NEW MODEL] - SupraSafety-18M · Tiny Content-Moderation Model

Hey r/LocalLLaMA ! SupraLabs is back with a new model: SupraSafety-18M . It's a BERT-style 18M params model trained from scratch on 2 T4 GPUs in Kaggle on the nvidia/Nemotron-3.5-Content-Safety-Dataset dataset for 7 epochs. It's built to run on edge devices , mobile phones , or…

13
r/LocalLLaMA community 2d ago

96gb+ 4090's and 5090 are literally a scam. I mods these cards myself

I run a small gpu lab in the USA and work closely with two factories in china designing/producing 48gb 4090 PCB's. The only recent card weve gotten was the 32gb 4080 super. PSA: 96gb 4090's and 5090's are a SCAM (as of Jun 2026) - you will not get the card, they do not exist.…

32
r/LocalLLaMA community 2d ago

Agentic Cyberdeck Dev

I developed this around August '25, but never had real polished panels. So, here we are with some decent panels, and new speakers for voice Al inferencing. This has local agentic GPS, chat, voice, vision analysis. This is a fun little project that I come back around to until I…

12
r/LocalLLaMA community 2d ago

What’s the latest on agent browser use?

What is the latest and greatest agent browser use framework? I remember trying browser use a few months back and it was ok but would fall apart after long workflows. Has there been improvements to agents controlling browsers and following a predefined workflow? Can local models…

32
r/LocalLLaMA community 2d ago

Dear poor people of this subreddit

I see people with multi-gpu setups but I'm sure there's a potato LLM runner out there somewhere. I have an old macbook pro (i5 8th gen, 8GB RAM) that I want to turn into a homelab. I want to run a small local model for experimenting and if possible, agentic tasks (like say…

22
r/LocalLLaMA community 2d ago

We built a calibration-aware Q4_K_M quant of Qwen3.5 0.8B that recovers 96.5% of the BF16 gap vs pure llama.cpp Q4_K_M (SpectralQuant)

Hey everyone, We just released our first release candidate from Spectral Labs: a Qwen3.5 0.8B Q4_K_M built using a new calibration-aware quantization approach we're calling SpectralQuant . The goal here was to see if we could make a standard Q4_K_M footprint behave more like a…

15
r/LocalLLaMA community 2d ago

Orthrus (diffusion head) trained Qwen 3.5/3.6 and Gemma 4 models are dropping soon

"Hi all, we are finalized with our testing and are preparing the release pipeline. We will be releasing support for the Qwen3.5, Qwen3.6, and Gemma4 very soon. Alongside the model checkpoints, we will be open-sourcing our complete end-to-end training and evaluation code. Stay…

19
r/LocalLLaMA community 2d ago

96 gig 5090s from Shenzhen's Huaqiangbei

We're visiting Shenzhen right now, and visited the Huaqiangbei electronics market. I've seen reports of 96 gig 5090s popping up on AliExpress, but never saw confirmation that it was real, so we asked around a little. One seller called a friend, and they said that a 5090 would…

37
r/LocalLLaMA community 2d ago

New deepseek vision model incoming?

Hello guys, it seems like DeepSeek added a new vision mode to their application. Does this mean, that they will release a new vision model? Edit: Guys.it is not an OCR model. I have just asked it to describe multiple images, which had no text in them.   submitted by  …

19
r/LocalLLaMA community 2d ago

Any better models in coding for single dgx spark in near future?

I’m an owner of single dgx spark with 128 gb unified memory. and I’m hosting through all my local network my ppm over lmstudio. I’m mainly using it for coding,some long document sorting tasks and some sequruty testing. my favorite rn is stepfun step-3.7-flash q3 xxl it’s a bit…

32
r/LocalLLaMA community 2d ago

Are there any qwen finetunes that were genuinely stronger than the base?

It's pretty popular to finetune qwen models but I never hear anyone say anything positive about them.   submitted by   /u/MrMrsPotts [link]   [comments]

30
r/LocalLLaMA community 3d ago

deepseek-ai/DeepSeek-V4-Pro-DSpark • Huggingface

https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-DSpark https://github.com/deepseek-ai/DeepSpec/blob/main/DSpark_paper.pdf   submitted by   /u/External_Mood4719 [link]   [comments]

18
r/LocalLLaMA community 3d ago

fine-tuned LiquidAI’s LFM2.5-230M on Fable-5 coding traces - its better than I expected it to be

fine-tuned LiquidAI’s LFM2.5-230M on Fable-5 traces and shipped it as GGUF tiny 230M coding-agent model. trained at 4096 ctx. exported Q4_K_M / Q8_0 / F16. runs locally. repo: https://hf.co/AKMESSI/lfm2.5-230m-fable-5   submitted by   /u/akmessi2810 [link]  …

35
r/LocalLLaMA community 3d ago

Another big tensor fix b9820

sched : reintroduce less synchronizations during split compute ( #20793 ) CUDA: Improve performance via less synchronizations between token ( #17795 ) Adds CPU-to-CUDA copy capability to ggml_backend_cuda_cpy_tensor_async() Adds function to relax sync requirements between input…

15
r/LocalLLaMA community 3d ago

When can we expect merged DeepSeek V4 Flash / MiniMax M3 llama.cpp support?

I am relatively new here, I have little experience in how long support development takes. I know there are forks. But not merged status means AFAIK that support is far from perfect. When can we expect stable full support for DeepSeek V4 Flash and/or MiniMax M3 in llama.cpp?…

4
r/LocalLLaMA community 3d ago

Ornith-1.0-35B Q3_K_M: ~17 GB VRAM, KLD-checked against BF16

I quantized deepreinforce-ai/Ornith-1.0-35B down to Q3_K_M so it fits comfortably on a single GPU. Produced locally with llama-quantize from the upstream BF16 GGUF — the quantizer took it from 16.01 BPW down to 3.87 BPW , landing at 16.8 GB on disk / ~17 GiB loaded VRAM , about…

15
r/LocalLLaMA community 3d ago

Findings from troubleshooting p2p on 4x5060 ti bifurcation.

I dumped the last week deep diving this and I’m I’ve been using Linux for 14 years and am a cloud systems engineer with a focus on supported Linux infrastructure for a private cloud provider. Essentially, if you are using a single 4x4 bifurcation pcie x16 card inserted into your…

5

DFlash support merged into llama.cpp

Sell ddr5 for vram?

Qwen3.6 27B local vs Opus 4.8, voxel engine in raw C with zero frameworks

Best case for dual RTX 3090 (250W each) on Crosshair VIII Hero?

Are there good closed vs open LLM rankings? Also, are 70B–350B models actually worth it?

How many of you do use Q1 or Q2 of Big models(100-250B)? How's it?

"DuckDuckGo is blocking with a CAPTCHA. Let me try other approaches:"

This application to join the GPT 5.6 Sol preview is wild

A barebones CPU-only inference engine for Qwen 3, written from scratch in pure C

We're probably going to need that soon.

Whisperian: It is one of the best applications for Android, if you want to use Mic with some local ASR models. And it is also available on Play Store.

Is Qwen3-VL-2B the only viable VLM for JSON extraction on a "potato"?

clark-labs/clark-air-sana-1.6b-1.58bit · Hugging Face

A Blind Visual Paradigm for Testing Skill Transfer in Small Models Without Fine-Tuning

Finally.. my rig is maxed out

US Ban Benchmark Updated: Toe-to-toe Between Two Big Names!

10x Kaioken SSJ1 4th grade, worth it in 2026? Can it run Qwen3.6?

Koboldcpp v1.116 released

I had 55 LLMs blind-grade each other (22k judgments, all open). Every model family with enough data is biased toward its own siblings. Qwen judges favor Qwen by ~0.9 points. Mistral penalizes its own by ~1.0.

2x RX 9060xt 16gb, is it worth it?

I built a tool to turn your Claude Code sessions into fine-tuning data for local models

Full document redaction with Qwen 3.6 27B with a Pi agent harness

Will Chinese Open Source Models be the only option soon?

Model Registry: Torrents for open models using Hugging Face as a fallback web seed.

If it doesn't make my PP better, I don't want it

Could AI game upscalers (like DLSS/FSR) benefit from lightweight game-specific adapters, especially for handheld gaming?

Does quantizing change the MTP draft rate?

Ornith 35B is great so far

Running GLM5.2 on budget hardware < $2500.

Even Google still believes in small models for coding.

Anyone still doing fine-tunes on consumer grade hardware?

Mythos was the first, now GPT-5.6

A Cory Doctorow Interview with Thoughts on AI and Some Advocacy of Local AI

[NEW MODEL] - SupraSafety-18M · Tiny Content-Moderation Model

96gb+ 4090's and 5090 are literally a scam. I mods these cards myself

Agentic Cyberdeck Dev

What’s the latest on agent browser use?

Dear poor people of this subreddit

We built a calibration-aware Q4_K_M quant of Qwen3.5 0.8B that recovers 96.5% of the BF16 gap vs pure llama.cpp Q4_K_M (SpectralQuant)

Orthrus (diffusion head) trained Qwen 3.5/3.6 and Gemma 4 models are dropping soon

96 gig 5090s from Shenzhen's Huaqiangbei

New deepseek vision model incoming?

Any better models in coding for single dgx spark in near future?

Are there any qwen finetunes that were genuinely stronger than the base?

deepseek-ai/DeepSeek-V4-Pro-DSpark • Huggingface

fine-tuned LiquidAI’s LFM2.5-230M on Fable-5 coding traces - its better than I expected it to be

Another big tensor fix b9820

When can we expect merged DeepSeek V4 Flash / MiniMax M3 llama.cpp support?

Ornith-1.0-35B Q3_K_M: ~17 GB VRAM, KLD-checked against BF16

Findings from troubleshooting p2p on 4x5060 ti bifurcation.