r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 9d ago

Six months ago I turned down $8,165 for an RTX 6000 PRO. Today the same vendor is selling them for $11,575. Oh, hindsight.

  submitted by   /u/__JockY__ [link]   [comments]

34
r/LocalLLaMA community 9d ago

Qwen code companion on vscode marketplace - thoughts

I just came across this extension in vscode few days ago and tried to use with LM studio hosted models and it really is pretty good compared to `continue`, `kilo`, `cline`, `roo` like I felt without much tweaks, gets straight to the point, if any tweaks required u could do…

36
r/LocalLLaMA community 9d ago

Gemma 4 26b a4b is genuinely the best model I have tried for language learning and scientific queries!

I know gemma 4 26b is (according to this sub) a bit behind for coding tasks but for language learning and scientific (health/biology/medical/clinical/biochem) queries it’s unbeaten even by Qwen 3.5/3.6. Since the competition in the small MOE models is generally between Qwen…

28
r/LocalLLaMA community 9d ago

I wrote a free 15-part series on LLM internals — real math, real tensor shapes, real hardware constraints. All grounded in Gemma 4 12B's actual config.

If you run open-source models and want to understand what's actually happening under the hood — I spent the last few months writing a 15-part series that covers the full stack from tokenization to production serving. Most articles are grounded in Gemma 4 12B as the running…

19
r/LocalLLaMA community 9d ago

Board where every tile is an agent

I've been hacking a project which I find extremely useful and wanted to share. Imagine a board where every tile is an agent those job is to maintain the tile. I tried to illustrate the idea with a video here. The project is open source on GitHub and you can also try it out here…

36
r/LocalLLaMA community 9d ago

Deep Neural Network that can turn any Image into a Playable Game! BUT LOCALLY, NOT ON DATACENTER

Hi everyone!! I really wanted to share my research what I've been working on. I wanted to build a nn that can simulate games, or at least start doing that Most video generators are too large to run on consumer hardware realtime, so I I designed a model that does this from…

14
r/LocalLLaMA community 9d ago

Bought 2x r9700, 5090 is now 7k and 6000 pro is at 13.5k, best option for 64 gb vram under 4k

after being frustrated with nvidia proces, I went with asrock r9700, not even dgx spark even they are at 7k now, did I make a mistake?   submitted by   /u/AppropriatePush6262 [link]   [comments]

20
r/LocalLLaMA community 9d ago

You can now convert EXL3 quants on Apple Silicon Mac

Hi, I'm here with an update. But this time it's quite a bigger news on local llm. Normally accessing the high fidelity quant like EXL3 is CUDA gated, and imagine you need 96GB-128GB with RTX cards, they are very specialized and expensive. But now on a more general basis, MacOS…

38
r/LocalLLaMA community 9d ago

z.AI as the number 2 gives praise to the number 1 open source model

  submitted by   /u/Charuru [link]   [comments]

27
r/LocalLLaMA community 9d ago

Any opinion about Qwen3.6-27B@BF16 vs Step3.7@IQ4_XS?

Obviously one is dense, slower but at full precision, and the other is MoE, 7x more params, less than ideal quant, and will eat up more memory, but my question is: Which would make saner decisions with less hand-holding, ie which is genuinely smarter? I've been using…

34
r/LocalLLaMA community 9d ago

Research Project: Injecting Natural-Language Tactical Intent into Multi-Agent Football Policies

Human Intent as a Control Interface for Multi-Agent Systems I've been exploring a project called Football Tactical AI . The idea is simple: Instead of directly controlling players, a human acts as a coach and gives tactical instructions in natural language. For example: "Press…

30
r/LocalLLaMA community 9d ago

[GLM 5.2 UD IQ2_M] That's the best pelican svg image I have ever seen

Computer Specs: rtx 5090 + rtx 3090 (x8 x8 bifurcated) Gigabyte AI TOP B850 Motherboard Ryzen 9950x3d 256gb DDR5 5600 (4x64gb) I didn't have high hopes because of the low quant but damn. This model is capable as hell just by looking at this image I can tell that. The tps is low…

25
r/LocalLLaMA community 9d ago

Best local LLM for English story summarization

Hello, which local LLM is currently the best at story summarization? The stories can be multiple pages long and are in English. Thanks!   submitted by   /u/DesperateGame [link]   [comments]

24
r/LocalLLaMA community 10d ago

Worlds Biggest Chat Title Dataset From SupraLabs

If you search "Chat title dataset" on huggingface a few dys ago, the biggest chat title dataset you would get from it was "ogrnz/chat-titles", but recently at supralabs we have curated a 115K filtered dataset whih breaks the world record for the biggest dataset from 10k samples…

32
r/LocalLLaMA community 10d ago

Best Settings for 48GB VRAM + Qwen 3.6 27B

Hey everyone, I've been running Qwen3.6 27B (Q8_0) across an RTX 4090 + RTX 3090 setup using llama.cpp with tensor split, and I wanted to share what's been working best for me so far. See if anyone has any better settings Hardware: RTX 4090 (24GB) + RTX 3090 (24GB), 48GB VRAM…

4
r/LocalLLaMA community 10d ago

Help with a Local Document RAG System (Storage + Ingestion + Query + Highlighting)

Hey folks, I’m working on designing a local, offline document retrieval + LLM pipeline and would love your input on the architecture. Here’s what I’m aiming for: Storage Upload PDF, DOCX, XLSX, CSV, tables All data stored locally (no cloud) Document Ingestion Watch folder (e.g.,…

29
r/LocalLLaMA community 10d ago

7900XTX 24GB vram, can finally fit Q6K+MTP with Qwen 3.6 27B at 131k context

OS: CatchyOS Instructions: Connect monitor to iGPU directly so when you boot Linux your dGPU vram is 100% free since by default when you use your dGPU it consumes about 700mb~1.2gb of lost context space, yes you can still game normally using this approach. Setup kvcache at…

30
r/LocalLLaMA community 10d ago

GLM 5.2: 98% of max level intelligence with less than half of tokens usage

According to this number of reasoning tokens from GLM 5.1 to GLM 5.2 more than doubled from 16.7k to 36.7k and for me as a local user with old junk Xeon setup this makes GLM 5.2 unusable to the extent where I had to shut down model after 12h of waiting it to respond to my math…

37
r/LocalLLaMA community 10d ago

Some llama.cpp B70 SYCL benchmarks

build: dd4623a74 (9640) | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: | | gemma4 12B Q8_0 | 11.78 GiB | 11.91 B | SYCL | -1 | pp512 | 1578.19 ± 7.82 |…

11
r/LocalLLaMA community 10d ago

AMD future GPU offerings. Some interesting offerings for a LLM build. What type of LLM rig would you build with these?

https://preview.redd.it/xrgj4u64dd8h1.png?width=1287&format=png&auto=webp&s=2a63e8ea1c6aadf7b61b5820a148a5413a2d7d74 From: https://www.youtube.com/watch?v=U8xBkDBVjPM&t=1459s   submitted by   /u/sooki10 [link]   [comments]

32
r/LocalLLaMA community 10d ago

Local AI for local office files

Which AI agent do you think is the best for working with local files (Excel, PDF, Word, txt, json, etc.)? What have you used for this? What workflows have you implemented?   submitted by   /u/Holiday-Display509 [link]   [comments]

29
r/LocalLLaMA community 10d ago

What is the best book for learning ML/Deep Learning maths?

I am 17 years old and have a particularly deep interest in ai architectures and llms. I regularly read new papers from arXiv and huggingface and tend to understand only half of it, mainly from intuition. However I understand its impossible to understand anything completely…

33
r/LocalLLaMA community 10d ago

Tool calling, opencode qwen3.6 27b 8K

Not sure I'm ready to post an issue in the opencode repo yet but wanted to see if this is common, return to the opencode window after walking away to let it do its thing to find its stopped with this in its thinking.. Started noticing more last week or so, the fix is easy just…

10
r/LocalLLaMA community 10d ago

Giving a local agent web access without paid search/scrape APIs: SearXNG + Scrapling

I wanted web access for a local-first agent without reaching for Tavily, Serper, Firecrawl, etc. For this agent path, I wanted no paid API keys, a search service I control, and page extraction I can run myself. What I ended up with is two tools: web_search and web_extract .…

6
r/LocalLLaMA community 10d ago

Local agent on 4090 - looking for LM Studio settings

I have moved on from Ollama to just dink around and instead want to start running a local agent from time to time. With the 24GB of a 4090 (Gigabyte OC edition) that should be quite possible. But no matter what settings I use for context and batching, token generation is slow as…

36
r/LocalLLaMA community 10d ago

RTX 5090 MSI, only inference or training at 475-500W. Make sure to not bend you cable!

I run this MSI 5090 at 475-500W daily, for mostly diffusion training, or LLM inference. Just by chance I decided to check the cable today and found this. No issues, errors or anything, just all by chance. I never gamed on this card, got it entirely for AI and machine learning.…

8
r/LocalLLaMA community 10d ago

[NEW MODEL] SupraLabs just released supra-title-FFT-preview, 115K samples, almost 10x our first chat title dataset

Hey r/LocalLLaMA ! Following up on Supra-Title-350M-exp (our first chat title generation model), we're releasing supra-title-FFT-preview , trained on a much larger and cleaner dataset. 🤗 supra-title-FFT-preview What changed Our first chat title model was trained on 12K samples…

17
r/LocalLLaMA community 10d ago

I benchmarked Claude's "Fast C++". It wasn't faster

  submitted by   /u/User_Deprecated [link]   [comments]

15
r/LocalLLaMA community 10d ago

Single RTX 3090 (MSI TRio) giving trouble on inference.

Hi, I'm having weird issues with my 3090 on inferencerence via lmstudio , it just: unloads the model/ model crashes + nvidia driver resets freezes the pc gives blue/black screen and the computer restarts or straight up restarts everything. I tried running it regularly,…

33
r/LocalLLaMA community 10d ago

$1800 (in GPU cost running with P2P running Qwen/Qwen3.6-27b-FP8 with 262K context and BF16 KV cache at 55 tok/s

Hey peeps, wanted to share what is possible for folks with an inference only single user use case with 1700 in GPU cost. Setup: 4x 5060 ti (16GB) with P2P If you are in the US and you keep an eye on facebook marketplace and places like slickdeals you can find some 5060 ti 16 GB…

30
r/LocalLLaMA community 10d ago

Best Local Agents - Jun 2026

A megathread that is overdue! Let's discuss and debate on what the best local agents available today are Prologue First a note on terminology: While most regular users are going to have a general sense of what these are, I think its worth a brief pause to preempt turbulence in…

6
r/LocalLLaMA community 10d ago

How do I set the right llama.cpp parameters?

--n-gpu-layers all --ctx-size 0 --reasoning-budget 0 --presence-penalty 1.1 --repeat-penalty 1.1 How do I figure out the optimal llama.cpp parameters for my setup? llama.cpp + Open WebUI in Docker with an AMD GPU (16GB VRAM) running gemma 4 12b and 26b models. Is it all about…

13
r/LocalLLaMA community 10d ago

Maximizing performance of 2x3090 + NVLink

Hey all, I have built myself a decent rig with the following specs: - Ubuntu 24.04 - 2x3090 founder’s with NVLink - Ryzen 7950x3d - 64GB DDR5 I am currently routing my display through an eGPU to maximize available VRAM. My current go-to is Qwen 3.6 27B Q8_0 with MTP and…

6
r/LocalLLaMA community 10d ago

How do you guys setup search with your AI models?

Been selfhosting my models for a while and I'd really like to integrate Gemma 4 12B as a simple voice assistant with search capabilities. I've tried using openwebui but the search is kind of broken with DDG and I really don't want to use API keys from Brave or Google etc. So…

25
r/LocalLLaMA community 10d ago

GLM-5.2-REAP50-GGUF

Has anybody tried these? How do they compare to Qwen 3.6 27b? Model Size Link GLM-5.2-REAP50-Q3_K_M-GGUF 182 GB https://huggingface.co/pipenetwork/GLM-5.2-REAP50-Q3_K_M-GGUF GLM-5.2-REAP50-Q2_K-GGUF 139 GB https://huggingface.co/pipenetwork/GLM-5.2-REAP50-Q2_K-GGUF  …

4
r/LocalLLaMA community 10d ago

Commission selects EUROPA consortium as the winner of the Frontier AI Grande Challenge, a project to build European open-source frontier AI model in all 24 EU languages

The European Commission has selected EUROPA, a European consortium led by the Italian company Domyn, as the winner of its Frontier AI Grand Challenge. Commission selects EUROPA consortium as the winner of the Frontier AI Grande Challenge, a project to build European open-source…

11
r/LocalLLaMA community 10d ago

Improving local models with an API based "consultant"?

I'm sure that someone else has come up with this before, but i just wanted to ask: Has it occurred to anyone to improve their local AI workflow by adding a more powerful API based "consultant" agent (GLM 5.2 now springs to mind) to call upon for refining plans, learnings and…

35
r/LocalLLaMA community 10d ago

The economics of AI are starting to favor open models

For the last couple of years, the assumption was pretty simple: Want the smartest model? Pay for a closed API. Want something cheaper? Accept a capability hit. Looking at recent model releases, that tradeoff is starting to break down. The most interesting part of the chart isn't…

15
r/LocalLLaMA community 10d ago

LQ50-24 English translate

here the full English using google translate for you guy   submitted by   /u/MundanePercentage674 [link]   [comments]

22
r/LocalLLaMA community 10d ago

Best Harness for Web Searching

Looking for opinions on the best software to do web searching resources. What I've tried: LM Studio + plugins Odysseus I think the problem they're both running into is the search engines they're using max out at like, 10 requests per day/hour or something without an api. I don't…

17
r/LocalLLaMA community 10d ago

Benchmarking or benchmarketing?

Maybe I’m getting cynical, but LLM benchmarking is starting to feel less like measurement and more like marketing and positioning. Every week there’s a new leaderboard score, new chart, new eval suite, or some claim that a model is suddenly the best. It feels like benchmarks…

35
r/LocalLLaMA community 10d ago

What's more impressive, GLM 5.1 -> 5.2 or Qwen 3.5 -> 3.6?

Write a single HTML file with a full-page canvas and no libraries. Simulate a realistic Döner Style kebab skewer rotating (vertically) in front of a gas powered heating element. Mentioning Döner activates GLM 5.2s german weights or something (Spiess = Skewer, Brenner = Burner).…

34
r/LocalLLaMA community 10d ago

Is my CPU and RAM too weak/ lees for local LLMs? Both are going 100% for simple test prompts. GPU is not getting used fully. In theory quen3.5:9b should fit and run on RTX3050 8 GB comfortably.

https://preview.redd.it/i69vee9mi88h1.png?width=1592&format=png&auto=webp&s=820720e8a3e1d5386d49119a235e2902acc13265 I am very new to this local llm world. Just started to exploring from past 3days. Share any troubleshooting tips.   submitted by   /u/mr_whoisGAMER [link]…

12
r/LocalLLaMA community 10d ago

Watching a local AI voice assistant get dumber (A 9B to 0.8B agent experiment on my RTX 5060 Ti)

I wanted to find the exact floor for running an intelligent, local voice assistant agent on consumer hardware. I kept the environment, tools, and prompts identical, I stepped the model sizes down through Qwen 3.5 9B, 4B, 2B, and 0.8B to see how agentic reasoning degrades. The…

12
r/LocalLLaMA community 10d ago

GLM-5.2 is the new leading open weights model on the Artificial Analysis Intelligence Index

  submitted by   /u/pscoutou [link]   [comments]

20
r/LocalLLaMA community 11d ago

The Eagle(3) has landed (for Qwen)

https://github.com/ggml-org/llama.cpp/releases/tag/b9723 Available in the latest release. Enabled via: --spec-type draft-eagle3 You'll need to feed it a draft model. There's issues with unsloth + eagle at the moment so I've personally tested against: Model:…

16
r/LocalLLaMA community 11d ago

New Agentic Benchmark Out: Claude Fable and GLM 5.2 Top Their Cohorts

You can read about it here: https://artificialanalysis.ai/articles/aa-briefcase This is a solid benchmark from Artificial Analysis. It basically tests an LLMs ability to plan and execute tasks. And more importantly, it is a new benchmark that is not saturated, so no one can…

32
r/LocalLLaMA community 11d ago

spec: support eagle3 for qwen3.5 & 3.6 by ruixiang63 · Pull Request #24593 · ggml-org/llama.cpp

let's try is it better than MTP   submitted by   /u/jacek2023 [link]   [comments]

5
r/LocalLLaMA community 11d ago

Has anyone here used VibeThinker-3B outside benchmarks?

Just curious, given the hype and benchmark numbers. Curious about real-world behavior: debugging, coding assistance, reasoning over messy prompts, local latency, failure modes, and whether it actually feels useful versus just optimized for verifiable evals.…

23
r/LocalLLaMA community 11d ago

Researchers trained a Deep Research agent with 32 H100s and open-sourced everything

Ohio State University's NLP team released QUEST-35B, an open-source Deep Research agent trained using ~32 H100s and ~8K synthetic samples. The team open-sourced the training recipe, code, weights and datasets. Benchmark results show competitive performance against several…

13

Six months ago I turned down $8,165 for an RTX 6000 PRO. Today the same vendor is selling them for $11,575. Oh, hindsight.

Qwen code companion on vscode marketplace - thoughts

Gemma 4 26b a4b is genuinely the best model I have tried for language learning and scientific queries!

I wrote a free 15-part series on LLM internals — real math, real tensor shapes, real hardware constraints. All grounded in Gemma 4 12B's actual config.

Board where every tile is an agent

Deep Neural Network that can turn any Image into a Playable Game! BUT LOCALLY, NOT ON DATACENTER

Bought 2x r9700, 5090 is now 7k and 6000 pro is at 13.5k, best option for 64 gb vram under 4k

You can now convert EXL3 quants on Apple Silicon Mac

z.AI as the number 2 gives praise to the number 1 open source model

Any opinion about Qwen3.6-27B@BF16 vs Step3.7@IQ4_XS?

Research Project: Injecting Natural-Language Tactical Intent into Multi-Agent Football Policies

[GLM 5.2 UD IQ2_M] That's the best pelican svg image I have ever seen

Best local LLM for English story summarization

Worlds Biggest Chat Title Dataset From SupraLabs

Best Settings for 48GB VRAM + Qwen 3.6 27B

Help with a Local Document RAG System (Storage + Ingestion + Query + Highlighting)

7900XTX 24GB vram, can finally fit Q6K+MTP with Qwen 3.6 27B at 131k context

GLM 5.2: 98% of max level intelligence with less than half of tokens usage

Some llama.cpp B70 SYCL benchmarks

AMD future GPU offerings. Some interesting offerings for a LLM build. What type of LLM rig would you build with these?

Local AI for local office files

What is the best book for learning ML/Deep Learning maths?

Tool calling, opencode qwen3.6 27b 8K

Giving a local agent web access without paid search/scrape APIs: SearXNG + Scrapling

Local agent on 4090 - looking for LM Studio settings

RTX 5090 MSI, only inference or training at 475-500W. Make sure to not bend you cable!

[NEW MODEL] SupraLabs just released supra-title-FFT-preview, 115K samples, almost 10x our first chat title dataset

I benchmarked Claude's "Fast C++". It wasn't faster

Single RTX 3090 (MSI TRio) giving trouble on inference.

$1800 (in GPU cost running with P2P running Qwen/Qwen3.6-27b-FP8 with 262K context and BF16 KV cache at 55 tok/s

Best Local Agents - Jun 2026

How do I set the right llama.cpp parameters?

Maximizing performance of 2x3090 + NVLink

How do you guys setup search with your AI models?

GLM-5.2-REAP50-GGUF

Commission selects EUROPA consortium as the winner of the Frontier AI Grande Challenge, a project to build European open-source frontier AI model in all 24 EU languages

Improving local models with an API based "consultant"?

The economics of AI are starting to favor open models

LQ50-24 English translate

Best Harness for Web Searching

Benchmarking or benchmarketing?

What's more impressive, GLM 5.1 -> 5.2 or Qwen 3.5 -> 3.6?

Is my CPU and RAM too weak/ lees for local LLMs? Both are going 100% for simple test prompts. GPU is not getting used fully. In theory quen3.5:9b should fit and run on RTX3050 8 GB comfortably.

Watching a local AI voice assistant get dumber (A 9B to 0.8B agent experiment on my RTX 5060 Ti)

GLM-5.2 is the new leading open weights model on the Artificial Analysis Intelligence Index

The Eagle(3) has landed (for Qwen)

New Agentic Benchmark Out: Claude Fable and GLM 5.2 Top Their Cohorts

spec: support eagle3 for qwen3.5 & 3.6 by ruixiang63 · Pull Request #24593 · ggml-org/llama.cpp

Has anyone here used VibeThinker-3B outside benchmarks?

Researchers trained a Deep Research agent with 32 H100s and open-sourced everything