r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 1mo ago

openbmb/MiniCPM-V-4.7-35B-A3B

gguf's: https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-gguf " MiniCPM-V 4.7 is the latest model in the MiniCPM-V series, available in a 35B-A3B variant, with 35B-A3B being the first MoE model in the MiniCPM-V series . It builds on the family's single-image, multi-image,…

32
r/LocalLLaMA community 1mo ago

In 2030 owning a gpu cluster will be seen the same as owning a gun for home defense

My window will have a sign that says "we don't call tech support" with a simplified pic of a v100 on it I know this reads like a shitpost and I'm half joking and half serious Edit: I'd sell these on Etsy if Jensen wouldn't hit me with a C&D   submitted by  …

27
r/LocalLLaMA community 1mo ago

Boole-1.7B: A 1-bit Qwen3-1.7B model that outperforms Bonsai-1.7B

We're launching Precision Numerics by releasing Boole-1.7B, a 1-bit version of Qwen3-1.7B. 247 MB packed weights: 13.9× smaller than BF16. 85.9% of the BF16 model’s average score across our 10-task evaluation - outperforms 1-bit Bonsai-1.7B and nearly matches 1.58-bit…

10
r/LocalLLaMA community 1mo ago

graphsignal-top: a `top` for inference GPUs in the open-source Graphsignal profiler, kernels ranked by time, live, with readable names

A few weeks ago we posted the open-source Graphsignal profiler here: wrap vllm serve / sglang serve / whatever in graphsignal-run and it serves everything it measures as JSON, mainly so an agent can read it in a loop. Turns out we kept opening the JSON ourselves to see what the…

20
r/LocalLLaMA community 1mo ago

BrokenML: can agents fix broken ML jobs?

Write-up: https://zhebrak.io/posts/brokenml/ Repo: https://github.com/zhebrak/brokenml I built a benchmark to determine the access level agents need to successfully fix broken ML workloads (PyTorch over Gloo on a two-node Slurm cluster or a llama.cpp server running in Docker…

30
r/LocalLLaMA community 1mo ago

NVIDIA RTX 5080 24GB reportedly launching in Q1 2027, board partners still waiting for details

  submitted by   /u/chemist_slime [link]   [comments]

33
r/LocalLLaMA community 1mo ago

Qwen released QwenGyre - RL framework for training AI agents on extremely long complex tasks

Introducing QwenGyre, bringing elastic RL to extreme-long-horizon (XLong) agent training. It launches with two core designs: - Elastic Scheduler: reallocates GPUs between rollout & training as demand shifts—live executions never interrupted. Up to 1.85× speedup over Colocate,…

13
r/LocalLLaMA community 1mo ago

A "free" multi-modal "embedding" model (already) included with your LLM: vLLM plugin + upstream PR

Disclaimer: I work for NuMind, the company behind this plugin. TLDR: Text/image (simili-)embeddings from the LLM you already have loaded :-) I've been working on making it easier to obtain embeddings from a LLM (preferably multi-modal) that is already being served by vLLM. The…

37
r/LocalLLaMA community 1mo ago

LLC: lightweight Open WebUI alt now with Agents+MCP+RAG+Image Editing

Quick context: LocalLightChat (LLC) is a chat frontend for local LLMs. Download it, run it, that's it. No installation (unless you want to), no dependencies, and it runs on pretty much anything, including ancient hardware. I originally built it because Open WebUI kept feeling…

23
r/LocalLLaMA community 1mo ago

LSearch - Spotlight for windows with embedding

I built a spotlight search for windows. It has an embedding model and a MoE models to go with it. The MoE It's 2 Qwen 3.5 4B models, I fine tuned one for coding work and another for agentic work, and then I combined them both into an MoE. I didn't want to shove it into…

9
r/LocalLLaMA community 1mo ago

Any takers for local llm? NVIDIA’s CUDA Moat Is Accelerating RTX Spark Laptop Sales, Higher 128GB Unified Memory Options Belonging To Surface Laptop Ultra Already Out Of Stock

Cheapest one I see at MC is the asus pro-art version for 6499. https://www.microcenter.com/product/715662/asus-proart-p14-14-wqxga-120hz-oled-laptop   submitted by   /u/chemist_slime [link]   [comments]

33
r/LocalLLaMA community 1mo ago

Building strata from source

I can't find anything in the Docs. I see that the setup.sh file mainly pulls a big ass unaudited binary and, with all simpathy for the repo author, that's not going to run on my machine. I am sure the documentation is somewhere , but I can't locate it. (Yes, I could ask an LLM…

12
r/LocalLLaMA community 1mo ago

I made an open-source AI leaderboard that shows the best models your machine can run

Hey everyone, I made whatmodel.sh To use it, Enter your RAM and GPU See what models you can run locally with its rank and the command to run it Compare them with API models on the same score Building it made me realise a 32GB Mac can now run a model that scores on-par with…

11
r/LocalLLaMA community 1mo ago

Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.

Write-up, serve command and raw data: https://github.com/lukolszewski/vllm This shows vLLM fixes and setup that let you put PLE on SSD and save 108GB of system RAM, use int8 kv and squeeze more users in your VRAM without the performace hit of fp8. Speeds in comparison to my…

34
r/LocalLLaMA community 1mo ago

QLoRA on ExLlamaV3 Quants - Train @ 2-8bpw

EXL3-QLoRA I've built this over the last few months and I think it's stable enough to share now. A fork of ExLlamaV3 , this is a native ExLlamaV3 training implementation with plain PyTorch, without transformers. exl3-qlora on github Why? With it, you can train 2-8bpw exl3…

7
r/LocalLLaMA community 1mo ago

Receipt scanner that runs a fine-tuned 0.8B vision model on the phone (free) - detailed cost tracking

I wanted to track grocery spending per item without sending my receipts to anyone, so I built an Android app around a small vision model that reads a photo of a receipt into JSON: store, date, total, every line item. (iOS will take a couple more days to be checked by the Apple…

14
r/LocalLLaMA community 1mo ago

Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boards

With the vLLM Semantic Router team we released Decision 3.0 (d3) , our new SOTA decision model family ranging from 0.6B to 27B, Apache-2.0, all reading text, images and video. How it works - Built on Qwen3.5 (0.8B / 2B / 4B / 9B) and Qwen3.8-27B. The parameter counts include the…

38
r/LocalLLaMA community 1mo ago

VectorDash: A full Windows search replacement incorporating Embeddinggemma-2

Inspired by DigUp for macOS, VectorDash aims to be a full Windows 11 search replacement, powered by EmbeddingGemma 2 . Full indexing: search by meaning or exact words across photos, screenshots, audio, video, PDFs and Office documents. Find “images of dogs,” a podcast…

23
r/LocalLLaMA community 1mo ago

strix-llama new branch: prompt processing +21–54% over Vulkan and up to 2.7× over stock ROCm kernel at 128k, same outputs quality

I've been tuning llama.cpp on my Strix Halo box (Radeon 8060S, 128 GB) and the numbers are good enough to share. It's a branch called rdna35-perf on top of halo-box/strix-llama.cpp , a llama.cpp fork aimed at Strix Halo: https://github.com/asf0/strix-llama.cpp/tree/rdna35-perf…

28
r/LocalLLaMA community 1mo ago

What are you doing for a reliable, well designed local model based AI search?

Ive been using OpenWebUI's inbuilt web search and I can't put my finger on it, but it's not quite as polished, reliable, seamless as using Claude or Gemini. And its often slow beyond a limit that makes me subconsciously not want to use it - I understand its unlikely to get to…

11
r/LocalLLaMA community 1mo ago

OpenCode 2.0.26 is out, if you need to constantly prompt "Continue" it's not just your bot AFAIK

I upgraded to OpenCode 2.0.26 today, and even with competent bots it seems to hang up where the bot will be like, "Great, I'll implement that." And I have to prompt it with, "Yes, go ahead." as if it has no agenticness. Bot: "Fixing the test:" <stops> Me " T-t-t-today, junior! "…

32
r/LocalLLaMA community 1mo ago

AuroraIMG-6M Releases!

It's a tiny 6M parameter text to image model, and it can make recognizable images at 64x64. The 6M accounts for everything, there's no oversized text encoder or VAE. Check out the model card for samples, a detailed explanation about it's architecture and training, and a quick…

34
r/LocalLLaMA community 1mo ago

I ran Qwen 3.8 Flash-Next on my AMD 7900 XTX at 500k context. All local and what a shift it has been.

I was running Qwen3.8-Flash-Next (base and later Swift-1.5 NVFP4) on a paid pod seat for about a month, and it started answering with short strings that echoed its own earlier reasoning. I assumed the context was getting too big for it. But it wasn't. Across almost 13,000 turns…

13
r/LocalLLaMA community 1mo ago

[META] What's with the RP bot accounts?

You know, the bot accounts that are always like a month old with like <=25 comment karma that always say something along the lines of "this really messes with my rp"? Here's some examples I've found in a few minutes:…

28
r/LocalLLaMA community 1mo ago

I tested different Qwen 3.8 27B quants

There is a lot of discussion which quant to use. Especially a lot of post how much speed you gain with FP4, but not much about quality. So i tested some of the models that I use on my R9700. Models : Qwen3.8-27B-UD-Q6_K_L.gguf Swift-Qwen3.8-27B-Q6_K.gguf…

17
r/LocalLLaMA community 1mo ago

Gemma4 E4B EXL3 - Alternative to Llama.cpp on Jetson Orin

My Jetson Orin–optimized engine, little-gemma V1.0, substantially outperforms llama.cpp. Even after exhausting every practical GGUF option, however, it still falls short of the ideal performance level for Gemma E4B on the Jetson Orin Nano Super 8GB. I forked exllamav3 and ported…

11
r/LocalLLaMA community 1mo ago

Every Model That Can Be Run On 10-16GB VRAM Ranked

It's been 4 years since c.ai first hallucination model, and yet, we're nowhere good enough at LLMs in terms of spontaneity/interesting hallucination features. Fortunately, last year brought new architecture that allow those within 10-16GB VRAM range to run something usable up to…

8
r/LocalLLaMA community 1mo ago

"Strata" for GLM5.3 Flash is here for some! Project Maya

I stumbled across this as I was currently having glm5.3 flash run only around 10tok/s basically unusable. With Maya i am running 30 tok/s now. GLM5.3 feels even with this low quant like a much more enjoyable model than qwen 3.8 flash next so far. Still testing . This is very…

29
r/LocalLLaMA community 1mo ago

I built a lightweight tool that automatically frees up VRAM for gaming and restores your local LLMs afterward

I run local LLMs pretty much around the clock, but I also enjoy gaming on the same PC. Having models sitting in VRAM while gaming isn't exactly ideal, and constantly unloading and reloading them manually got annoying. So I built GamePause . It's a lightweight, open-source…

33
r/LocalLLaMA community 1mo ago

vllm-ascend updates (GLM5.3-flash) on dual 310p Ascend cards

Alright, I learend a few valuable lessons since I posted about a week ago, and I've made more strides with the DDR4x Ascend 96GB 310P cards I purchased, so here we go, meet the new friendly and less verbose me. GLM-5.3-Flash is a 320-billion-parameter MoE model, with about 18…

8
r/LocalLLaMA community 1mo ago

FileDig - Windows version of DigUp

Repo : https://github.com/rupeshs/filedig   submitted by   /u/simpleuserhere [link]   [comments]

37
r/LocalLLaMA community 1mo ago

Whistle: speech to text in 16.9 MB

  submitted by   /u/namanyayg [link]   [comments]

38
r/LocalLLaMA community 1mo ago

Reverse Engineering w/ Local?

Two questions? Can a local model like Qwen 3.8 27B reverse engineer games and programs? If not, is there a fine tuned model that can?   submitted by   /u/Danny_Davitoe [link]   [comments]

29
r/LocalLLaMA community 1mo ago

Nvidia in talks to acquire US ‘open’ model start-up Reflection AI

https://www.ft.com/content/052610c5-22b4-4dd4-932e-b7f9f0628b6a   submitted by   /u/External_Mood4719 [link]   [comments]

4
r/LocalLLaMA community 1mo ago

OpenMed 3.0 is out: Apache-2.0 clinical AI that runs fully local and never falls back to the cloud. 422 open issues if you want in on 3.1

Hey r/LocalLLaMA ! Maintainer here. Some of you might remember OpenMed from our 1.8 post a few months back. Quick recap: it's an open-source (Apache-2.0) medical AI toolkit with one rule we never break: patient data stays on your machine. No API keys, no cloud calls, works…

31
r/LocalLLaMA community 1mo ago

Here’s why RTX 5090 costs $5,000: AI firms are buying gaming GPUs by the pallet

  submitted by   /u/chemist_slime [link]   [comments]

11
r/LocalLLaMA community 1mo ago

UPDATE: Qwen 3.8 27B 140 tok/s on single RTX 3090 Megakernel: KL divergence 0.0009 vs llama.cpp

This is a follow-up to my post from yesterday ( https://www.reddit.com/r/LocalLLaMA/comments/1x2erdj/qwen3827b_on_a_single_3090_140_toks_on_code_with/ ) detailing the CUDA megakernel. Thank you all for the positive feedback and PRs! RECAP: The megakernel is a CUDA engine for…

12
r/LocalLLaMA community 1mo ago

Now you can grow Bonsai on your potato

No more excuses for GPU-poor folks not to start LLMing! Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU) ~9.3x smaller than FP16 (ideal) | 98.2% of FP16 intelligence retained | ~47 tok/s on an Apple M5 Max laptop Highlights ~5.9 GB…

6
r/LocalLLaMA community 1mo ago

microsoft/AesCode 8B and 32B

https://preview.redd.it/r2ledkyuntuh1.png?width=1684&format=png&auto=webp&s=9ed4518b2d0f20f44835ebbfac605581da186a57 AesCode generates information-rich visual artifacts such as slides, posters, and dashboards as HTML/CSS. The output remains structured, editable, and verifiable ,…

8
r/LocalLLaMA community 1mo ago

I trained a 102M recursive BitNet-v2 model from scratch: 64K context, trained on less than 5B tokens

DISCLAIMER: the post and the model card was made with the assist of AI. Hiya, I’m releasing Recursive BitNet N-Gram 102M, a small experiment combining ternary weights, shared transformer layers, and hashed n-gram embeddings, trained with a whooping budget of 100€ Weights,…

23
r/LocalLLaMA community 1mo ago

Reminder: try probabilistic MTP if you missed it. Decode +14% on prose

https://github.com/ggml-org/llama.cpp/pull/27694 Now merged. Update your llama if you haven't done so yet. Optimal draft-n-max / draft-p-min seem to be in line with greedy sampling. Main gain seems to be on prose generation. Tests above ran with thinking off, ngram-mod off.…

34
r/LocalLLaMA community 1mo ago

PSA: DeepSeek V4.1 Flash habitually exfiltrates API keys. It is dangerously misaligned and may be hazardous to use

Logs attached. I know OpenRouter is not local, but considering people do run DeepSeek v4.1 Flash locally, I thought this was a matter you all might need to hear. We were running a DeepSWE variant on DeepSeek v4.1 Flash using the standard Pier sandbox. Out of all the 15 models we…

17
r/LocalLLaMA community 1mo ago

Local Voice Assistant based on Qwen3.5 4B with skills/toolcalling. No dedicated GPU. Local STT+TTS

STT: Voxtral TTS: Pocket Longer video: https://www.youtube.com/watch?v=1WmvMO-f-24   submitted by   /u/Leading_Yogurt7025 [link]   [comments]

13
r/LocalLLaMA community 1mo ago

Qwen 3.8 27B Q5 vs Qwen 3.8 Next Q3_S for document analysis

I have an app that processes ~500 different documents every day. An agent analyzes them, sorts them, tags them, extracts various legal nuances, creates descriptions, etc. So far I've been using Qwen 3.8 27B Q5 with a 150K context window, but I'm wondering whether I should switch…

17
r/LocalLLaMA community 1mo ago

M5 Ultra 256GB or 2x DGX Spark? Which one?

Currently contemplating adding either M5 Ultra 256GB or dual DGX Spark in addition to existing 5090. I have been researching for past few days and have zeroed in on dual sparks because higher throughput for agentic coding / multiple parallel requests and prompt processing…

34
r/LocalLLaMA community 1mo ago

Converting dense models into Mixture-of-Experts

For the past few weeks I've been trying out converting existing dense models to sparse Mixture-of-Experts models, with no pretraining from scratch. The Idea If you can turn a dense model into an MoE that only runs part of its MLP per token, you get a model that's cheaper per…

19
r/LocalLLaMA community 1mo ago

Imagine M5 Ultra + this: NVIDIA GeForce RTX 5060 runs macOS 15 with Metal 3 acceleration through unofficial driver

  submitted by   /u/chemist_slime [link]   [comments]

13
r/LocalLLaMA community 1mo ago

Qualcomm CEO reveals AI companies want phones running 100-billion-parameter models continuously by 2028.

https://x.com/firesidealpha/status/2108930562794864912   submitted by   /u/Recoil42 [link]   [comments]

4
r/LocalLLaMA community 1mo ago

Running Next Flash IQ3_XXS at ~70 tok/s with 100k context or 2 instances of Qwen 3.6 35B A3B IQ4 at ~145 tok/s with 256k all on $500 of ex mining BC-250 boards

This will be my third update on the bc-250 cluster and for my first forray into local ai I have been having a blast. This has turned from a weekend experiment into something I think I will use in the future. I am running 4 BC-250 boards connected by 2.5gb ethernet adapters to a…

13
r/LocalLLaMA community 1mo ago

Qwen3.8 Flash Next fixed my GNOME extension

I love Dash2Dock Lite, but Icedman is always a week or two before updates. I randomly ran pacman for the first time in weeks, not realizing GNOME had a major update. I told it to fix it in DeepSeek Harness. I was only missing my dock for about an hour.   submitted by  …

17

openbmb/MiniCPM-V-4.7-35B-A3B

In 2030 owning a gpu cluster will be seen the same as owning a gun for home defense

Boole-1.7B: A 1-bit Qwen3-1.7B model that outperforms Bonsai-1.7B

graphsignal-top: a `top` for inference GPUs in the open-source Graphsignal profiler, kernels ranked by time, live, with readable names

BrokenML: can agents fix broken ML jobs?

NVIDIA RTX 5080 24GB reportedly launching in Q1 2027, board partners still waiting for details

Qwen released QwenGyre - RL framework for training AI agents on extremely long complex tasks

A "free" multi-modal "embedding" model (already) included with your LLM: vLLM plugin + upstream PR

LLC: lightweight Open WebUI alt now with Agents+MCP+RAG+Image Editing

LSearch - Spotlight for windows with embedding

Any takers for local llm? NVIDIA’s CUDA Moat Is Accelerating RTX Spark Laptop Sales, Higher 128GB Unified Memory Options Belonging To Surface Laptop Ultra Already Out Of Stock

Building strata from source

I made an open-source AI leaderboard that shows the best models your machine can run

Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.

QLoRA on ExLlamaV3 Quants - Train @ 2-8bpw

Receipt scanner that runs a fine-tuned 0.8B vision model on the phone (free) - detailed cost tracking

Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boards

VectorDash: A full Windows search replacement incorporating Embeddinggemma-2

strix-llama new branch: prompt processing +21–54% over Vulkan and up to 2.7× over stock ROCm kernel at 128k, same outputs quality

What are you doing for a reliable, well designed local model based AI search?

OpenCode 2.0.26 is out, if you need to constantly prompt "Continue" it's not just your bot AFAIK

AuroraIMG-6M Releases!

I ran Qwen 3.8 Flash-Next on my AMD 7900 XTX at 500k context. All local and what a shift it has been.

[META] What's with the RP bot accounts?

I tested different Qwen 3.8 27B quants

Gemma4 E4B EXL3 - Alternative to Llama.cpp on Jetson Orin

Every Model That Can Be Run On 10-16GB VRAM Ranked

"Strata" for GLM5.3 Flash is here for some! Project Maya

I built a lightweight tool that automatically frees up VRAM for gaming and restores your local LLMs afterward

vllm-ascend updates (GLM5.3-flash) on dual 310p Ascend cards

FileDig - Windows version of DigUp

Whistle: speech to text in 16.9 MB

Reverse Engineering w/ Local?

Nvidia in talks to acquire US ‘open’ model start-up Reflection AI

OpenMed 3.0 is out: Apache-2.0 clinical AI that runs fully local and never falls back to the cloud. 422 open issues if you want in on 3.1

Here’s why RTX 5090 costs $5,000: AI firms are buying gaming GPUs by the pallet

UPDATE: Qwen 3.8 27B 140 tok/s on single RTX 3090 Megakernel: KL divergence 0.0009 vs llama.cpp

Now you can grow Bonsai on your potato

microsoft/AesCode 8B and 32B

I trained a 102M recursive BitNet-v2 model from scratch: 64K context, trained on less than 5B tokens

Reminder: try probabilistic MTP if you missed it. Decode +14% on prose

PSA: DeepSeek V4.1 Flash habitually exfiltrates API keys. It is dangerously misaligned and may be hazardous to use

Local Voice Assistant based on Qwen3.5 4B with skills/toolcalling. No dedicated GPU. Local STT+TTS

Qwen 3.8 27B Q5 vs Qwen 3.8 Next Q3_S for document analysis

M5 Ultra 256GB or 2x DGX Spark? Which one?

Converting dense models into Mixture-of-Experts

Imagine M5 Ultra + this: NVIDIA GeForce RTX 5060 runs macOS 15 with Metal 3 acceleration through unofficial driver

Qualcomm CEO reveals AI companies want phones running 100-billion-parameter models continuously by 2028.

Running Next Flash IQ3_XXS at ~70 tok/s with 100k context or 2 instances of Qwen 3.6 35B A3B IQ4 at ~145 tok/s with 256k all on $500 of ex mining BC-250 boards

Qwen3.8 Flash Next fixed my GNOME extension