r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 1h ago

NVIDIA shipped OpenShell, an open source sandbox that gives local and open agents real runtime limits instead of prompt rules. Over 100 firms joined the safety stack. OpenAI did not.

https://x.com/JensenHuang/status/2104499465055023424   submitted by   /u/InternationalGap3698 [link]   [comments]

14
r/LocalLLaMA community 3h ago

3090 for $1500???

A couple weeks ago, I thought they were selling for 1200. On eBay now, the cheapest "buy it now" seems to be just shy of 1500. It feels like we're in a game of GPU musical chairs now.   submitted by   /u/sleight42 [link]   [comments]

13
r/LocalLLaMA community 4h ago

modified qwen 3.8 27b modifies windows credential dumper to bypass EDR detection

link to original article TLDR: security researcher eddie zhang used a modified+uncensored local qwen 3.8 27b to create an executable capable of dumping LSASS memory for credential harvesting while evading 2 modern EDR security products. this makes me reflect on how cloud…

15
r/LocalLLaMA community 4h ago

Minisforum MS-S1 MAX-P495 @ €7.799,00

MINISFORUM MS-S1 MAX-P495 – Minisforum EU Expected to ship mid october. At that price point, it doesn't make a whole lot of sense in my opinion. I get that ram prices are where they are, i get that it's a newer model of hardware, but twice the price for 50% more ram and ~5-10%…

20
r/LocalLLaMA community 5h ago

3090 NVLink bridges: are there clones of these? why are they so insanely expensive?

I think I need a 3-slot for my two cards. but holy fuck these things are pricey.   submitted by   /u/starkruzr [link]   [comments]

18
r/LocalLLaMA community 5h ago

Liked Muse, so I cut the 30B model in half by width, distilled it back, and it does 57 of 60 tool tasks its parent does 60 of

I've liked how Muse-Glimmer worked, so I wanted to see if I could produce a smaller "kid" out of it. Ornith's sharp decisions on when to think and which tool to call were the other thing I liked, so Ornith-1.0-9B got to be the policy teacher while the parent wrote the words. No…

37
r/LocalLLaMA community 5h ago

GPT-3 is discontinued today

It had such a long run. It was my first introduction to modern language models. I remember getting slightly excited over it. And now it lives purely in our memories. Arguably what's more infuriating is that they suggest using GPT-5.6 Terra as a replacement. Keep in mind that…

20
r/LocalLLaMA community 6h ago

macOS 27 ships a free local LLM on Apple Silicon Macs. I made it easy to use from Node and Python

Apple Silicon Macs on macOS 26+ come with a small LLM built in. No download, no API key, and nothing leaves your Mac. Why I built it I was making a tool that writes API docs from code, and I didn't want users to install Ollama or paste an API key. Apple's model was already on…

29
r/LocalLLaMA community 7h ago

I’m calling this the Monstrosity. 5 ex mining BC-250 boards Qwen3-Coder-Next Q4 at 40 tok/s

Using an asrock 12 unit case running one board as the main with the rest of them headless. About 71GB of vram exposed. So far 40 tok/s is with 30k context and it dips to around 30 tok/s at 100k context. This is all over the 1gb Ethernet that is on the boards already. I have 2…

36
r/LocalLLaMA community 11h ago

FT: Corporate America rejects overpriced frontier, embraces open models

  submitted by   /u/chocolateUI [link]   [comments]

17
r/LocalLLaMA community 11h ago

Qwen company already rushed out a Jev competitor. No open weights yet.

It's called decision-model-preview. There is only a docs page. No announcement or anything. I can't post a link because reddit's filters just deletes posts that contain a link to the cloud platform that hosts it. But you'll find the page if you Google the model name.  …

28
r/LocalLLaMA community 12h ago

Qwen plays World of Warcraft

Been doing a bunch of vibe coding lately. Had my agents host a private WoW server for me, then built out a web browser client so you can play without installing the game and it has mobile controls. Afterwards, created a custom mcp to drive the client and have finer game control…

5
r/LocalLLaMA community 13h ago

Qwen3.8-Flash-Next 177B NVFP4(119GiB): SSD streaming at 9-10 tok/s on one 16 GB RTX 5060 Ti + 32 GB RAM

We built an inference engine for MoE models that don't fit in VRAM + RAM. Most of the model stays on the SSD, and experts are read as tokens need them. This started as a proof of concept, and poc worked, we are getting 9-10 tok/s decode on Qwen3.8-Flash-Next NVFP4 (9.06 on the…

6
r/LocalLLaMA community 14h ago

Swift 1.5 Qwen3.8 27b (A must-have for low thinking!)

Just made this post for those who missed it : https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-27b UkisAI released their updated Qwen 27B (tuned for token efficiency). I grabbed the IQ4_XS quant to test against Unsloth's Q4_K_S: Low-thinking: UkisAI consistently beat Unsloth in…

9
r/LocalLLaMA community 14h ago

... so, yeah.

Finally got 3.8-Flash-Next running on my M4Pro 48GB Mac with https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF Dense 3.8-27B is just faster... and maybe better due to quantization level... EDIT: Hold a second, Flash-Next is actually performing faster than 27B…

20
r/LocalLLaMA community 14h ago

Imbalanced VRAM usage between two GPUs in llama.cpp. Anyone successfully solve this?

There is always at least 1+GB of VRAM not usable not matter how I set the --tensor-split (-ts) param. I tiny shift toward one side will move the weight significantly to the other side. 😵‍💫 Adjusting context will increase/decrease usage on both side. --tensor-split 499,501 =…

12
r/LocalLLaMA community 16h ago

Naive-N0.5-Flash - 309B-A15.5B

https://naive.ai/en/research/ Built for coding and AI R&D 1M context Hybrid SWA/DSA   submitted by   /u/nullmove [link]   [comments]

35
r/LocalLLaMA community 16h ago

Public MCP server for Canadian privacy law data (free, no auth) - works with any client that speaks Streamable HTTP

I built this, so the disclosure goes up front. It's a free, public MCP server plus a REST API with Canadian privacy law data. The MCP endpoint is at https://movahedi.ca/mcp and it uses Streamable HTTP, so any client that speaks that transport can connect. No signup, no API key,…

36
r/LocalLLaMA community 16h ago

Don't trust frontier models when asking about budget hardware!

Early this year when I was first looking at building up my inference capability you could get the 16GB Tesla P100s for between $60 and $80. Asked claude about it, told me absolutely not worth it. No tensor cores, bad int4/int8, no BF16, not worth it. Needs special power…

36
r/LocalLLaMA community 17h ago

Mimo v2.6 flash MOPD

  submitted by   /u/Automatic-Arm8153 [link]   [comments]

16
r/LocalLLaMA community 18h ago

Updated from 3x3090(2x3090, 1x3090TI) to 2x5090

Upgraded from 3x RTX 3090s to 2x RTX 5090s on my homelab server and picked up a solid speed jump on top of it from a software update (speculative decoding + NVFP4). Setup: llama.cpp (build b11216), running Qwen3.8-27B (abliterated, Q8_0). Blue = old 3090 setup, green = new 5090…

37
r/LocalLLaMA community 18h ago

is switching from llama cpp to vllm worth it

I have hp z8 g4 with 512 ram and 1x3090 1x5060 16gb. has anyone made the transition from llama cpp to vllm recently? is it worth it? docker under windows or full linux install? I am mainly interested in the model support, it seems that many new local models are supported day 0…

37
r/LocalLLaMA community 18h ago

Adding logit penalty for "wait", "maybe" and "perhaps" to Qwen models improves their accuracy

Meta came out with a banger paper https://arxiv.org/pdf/2606.00206 , but it did not look at various quantizations supported in llama.cpp. So I did a run on 50 random MATH-500 questions ( https://huggingface.co/datasets/HuggingFaceH4/MATH-500 ) and ran it on various quantizations…

10
r/LocalLLaMA community 19h ago

How do you guys give your models web browsing capabilities?

I am using the Deepseek Harness, which has webfetch plugins by default. While it can help browse the internet, I myself have to give it specific URLs to search. But, apparently you can give it the full web browser and search engine capabilities. Unfortunately, it apparently…

33
r/LocalLLaMA community 20h ago

We released VeriLoop E2 (27B, Apache-2.0). The design question behind it: should an LLM be allowed to commit its own state?

Disclosure: I’m one of the authors of VeriLoop E2, a 27B model post-trained from Qwen3.8-27B. The weights are Apache-2.0; the harness we evaluate it in is not open source (details at the bottom). This is a release post, but rather than a benchmark dump I want to talk about the…

28
r/LocalLLaMA community 20h ago

What if open-source AI focused less on giant models and more on reusable capabilities?

Instead of everyone building another general-purpose model, the community could distill open models into domain specialists—biology, Python, accounting, OCR, and more. Developers could combine these capabilities into local tools: small model + OCR + accounting → local accounting…

13
r/LocalLLaMA community 21h ago

Worth going from Qwen3.8 27B to flash next or maaybe deepseek v4 flash?

I am running qwen3.8 27b on my dual rtx 3090 (fp8 quant, unquantized cache, 129k context) and I think it works decently well with hermes, opencode etc. But! I am tempted by the new models coming out such as qwen3.8 flash next, deepseek v4 flash, glm 5.3 flash. However, there is…

20
r/LocalLLaMA community 21h ago

Zer0Fit - Zero-shot predictions, classifications, and regressions using Google ML research models running locally as a dockerized MCP

AI grad student here. With all the recent interest in Jev, I thought I would share something I built la few months ago that brings ML models and LLMs together in a different way than Jev does for different use cases. https://github.com/porespellar/Zer0Fit Background: A few…

16
r/LocalLLaMA community 22h ago

The Opus 5.5 posts about motion graphics are cool but Qwen 27B made this on a 4090

Saw the hundreds of tweets where people just keep asking Opus 5.5 for motion graphic videos. Decided to ask qwen to look at them and make its own. Quite amazing what local can achieve. **EDIT** It looks laggy because of reddits .gif limit btw the full high res version (with…

20
r/LocalLLaMA community 23h ago

Anyone with experience using dual SXM2 to PCIe card?

As per the title. My thinking is that if you run one of these then you would just need a computer with a pcie slot to be up and running. With an nvlink baseboard you need a separate PLX adapter card as well as the mini SAS cables from the baseboard connected up?   submitted…

22
r/LocalLLaMA community 1d ago

Nonobench v1.2: 43 LLMs on nonogram puzzles. Open-weight DeepSeek V4 Pro ties for 4th, and no open model solves the new 20×20 Hard mode

Follow-up to my January post: https://www.reddit.com/r/LocalLLaMA/comments/1q4i19c/benchmarking_23_llms_on_nonogram_logic_puzzle/ . That thread shaped v1.2: Reasoning effort is explicit per run Every prompt and output is public. All current top ranking private and open weight…

7
r/LocalLLaMA community 1d ago

Which of the 16gb VRAM qwen3.8 27b’s is the best?

I’m having a hard time finding out which one gives you fastest speed, maximum context with best possible quality. I can run unsloth qwen3.8 27b iq4_xs with 65k q8 kv, context without MTP and vision offloaded to cpu. But also kinda slow for agentic work at like 30ish tok/s )I…

30
r/LocalLLaMA community 1d ago

Another "Harness matters" post (codex cli > pi and opencode)

I run my own LLM while also having a Openai subscription. Also tried DeepSeek (latest flash now). I run Qwen 3.8 flash Next at an amazing speed on my 2x3090 + Ram! But local LLM never did worked for me outside some demos like build me a "3D Mario Game, multistage" which I've…

27
r/LocalLLaMA community 1d ago

5 items in MiMo-V2.6

3 visible models are 1T, 311B and 9B, let's dream about two more... ;)   submitted by   /u/jacek2023 [link]   [comments]

32
r/LocalLLaMA community 1d ago

Qwen, where's the small stuff? (1B/2B/4B)

I know Qwen is a key player in the local LLM space and has consistently introduced truly impactful technologies—like n-gram in Qwen-Next and the recent Qwen 3.8 27B, which is an amazing local model. However, my question is: why are we seeing fewer small-scale models lately—such…

24
r/LocalLLaMA community 1d ago

Splash fork optimised for M5 Max: ~1.5× faster (1.25× single request)

I’ve spent the last couple of days with Opus 5.5 working on a fork of Inco’s excellent and already blazingly fast Splash engine to optimise it for M5 Max chips. Taking liberties and referring to it as Splish. Roughly the opposite direction to u/Erp4759 ’s great M1 port ( Splash…

36
r/LocalLLaMA community 1d ago

When is the next generation of "B tier" models releasing?

The only things released in the last couple weeks seem to be bigger models like Qwen, DeepSeek, GLM, etc. Where are the Laguna's, Nemotrons, Olmo's, LongCats, Minimax, etc releases?   submitted by   /u/jazir55 [link]   [comments]

35
r/LocalLLaMA community 1d ago

SupersonicLabs/Julia-1 · Hugging Face

New open source Jev like model for running on local devices from a group called Supersonic Labs. It's a 144M param local non-generative local classifier. From their site : Julia 1 opens our research into compact decision models. It builds on mmBERT-small , a multilingual…

15
r/LocalLLaMA community 1d ago

Navigating Cost Efficient Hardware in these Volatile Times

Where to even begin on this one...I guess I should start by acknowledging the risk vs reward for vendors other than nvidia, so: Yes I understand that nvidia are dominant currently on speed (llm and imagegen) and software ecosystem. I am too am hopefuly that software stack…

32
r/LocalLLaMA community 1d ago

42x Faster Prompt Lookup Drafting in llama.cpp

  submitted by   /u/Available_Pressure47 [link]   [comments]

30
r/LocalLLaMA community 1d ago

Koboldcpp v1.122 released

  submitted by   /u/Fcking_Chuck [link]   [comments]

18
r/LocalLLaMA community 1d ago

Run Qwen3.8+Flash-Next and tiny models on Apple Silicon up to 3x faster

Maybe you'll like it? I hope I get to use my self-promotion credit a tiny little bit here after being in the community so long haha. I was the top of MLX.fast for a while and remain the winner on chips below M5. If you have capacity to contribute further enhancements I'd love…

10
r/LocalLLaMA community 1d ago

Improved and fixed template for GPT-OSS (again). Includes preserve_thinking and fix for Unsloth-induced bug

I posted an updated GPT-OSS template a couple of months ago , which was based on Unsloth's version . It turns out that both Unsloth's version (and, thus, mine) contain a very serious bug that can degrade the model when chat history is replayed and contains previous reasoning…

14
r/LocalLLaMA community 1d ago

I added Qwen-Image 2.1 + LoRA support to TensorSharp (GGUF, local inference)

I maintain TensorSharp , an open-source inference engine. It can now run Qwen-Image 2.1 locally for text-to-image generation and image editing, with support for its LoRA adapters. I’ve added configs for regular style and editing LoRAs, plus accelerated adapters with their own…

14
r/LocalLLaMA community 1d ago

Qwen 3.8 flash next is based on Qwen 4 architecture, if the announced Qwen 4 27b is also the same architecture with n-grams does it mean I can actually have faster inference on a single 3090 without tweaking much?

I wish Qwen also released dataset and method to fully train a model ourselves but it is what it is. However, I come here with my stupid question because someone can answer it better. And will the model still be an over thinker of faster inference will make up for that.  …

5
r/LocalLLaMA community 1d ago

New to playing around with local ai. why are they free?

I am new to playing around with local ai and have a ton to learn about it but I was curious, why are they (who is they?) releasing them for free, don't they want you to pay them to use them, why release free models?   submitted by   /u/poofph [link]   [comments]

20
r/LocalLLaMA community 1d ago

85 GB DeepSeek-V4-Flash at ~3 tok/s on a 12 GB RTX 3060 + 64 GB DDR5 RAM - Overspill for FreeToken, inspired by Colibri

I've been experimenting with ways to run MoE models that don't fit comfortably in RAM, and I ended up making Overspill, a disk tier for FreeToken . The basic idea came from looking at how Colibri handles experts across disk/RAM/VRAM so I took inspiration from the general…

11
r/LocalLLaMA community 1d ago

Splash 1.1.0 released, GGUF quants support, MLX import and more

On my M5 Pro 64GB I can comfortably work in an agentic setup with the Qwen3.8 27B model in good quality (Unsloth UD-Q4_K_XL) at a decent speed of 50 t/s. Splash combines optimized kernels, excellent speculative decoding, a well-implemented prefix cache, and mixed-weight support…

26
r/LocalLLaMA community 1d ago

Getting stupidly good results on my 4x3060ti setup.

About a month ago I was having FOMO and was going to spend coin I don't really have on new graphics cards. Instead of doing that though I decided to spend the money on a new motherboard+cpu and try to utilize my 3060tis I had lying around from my old crypto miner. Yes, I…

22
r/LocalLLaMA community 1d ago

Just bought a second 3090 but now I don't see the benefits right now.

Hi, My local AI server consists of 96GB Ddr5 and one rtx 3090. I got plenty of stuff running, like krea2, qwen image 2.1, minimax h3, ltx 2.5, qwen 3.6, qwen 3.8 q4...,got even qwen 3.8 Flash next running. But now I am thinking of what I can utilize the second card for. I am a…

37

NVIDIA shipped OpenShell, an open source sandbox that gives local and open agents real runtime limits instead of prompt rules. Over 100 firms joined the safety stack. OpenAI did not.

3090 for $1500???

modified qwen 3.8 27b modifies windows credential dumper to bypass EDR detection

Minisforum MS-S1 MAX-P495 @ €7.799,00

3090 NVLink bridges: are there clones of these? why are they so insanely expensive?

Liked Muse, so I cut the 30B model in half by width, distilled it back, and it does 57 of 60 tool tasks its parent does 60 of

GPT-3 is discontinued today

macOS 27 ships a free local LLM on Apple Silicon Macs. I made it easy to use from Node and Python

I’m calling this the Monstrosity. 5 ex mining BC-250 boards Qwen3-Coder-Next Q4 at 40 tok/s

FT: Corporate America rejects overpriced frontier, embraces open models

Qwen company already rushed out a Jev competitor. No open weights yet.

Qwen plays World of Warcraft

Qwen3.8-Flash-Next 177B NVFP4(119GiB): SSD streaming at 9-10 tok/s on one 16 GB RTX 5060 Ti + 32 GB RAM

Swift 1.5 Qwen3.8 27b (A must-have for low thinking!)

... so, yeah.

Imbalanced VRAM usage between two GPUs in llama.cpp. Anyone successfully solve this?

Naive-N0.5-Flash - 309B-A15.5B

Public MCP server for Canadian privacy law data (free, no auth) - works with any client that speaks Streamable HTTP

Don't trust frontier models when asking about budget hardware!

Mimo v2.6 flash MOPD

Updated from 3x3090(2x3090, 1x3090TI) to 2x5090

is switching from llama cpp to vllm worth it

Adding logit penalty for "wait", "maybe" and "perhaps" to Qwen models improves their accuracy

How do you guys give your models web browsing capabilities?

We released VeriLoop E2 (27B, Apache-2.0). The design question behind it: should an LLM be allowed to commit its own state?

What if open-source AI focused less on giant models and more on reusable capabilities?

Worth going from Qwen3.8 27B to flash next or maaybe deepseek v4 flash?

Zer0Fit - Zero-shot predictions, classifications, and regressions using Google ML research models running locally as a dockerized MCP

The Opus 5.5 posts about motion graphics are cool but Qwen 27B made this on a 4090

Anyone with experience using dual SXM2 to PCIe card?

Nonobench v1.2: 43 LLMs on nonogram puzzles. Open-weight DeepSeek V4 Pro ties for 4th, and no open model solves the new 20×20 Hard mode

Which of the 16gb VRAM qwen3.8 27b’s is the best?

Another "Harness matters" post (codex cli > pi and opencode)

5 items in MiMo-V2.6

Qwen, where's the small stuff? (1B/2B/4B)

Splash fork optimised for M5 Max: ~1.5× faster (1.25× single request)

When is the next generation of "B tier" models releasing?

SupersonicLabs/Julia-1 · Hugging Face

Navigating Cost Efficient Hardware in these Volatile Times

42x Faster Prompt Lookup Drafting in llama.cpp

Koboldcpp v1.122 released

Run Qwen3.8+Flash-Next and tiny models on Apple Silicon up to 3x faster

Improved and fixed template for GPT-OSS (again). Includes preserve_thinking and fix for Unsloth-induced bug

I added Qwen-Image 2.1 + LoRA support to TensorSharp (GGUF, local inference)

Qwen 3.8 flash next is based on Qwen 4 architecture, if the announced Qwen 4 27b is also the same architecture with n-grams does it mean I can actually have faster inference on a single 3090 without tweaking much?

New to playing around with local ai. why are they free?

85 GB DeepSeek-V4-Flash at ~3 tok/s on a 12 GB RTX 3060 + 64 GB DDR5 RAM - Overspill for FreeToken, inspired by Colibri

Splash 1.1.0 released, GGUF quants support, MLX import and more

Getting stupidly good results on my 4x3060ti setup.

Just bought a second 3090 but now I don't see the benefits right now.