r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 4d ago

Are there any modern completion (non-chat) models?

Are all modern LLMs tuned for chat? Are there any that do bare text completion? I honestly couldn't find any on hugging face.   submitted by   /u/MackThax [link]   [comments]

26
r/LocalLLaMA community 4d ago

New sampler + verifier *drastically* improves tiny 0.5b model coding performance

I read it with a little bit of effort The tiny model result is insane, theoretically this could make make a 0.5b on-par with a 2/3/4b ish class model in coding with no weights change*. And for large models it could maybe fix let's say 30-50% hallucination problems (educated…

21
r/LocalLLaMA community 5d ago

NVIDIA has released Nemotron-TwoTower-30B-A3B-Base-BF16, an unusual diffusion-based language model built from the Nemotron 3 Nano 30B-A3B backbone.

NVIDIA has released Nemotron-TwoTower-30B-A3B-Base-BF16, an unusual diffusion-based language model built from the Nemotron 3 Nano 30B-A3B backbone. Instead of generating strictly one token at a time, it uses a frozen autoregressive context tower plus a diffusion denoiser tower…

38
r/LocalLLaMA community 5d ago

USB4 RDMA seems doable

Just found this blog : https://blog.hellas.ai/blog/thunderbolt-ibverbs/ Experimental implementation of RDMA, demonstrated on two Strix Halo Did a quick search and can't really find it posted before ? This could be huge as it shoud work with any USB4 host.   submitted by…

26
r/LocalLLaMA community 5d ago

Has anyone tried to hack into their own system using a local model?

With all this talk about Mythos being able to hack into. US government systems, I was wondering if anyone has tried to get root on their own system using a local model?   submitted by   /u/MrMrsPotts [link]   [comments]

18
r/LocalLLaMA community 5d ago

Worse quality with MTP - Qwen 3.6, Gemma 4

Hi. I am self-hosting Qwen 3.6 27B Q8_K_XL with Llama.cpp on 4x5070ti. (All 4 cards are on single x16 slot bifurcated to 4x4 with risers). I've been testing it on several work repos with Opencode CLI and in like 8/10 situations the output of non-MTP model is far superior to the…

8
r/LocalLLaMA community 5d ago

GLM 5.2 on Dual Strix Halo (256GB): Worth it?

  submitted by   /u/Intrepid_Rub_3566 [link]   [comments]

21
r/LocalLLaMA community 5d ago

Could you help me test MTP for GLM-4.7-Flash?

Some of you may remember old models from GLM: GLM Air or GLM Flash. I know they’re outdated, but I have a soft spot for them, so I am currently working on enabling MTP for them in llama.cpp. If you know how to compile llama.cpp from source and have the hardware to run…

23
r/LocalLLaMA community 5d ago

If LLMs are so good at coding…

How come things like ROCm and the intel stack aren’t able to rapidly improve their software ecosystems to be a match for CUDA? Until the software from other vendors catches up with NVIDIA, they’re always going to get away with charging a massive premium on their “it just works”…

38
r/LocalLLaMA community 5d ago

Anthropic accuses Alibaba of campaign to ‘brazenly’ and ‘illicitly’ extract AI capabilities

https://www.cnbc.com/2026/06/24/anthropic-alibaba-distillation-campaign.html https://www.bloomberg.com/news/articles/2026-06-24/anthropic-accuses-alibaba-of-illicitly-accessing-its-ai-models   submitted by   /u/External_Mood4719 [link]   [comments]

10
r/LocalLLaMA community 5d ago

[NEW MODEL] SupraWeather-Nano-Preview Just released!

SupraWeather Nano is live! ⛈️ We just released SupraWeather-Nano (preview), a small FT-Transformer model purpose-built to classify weather phenomena from raw tabular meteorological features. https://huggingface.co/SupraLabs/SupraWeather-Nano-Demo https://huggingface.co/SupraLabs…

25
r/LocalLLaMA community 5d ago

Locked Dell quote for 6x RTX PRO 6000 Max-Q at $8,960 — expires tonight. What would you do?

Building an inference cluster to run GLM 5.2 locally. Got a Dell quote locked at $8,959.99/unit for 6x RTX PRO 6000 Blackwell Max-Q (300W). List price just jumped to $15,999 yesterday. Quote expires in ~3 hours and I can't swing all 6 right now. I have a second quote for 2 units…

36
r/LocalLLaMA community 5d ago

For users with 4x-8x 6000 PROs, how is your experience with bigger models lately? (GLM 5.2, Kimi 2.7, DeepSeek V4 Pro)

Hello guys, hoping you're doing fine! I was wondering, for users with 4x-8x 6000 PROs (so between 384 and 768GB VRAM), how are bigger models working for you? I have planned to either jump to 4 or 8 from my actual system, and want to see the experiences with these lately. In…

8
r/LocalLLaMA community 5d ago

Gemma4-26B-A4B & 31B-QAT Uncensored Balanced are out with MTP (35% & 53% speed boost)!

First of all, I'm stoked to announce we are almost at 20 million downloads on HF! (counted only on my own account, no duplicates/quants/finetunes/etc) and almost 5000 members on Discord! Two releases this time, as promised, the bigger Gemma 4 QATs, both Balanced, both with MTP :…

6
r/LocalLLaMA community 5d ago

Anybody used DwarfStar with DeepSeek V4 Flash on 1x DGX Spark yet? What are your thoughts?

Hey fellow localites, Has anybody used DwarfStar with DeepSeek V4 Flash on 1x DGX Spark yet? What are your thoughts? Based on what I read, with its MoE approach, and its unified memory first then bleed into SSD approach, you can apparently load DS4 Flash and it runs well, with…

31
r/LocalLLaMA community 5d ago

Any chance I could cluster my DGX Spark (128GB unified memory) and my AMD Ryzen AI Max 395 (128GM unified memory) together to run 1 model?

Hey all, So I have a Nvidia DGX Spark and an AMD Strix 395, both have 128GB of unified memory. The Spark has 200Gbit network and the AMD Strix has 5Gbit ethernet (but it has a pcie gen 4x4 slot). Is there any chance I can cluster the 2 together to run a larger model that can fit…

30
r/LocalLLaMA community 5d ago

Got GLM-5.2 + MTP speculative decode running on 4× DGX Spark (GB10) — and the build piece the public recipe is missing

TL;DR: the recipe's image-build mods aren't actually public – I reconstructed them from the public kernels (with Claude) – and you have to build vLLM at the author's exact pinned ref or the real AWQ weights crash on load. Running now at ~9.4 tok/s on my own 4× GB10. Saw a link…

20
r/LocalLLaMA community 5d ago

Gefen is a drop-in replacement for the AdamW optimizer, claims 8x memory reduction in training (GitHub available)

Paper: https://arxiv.org/abs/2606.13894 GitHub: https://github.com/ndvbd/Gefen   submitted by   /u/indicava [link]   [comments]

37
r/LocalLLaMA community 5d ago

Do cloud chatbot's system prompts make them stupider?

When I am talking with Chat GPT or Claude about abstract concepts, I am often surprised by how they seem kind of dumb... like they aren't benefitting from their extra parameters over top open models like Kimi or GLM. In fact they often seem stupider than these open models that I…

13
r/LocalLLaMA community 5d ago

SDXL running locally in the browser on WebGPU, open-source

I needed simple local image generation without the usual setup. No virtual environments, no ComfyUI with a complex graph and installation as an exe. So i tried to push the whole thing into the browser and run it on WebGPU. It's a browser extension. You install it, then it loads…

13
r/LocalLLaMA community 5d ago

MINISFORUM DEG1 Oculink eGPU Dock Refurbished - $59

I got one of these refurbished units last year. I have nothing but good things to say about it. It works great. It has heft to it to keep the GPU secure. And unlike some cheaper Oculink docks, it has redrivers for signal integrity.   submitted by   /u/fallingdowndizzyvr…

20
r/LocalLLaMA community 5d ago

I reverse engineered Windows Copilot into a free OpenAI compatible API (GPT-4, no API key, no billing)

So Microsoft gives you GPT-4 for free in Copilot. They just don't give you an API for it. So I made one. It logs into your own Microsoft account once, saves the session, and exposes a local server at http://localhost:8000/v1 that speaks the OpenAI format. Point the official…

24
r/LocalLLaMA community 5d ago

Has anyone else found vLLM outputs noticeably worse than llama.cpp for the same model?

I'm wondering if anyone else has come across this. I've tested the same model on llama.cpp and vLLM with similar settings and quantizations. The performance and concurrency in vLLM are much noticeably better, but sometimes the model feels less reliable. Some things I've noticed:…

27
r/LocalLLaMA community 5d ago

Sipp - an open-source library for in-browser inference built on llama.cpp

GitHub: https://github.com/noumena-labs/Sipp   submitted by   /u/lordhiggsboson [link]   [comments]

9
r/LocalLLaMA community 5d ago

Big News for AMD / Strix Halo+ Owners

Admittedly this is news for me, but I'm hoping it could be of some use to others here as well! So, THE NPU IS USABLE!! I've owned an AMD Ryzen 395 Max AI+ (or whatever the naming is lol) for about a year now and have relied solely on GGUFs and Vulkan. I acknowledge that the AMD…

19
r/LocalLLaMA community 5d ago

Build a LLM from Scratch using MLX

You probably have a burning desire to grasp the inner workings of LLMs. By now, terms like Attention, Transformers, and Tokenizers are likely ringing in your ears, yet the actual mechanics often feel like they slip away just as quickly as you study them. The truth is, the most…

7
r/LocalLLaMA community 5d ago

OpenAI and Broadcom unveil LLM-optimized inference chip

https://openai.com/index/openai-broadcom-jalapeno-inference-chip/ Quoted from the start of the blog post: Early testing shows that the first-generation accelerator will deliver performance per watt substantially better than current state-of-the-art Built from the ground up for…

11
r/LocalLLaMA community 5d ago

The Swiss Federal Supreme Court is evaluating Heretic

“Oh no, are they banning abliterated models now?!?” If that was your first thought when you read the title I can’t blame you. But that’s actually not what’s happening in this case. Instead, the Swiss Federal Supreme Court is evaluating Heretic for their own use! As it turns out,…

16
r/LocalLLaMA community 5d ago

I did some model hacks, and got GLM5.2 from about 2.5 tok/s to >50 tok/s on my GH200 system.

G'day. This is part 3 on my Local LLM adventures. I have a crazy system hacked server-to-desktop system : Component Spec GPUs 2x Hopper H100, 96 GB HBM3 each CPUs 2x Grace, 72 cores each Host memory 480 GB LPDDR5X per Grace, 960 GB total So I can run technically run GLM5.2.…

34
r/LocalLLaMA community 5d ago

The Bank of Korea just released a report about AI productivity

I am sorry for sharing an article from a Korean website that you might not be familiar with. But South Korea is the only country currently making a lot of money from the AI boom. BigTech in the USA are paying huge amounts of money to buy semiconductor chips from Samsung and SK…

27
r/LocalLLaMA community 5d ago

Nex-N2-Mini-Ultra-Uncensored-Heretic Is Out Now, an Agentic Model With Agentic Thinking Now Uncensored With 5/100 Refusals and 0.0020 KLD, Available in Safetensors and GGUF Formats!

Safetensors: https://huggingface.co/llmfan46/Nex-N2-mini-ultra-uncensored-heretic GGUFs: https://huggingface.co/llmfan46/Nex-N2-mini-ultra-uncensored-heretic-GGUF Find all my models here: HuggingFace-LLMFan46 If you like my work and find my models useful, then I would really…

36
r/LocalLLaMA community 5d ago

Qwen-AgentWorld-35B-A3B for Coding?

Benchmark from its model card. Removed online models & Qwen-AgentWorld-397B-A17B from the table. Just Open models. Model MCP Search Term. SWE Android Web OS Overall DeepSeek-V4-Pro 63.27 27.61 51.26 59.44 55.17 50.32 63.70 52.97 GLM-5.1 67.60 22.46 47.32 52.07 59.10 51.50 59.13…

11
r/LocalLLaMA community 5d ago

llama.cpp's web UI now supports executing model generated JavaScript in the browser, through Web Workers (opt in)

A pull request adding a new run_javascript tool was merged into mainline a couple of weeks ago. I could not find any discussion about it here, or elsewhere for that matter. Maybe this has gone largely unnoticed (or maybe I suck at searching)? The feature does not seem to have…

24
r/LocalLLaMA community 5d ago

How Baidu's newly released Unlimited-OCR transcribes dozens of pages in one forward pass

https://i.redd.it/zjduf8zns79h1.gif Baidu released Unlimited-OCR 2 days ago, and they claim it can transcribe dozens of pages in one forward pass. I read the research paper, and decided to make a post ( link if anyone's interested) Problem they are solving The problem it targets…

11
r/LocalLLaMA community 5d ago

Qwen3.6 27B more dumb in vLLM compared to llama.cpp

Hello, I recently bought a new RTX 5060Ti to pair with the RTX 5060Ti I already own, now I have 32GB of VRAM. Up until now for convenience I've used llama.cpp, for goodness' sake it works excellently when only 1 user is using it, but now there are 2 of us using it and llama.cpp…

34
r/LocalLLaMA community 5d ago

llama.cpp updates - granite-speech-4.1-2b, LFM2.5-ColBERT/Embedding-350M, Vulkan backend related changes & Misc items

Supported Models : granite-speech-4.1-2b-plus by 24818 LFM2.5-ColBERT-350M & LFM2.5-Embedding-350M by 24913 Vulkan : vulkan: link ggml-cpu when GGML_VULKAN_CHECK_RESULTS / RUN_TESTS are enabled #24444 vulkan: make mul_mm ALIGNED a spec constant #24689 vulkan: support CONV_3D…

27
r/LocalLLaMA community 6d ago

New EU model (Domyn) will be 400b.

The source is in Italian, but a well respected newspaper (like Financial Times) https://www.ilsole24ore.com/art/frontier-grand-challenge-domyn-guidera-progetto-dell-ai-sovrana-AIgNTNoD?refresh_ce=1 They are a startup that has already created a closed 260b model (Domyn Large) for…

38
r/LocalLLaMA community 6d ago

PCIE 5.0 16x split into 2x8 with riser cable

Hey guys Thanks in advance for your help and knowledge! My setup is born out of the parts I had at hand. Wanting to maximise VRAM with an RTX 4070 that I had in another system that I only used once or twice a year. So right now my system is 14600kf, 32gb RAM and rtx 5070ti 16gb…

34
r/LocalLLaMA community 6d ago

Qwen-AgentWorld-397B-A17B

It looks like a new model, mentioned on https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B and on https://qwen.ai/blog?id=qwen-agentworld   submitted by   /u/Shoddy_Bed3240 [link]   [comments]

18
r/LocalLLaMA community 6d ago

Unlimited-OCR is now on ModelScope! A 3.3B multilingual OCR model for one-shot parsing across single images, multi-page documents, and PDFs. License: MIT

Full-document parsing instead of cropped-region OCR 32K output length for long OCR sequences Base and gundam image modes for different document layouts Transformers inference + SGLang serving with OpenAI-compatible streaming requests Built to push DeepSeek-OCR-style document…

22
r/LocalLLaMA community 6d ago

Qwen-AgentWorld-35B-A3B: a 3B-active MoE trained to simulate MCP, terminal, SWE, Android, web and OS environments

Qwen just released Qwen-AgentWorld-35B-A3B — a 35B-parameter MoE with only ~3B active parameters per token. The interesting part: this is not positioned as a standard chat/instruction model or a full autonomous agent. It is a language world model trained to predict what an…

6
r/LocalLLaMA community 6d ago

GitHub - QwenLM/Qwen-AgentWorld: Qwen-AgentWorld: Language World Models for General Agents

  submitted by   /u/dan945 [link]   [comments]

5
r/LocalLLaMA community 6d ago

Speaking of those chinese chips... "Chinese supercomputer displaces US machines as world's fastest for first time since 2017"

  submitted by   /u/johnnyApplePRNG [link]   [comments]

25
r/LocalLLaMA community 6d ago

Seems this community might have missed it: Bill that would mandate AI chip location tracking gains industry support | Half a dozen companies have come out in support of the Chip Security Act, which would require location-tracking mechanisms for America’s most advanced computing…

Web / reddit search have not found this posted in this sub, even though it is several days old news. So I do post. Related links: https://www.reddit.com/r/politics/comments/1uahgcs/bill_that_would_mandate_ai_chip_location_tracking/…

9
r/LocalLLaMA community 6d ago

Mimo 2.5 is _fast_ at large context (dual RTX Pro 6000)

For agentic work fast high context is king, OpenCode fills the window quickly and most models that feel snappy at 8k context turn into dial-up ADSL brrr by the time you're at 150k context deep. So I've been testing lots of models and runners trying to get "local Sonnet" on 2x…

14
r/LocalLLaMA community 6d ago

What are the top Chinese GPU rental platforms?

This post has me intrigued ... but not to buy, I want to rent/lease one of these FRANKNVIDIA GPUs. I'll learn Chinese. I'll VPN in through the great firewall on the backs of carrier pigeons if I have to. I don't care. Where's the vast.ai of China at?   submitted by  …

10
r/LocalLLaMA community 6d ago

MiniMax2.7 @47tg 1200pp

MiniMax 2.7 REAP Q4 on 96GB VRAM and 192 GB DDR5 udimm ram on a b840 MSI board and 9900X cpu. 1250W PSU and all cards are power limited. Linux Ubuntu. Agent class model. Excellent instruction following and tool calling. I run this model in a round robin loop with 3 sequencing…

19
r/LocalLLaMA community 6d ago

I want to add a second 7900XTX, question about pcie2/3/4

I've got a 7900XTX in my old gaming PC, now I want more vram and if I stay on one GPU I can only reasonably get 32GB and that just doesn't sound good enough. Using two slots, 48GB sounds way better and is much cheaper. I think 48GB is the minimum I want to have after going to…

26
r/LocalLLaMA community 6d ago

Tmax-27b - a Qwen3.6-27b terminal agent for small GPUs trained with DPPO (RL)

Hey everyone, wanted to share some work on making the new Tmax-27B terminal agent actually runnable on consumer hardware. What is Tmax-27B? Ai2 just released Tmax, a family of terminal-agent LLMs trained with DPPO (RL) on top of Qwen3.6. The 27B model hits ~43% on Terminal Bench…

32
r/LocalLLaMA community 6d ago

OpenMythos benchmarks

Hey everyone! OpenMythos benchmarks are finally here sorry it took about a week to post these. The delay was mainly because SWE-bench results weren't matching up with Qwen 3.6 27B official numbers. Turns out Qwen used a different eval harness and also refined/filtered the…

12

Are there any modern completion (non-chat) models?

New sampler + verifier *drastically* improves tiny 0.5b model coding performance

NVIDIA has released Nemotron-TwoTower-30B-A3B-Base-BF16, an unusual diffusion-based language model built from the Nemotron 3 Nano 30B-A3B backbone.

USB4 RDMA seems doable

Has anyone tried to hack into their own system using a local model?

Worse quality with MTP - Qwen 3.6, Gemma 4

GLM 5.2 on Dual Strix Halo (256GB): Worth it?

Could you help me test MTP for GLM-4.7-Flash?

If LLMs are so good at coding…

Anthropic accuses Alibaba of campaign to ‘brazenly’ and ‘illicitly’ extract AI capabilities

[NEW MODEL] SupraWeather-Nano-Preview Just released!

Locked Dell quote for 6x RTX PRO 6000 Max-Q at $8,960 — expires tonight. What would you do?

For users with 4x-8x 6000 PROs, how is your experience with bigger models lately? (GLM 5.2, Kimi 2.7, DeepSeek V4 Pro)

Gemma4-26B-A4B & 31B-QAT Uncensored Balanced are out with MTP (35% & 53% speed boost)!

Anybody used DwarfStar with DeepSeek V4 Flash on 1x DGX Spark yet? What are your thoughts?

Any chance I could cluster my DGX Spark (128GB unified memory) and my AMD Ryzen AI Max 395 (128GM unified memory) together to run 1 model?

Got GLM-5.2 + MTP speculative decode running on 4× DGX Spark (GB10) — and the build piece the public recipe is missing

Gefen is a drop-in replacement for the AdamW optimizer, claims 8x memory reduction in training (GitHub available)

Do cloud chatbot's system prompts make them stupider?

SDXL running locally in the browser on WebGPU, open-source

MINISFORUM DEG1 Oculink eGPU Dock Refurbished - $59

I reverse engineered Windows Copilot into a free OpenAI compatible API (GPT-4, no API key, no billing)

Has anyone else found vLLM outputs noticeably worse than llama.cpp for the same model?

Sipp - an open-source library for in-browser inference built on llama.cpp

Big News for AMD / Strix Halo+ Owners

Build a LLM from Scratch using MLX

OpenAI and Broadcom unveil LLM-optimized inference chip

The Swiss Federal Supreme Court is evaluating Heretic

I did some model hacks, and got GLM5.2 from about 2.5 tok/s to >50 tok/s on my GH200 system.

The Bank of Korea just released a report about AI productivity

Nex-N2-Mini-Ultra-Uncensored-Heretic Is Out Now, an Agentic Model With Agentic Thinking Now Uncensored With 5/100 Refusals and 0.0020 KLD, Available in Safetensors and GGUF Formats!

Qwen-AgentWorld-35B-A3B for Coding?

llama.cpp's web UI now supports executing model generated JavaScript in the browser, through Web Workers (opt in)

How Baidu's newly released Unlimited-OCR transcribes dozens of pages in one forward pass

Qwen3.6 27B more dumb in vLLM compared to llama.cpp

llama.cpp updates - granite-speech-4.1-2b, LFM2.5-ColBERT/Embedding-350M, Vulkan backend related changes & Misc items

New EU model (Domyn) will be 400b.

PCIE 5.0 16x split into 2x8 with riser cable

Qwen-AgentWorld-397B-A17B

Unlimited-OCR is now on ModelScope! A 3.3B multilingual OCR model for one-shot parsing across single images, multi-page documents, and PDFs. License: MIT

Qwen-AgentWorld-35B-A3B: a 3B-active MoE trained to simulate MCP, terminal, SWE, Android, web and OS environments

GitHub - QwenLM/Qwen-AgentWorld: Qwen-AgentWorld: Language World Models for General Agents

Speaking of those chinese chips... "Chinese supercomputer displaces US machines as world's fastest for first time since 2017"

Seems this community might have missed it: Bill that would mandate AI chip location tracking gains industry support | Half a dozen companies have come out in support of the Chip Security Act, which would require location-tracking mechanisms for America’s most advanced computing…

Mimo 2.5 is _fast_ at large context (dual RTX Pro 6000)

What are the top Chinese GPU rental platforms?

MiniMax2.7 @47tg 1200pp

I want to add a second 7900XTX, question about pcie2/3/4

Tmax-27b - a Qwen3.6-27b terminal agent for small GPUs trained with DPPO (RL)

OpenMythos benchmarks

New sampler + verifier drastically improves tiny 0.5b model coding performance