r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 11d ago

EvoTensile: Evolutionary algorithms for AMD Tensile GEMM kernel tuning

There has been an effort to tune kernels in hipBLASLt so the most basic matmuls can run faster. It's known that on Strix Halo (gfx1151), GEMM with NN and TN input layouts (used in inference) are already well-tuned, while NT and TT layouts (used in training) are not yet tuned.…

8
r/LocalLLaMA community 11d ago

GLM-5.2 can now run locally in llama.cpp and Unsloth Studio.

The 2-bit model retains ~82% accuracy after we shrunk it from 1.51TB to 238GB (-84% size). Run on a 256GB Mac or RAM/VRAM setups. GLM-5.2 is the strongest open model to date. Check the graph for the accuracy of each GLM-5.2-GGUF quantization. Full guide:…

35
r/LocalLLaMA community 11d ago

Little late thank you to the DeepSeek team!

7 moths ago I posted https://www.reddit.com/r/LocalLLaMA/s/Z32skdSKzY Just wanted to thank you for DeepSeek V4 Pro and extra big Thank You for the Flash version that fits on my local hardware! Thank You!!!!   submitted by   /u/Sorry_Ad191 [link]   [comments]

37
r/LocalLLaMA community 11d ago

[NEW MODEL] SupraLabs just released SupraVL-Nano-900k, a Vision-Language Model built entirely from scratch!

Hey r/LocalLLaMA ! We just released SupraVL-Nano-900k , our first VLM. It has ~900k parameters, was trained from scratch on Flickr8k, and the entire architecture fits in a single Jupyter notebook. This is not a production model, it's a fully transparent, readable blueprint for…

27
r/LocalLLaMA community 11d ago

SETI @ Home aka distributed LLM inference engine. Does this exist and if not, should we make one?

This seems logical for the benefit of civilisation. I have a 5 GPU system to contribute.   submitted by   /u/HockeyDadNinja [link]   [comments]

26
r/LocalLLaMA community 11d ago

2 weeks since the release of Gemma 4 12b Unified, how are we feeling about it?

I'm looking for a good model to run on a 5090 and have ample context ~128k. This model looks good for me, it seems to have good performance in the 12b range, almost comparable to Gemma 4 26B A4B. Building a custom harness for it and have ~300m of tokens to fine tune on. Do you…

6
r/LocalLLaMA community 11d ago

GLM-5.2 is above GPT-5.5 in AA-Briefcase, Artificial Analysis' new agentic knowledge work eval

  submitted by   /u/analysis_scaled [link]   [comments]

7
r/LocalLLaMA community 11d ago

GLM-5.2 (744B, 2-bit) at 7.3 tok/s on 4×3090 + 192GB — and why IQ1_M wasn't any faster

TLDR: For the first time, I feel relief that they could shut down the cloud services and I would be ok. I got my 4th 3090 and then unsloth dropped the Q2 and Q1. I wrote nothing else here its from CC, so it might be wrong. GLM-5.2 UD-IQ2_M runs across 4×3090 + RAM expert offload…

7
r/LocalLLaMA community 11d ago

LQ50/LQ50-24GB cost around $1200

Well found this shit on TAOBAO very expensive   submitted by   /u/MundanePercentage674 [link]   [comments]

12
r/LocalLLaMA community 11d ago

DiffusionGemma 26b on a 4090 at up to 475t/s... and some thoughts...

Figured I'd post up a bit of info for anyone else who was thinking about messing with this model on a 3090/4090. Obviously I can't use the nvfp4, but I got it up and running in vLLM using diffusiongemma-26B-A4B-it-AWQ-INT4. Had to run it in a custom vLLM docker they provide for…

34
r/LocalLLaMA community 11d ago

Giving GLM-5.2 a spin locally on CPU only! (poor man's rig for big models)

This is the UD-Q2-K_XL quant. Hardware is: Model: Dell PowerEdge R740 CPU: Dual Xeon 6248R (24 cores each) RAM: 768 GB (All memory channels populated) I'm using ik_llama.cpp which provides some significant performance improvements over the base llama.cpp for CPU-only inference.…

5
r/LocalLLaMA community 11d ago

I have an old multi-GPU node lying around at work...

My employer has a GPU node that is mostly sitting idle. It contains 8 NVIDIA Quadro RTX 6000 GPUs with a total of 192 GB VRAM, and 512 GB RAM, and approximately 112 CPU threads to play with. I want to suggest we repurpose it for local inference. I need to make a case for this…

15
r/LocalLLaMA community 11d ago

Local Qwen isn't a worse Opus, it's a different tool

  submitted by   /u/cafedude [link]   [comments]

37
r/LocalLLaMA community 11d ago

Updates on North Mini Code: 4 bit quant + Ollama + OpenRouter

Hey! We heard the feedback on making the model more portable and accessible. So in light of that we have 2 updates to share. First, you can pull a new 4-bit quant straight from Hugging Face , so it’s now small enough to run on a Mac or whatever local hardware you’ve got. It…

21
r/LocalLLaMA community 11d ago

LFM2.5-Embedding-350M & LFM2.5-ColBERT-350M

LFM2.5-Embedding-350M is a dense bi-encoder for fast multilingual retrieval. It produces a single vector per document — the smallest, fastest index — for reliable cross-lingual search across 11 languages. Best-in-class multilingual accuracy for a dense embedder of its size.…

27
r/LocalLLaMA community 11d ago

poolside/Laguna-M.1 · Hugging Face - 225B-A23B

Laguna M.1 Laguna M.1 is a 225B total parameter Mixture-of-Experts model with 23B activated parameters per token designed for agentic coding and long-horizon work. Highlights Large sparse MoE for agentic coding : Laguna M.1 is a 70-layer MoE transformer with 225B total…

26
r/LocalLLaMA community 11d ago

Cutting LLM Token Costs with rtk, headroom, and caveman - savings measured on real workloads

rtk , headroom , and caveman keep showing up whenever someone posts about cutting their token bill 60-90%. I wanted to know what they save on an actual bill instead of a benchmark, so I replayed all three over my own Claude Code history. My corpus was 500 of my own Claude Code…

11
r/LocalLLaMA community 11d ago

My suitcase robot gets high now off a real gas sensor wired straight into the LLM sampler. Smoke raises temperature/top_p/top_k live, so his speech genuinely gets loopier and never repeats.

Follow-up on Sparky, my offline suitcase robot I keep overdeveloping. He gets high now, and there's no scripted "stoned mode" anywhere in it. A real MQ-2 gas sensor sits in the case. Every 0.5s I read it against an adaptive clean-air baseline and turn a smoke hit into a 0 to 10…

30
r/LocalLLaMA community 11d ago

GLM-5.2 inference is free on Hugging Face for the next 6 hours

doc: https://huggingface.co/docs/inference-providers/index a cool prompt to try first: https://huggingface.co/chat/r/aFATtCW?leafId=ed28d5b0-d99b-40be-ba8b-315b1f450e5a   submitted by   /u/paf1138 [link]   [comments]

32
r/LocalLLaMA community 11d ago

GLM-5.2 Is The Best Open Weight Creative Writing Model

As Per Sam Paech's Creative Writing Benchmark on EQ Bench: https://eqbench.com/creative_writing.html   submitted by   /u/Few_Painter_5588 [link]   [comments]

24
r/LocalLLaMA community 11d ago

unsloth GLM-5.2-GGUF , including 2bit at 238GB

  submitted by   /u/okaycan [link]   [comments]

16
r/LocalLLaMA community 11d ago

What's the best place to sell a barely-used RTX PRO 6000 Blackwell Max-Q (96GB)?

Bought an RTX PRO 6000 Blackwell Max-Q for local inference, but honestly it’s massive overkill for what I actually need. Turns out I just want a big model loaded that I can occasionally ping for thoughts — speed doesn’t matter to me. No training, no heavy coding workloads (I’ve…

16
r/LocalLLaMA community 11d ago

the power of intelligence is better in the hands of the people than in the board rooms of tycoons.

Hey [ r/localllama ]( r/localllama ). I wanted to share what's new with our open source PearlOS project since you all last saw (90 days ago). But first I want to give a massive thank you to this community, both your feedback and support were essential in getting us this far.…

22
r/LocalLLaMA community 11d ago

GLM's founder says GLM-fable before the end of the year?!

  submitted by   /u/Charuru [link]   [comments]

23
r/LocalLLaMA community 11d ago

OSS models decisively overtook Proprietary models in market share (based on the last 3 months of OpenRouter data)

  submitted by   /u/Comfortable-Rock-498 [link]   [comments]

8
r/LocalLLaMA community 11d ago

NVFP4 kv cache quantization on sm120 will make 32GB VRAM systems very capable

The best i can get from Qwen3.6-27B on my 32GB VRAM (2 x 5060) is ~60 tok/sec gen speed at context size 196608. (sakamakismile text nvfp4). Fp8 kv quantization. NVFP4 kv cache quantization can’t get here fast enough. Reminds me of the time there was this game i couldn’t play on…

38
r/LocalLLaMA community 12d ago

gave my local llm agent mcp tools for local image + video gen, so it just generates when i ask (fully offline+free)

free and open source, runs fully offline. the local llm agent does the image and video gen itself via mcp tools. details and github in the comments.   submitted by   /u/GroundbreakingMall54 [link]   [comments]

33
r/LocalLLaMA community 12d ago

Le Chaton Fat Flash local when?

We are very happy with Le Chaton Fat SOTA but most of us would like to run it locally. You know, for privacy and sovereignty reasons. Does anyone have any updates when a local "flash" or "small" version is available?   submitted by   /u/corpo_monkey [link]  …

31
r/LocalLLaMA community 12d ago

Kwai-Keye/Keye-VL-2.0-30B-A3B-GGUF · Hugging Face

Meet Keye-VL-2.0-30B-A3B — the latest 30B-class flagship base model in the Keye series, purpose-built to push the frontier of long-video understanding and to unlock the first generation of Agent capabilities in the Keye family. Highlights Outstanding Video Understanding and…

29
r/LocalLLaMA community 12d ago

GLM-5.2 Flash when? (joke)

I'm very happy with Z.ai's decision to open source GLM 5.2... With that being said, a successor to GLM-4.7-flash would be AMAZING. Literally anything in the 27-120B range (MoE or dense) 🤤   submitted by   /u/ILoveToyota37 [link]   [comments]

29
r/LocalLLaMA community 12d ago

Does anyone have enough compute to make a distillation dataset out of GLM5.2?

Same as title. Some lucky ppl among us have massive amounts of compute and can run even GLM 5.2. Can someone plss make a BIG distillation dataset (eg 700k-1M examples) so that we can train smaller models like Qwen3.5 properly on it and have better models? It would be amazing for…

28
r/LocalLLaMA community 12d ago

I have a M5 Max MacBook Pro with 128gb of ram, what models should I run on it?

Yes I know this is a simple question I could just ask Claude or something but I want to see what the community suggests For context it’s a 16in MacBook Pro and i use Hermes agent as a harness connected to LM studio as obviously it’s preferable to be running MLX models especially…

4
r/LocalLLaMA community 12d ago

LocalLLaMA crowdsourced coding dataset

I feel like many people in this community (myself included) are constantly, eagerly awaiting new small model releases, or improvements to existing models, etc. Sometimes I wish there were more community-released models (similarly to how there are sometimes community-released…

20
r/LocalLLaMA community 12d ago

Quick thoughts on GLM-5.2 (Bonus: Censorship question answers)

I've been working with GLM-5.2 pretty much non-stop since it was released as an API. So yeah, take it with a grain of salt as API inference is not perfectly controllable. I'm calling it through Z.ai - so I'd like to think that it's a high quality iteration of the model, but I…

27
r/LocalLLaMA community 12d ago

price rising effect is wild..

https://preview.redd.it/6f2gghbgqy7h1.png?width=2980&format=png&auto=webp&s=eacde26d8d0154aabc9884d4c31607aa12ace68c Q.01 out soon? i dont really need precision after all..   submitted by   /u/zakadit [link]   [comments]

32
r/LocalLLaMA community 12d ago

CEOs of Anthropic and Google DeepMind call for U.S.-led AI coalition in meeting at G7

https://www.cnbc.com/2026/06/17/anthropic-amodei-google-hassabis-us-ai-coalition-g7.html https://www.politico.eu/article/ai-artificial-intelligence-anthropic-china-g7/   submitted by   /u/External_Mood4719 [link]   [comments]

32
r/LocalLLaMA community 12d ago

What have you been working on lately?

This subreddit is full of tinkerers, but all I see is what tools everyone is using, and not what they're using them for. So I'm curious: what's everyone working on?   submitted by   /u/Sufficient-Scar4172 [link]   [comments]

10
r/LocalLLaMA community 12d ago

Leaked financial docs show OpenAI is losing billions of dollars a year

  submitted by   /u/johnnyApplePRNG [link]   [comments]

16
r/LocalLLaMA community 12d ago

llama.cpp now supports model management (downloading etc) via API

#23976 got merged a couple hours ago, which means llama.cpp can now not only load/unload models on demand from a directory, but also download them on demand. No UI yet, but that's coming pretty soon. This means you can now deploy llama.cpp, expose the API, and manage the…

6
r/LocalLLaMA community 12d ago

I released Inflect-Nano, an ultra-extreme tiny 4.63m parameter TTS model.

I’ve been experimenting with how small a usable neural TTS model can realistically get, and I just released Inflect-Nano-v1 . As far as I researched (though I could be wrong on this), Inflect-Nano-v1 is the #2 smallest TTS model publicly released (after TinyTTS) , and it…

24
r/LocalLLaMA community 12d ago

Lin Junyang AI Lab Closes Round at $2B Valuation

A new lab from Lin Junyang can only be good news for open source / weights, I think. Excited to see what the lead responsible for the Qwen line does next.   submitted by   /u/rmhubbert [link]   [comments]

38
r/LocalLLaMA community 12d ago

GLM 5.2 Release Video [Made with GLM 5.2]

Everyone's probably seen the remotion thing that went viral a couple months back with CC. Its basically that with GLM 5.2 as the model provider. Close to Fable but still a step below on creativity, top is still Gemini 3.1 pro for vid creation but at least I can see why Design…

21
r/LocalLLaMA community 12d ago

We need a 80-160B model urgently. The unified memory device market needs more Models.

Hello guys, I will keep myself short. There are so many people that have a lot but not enough of "slow" RAM. Anybody with a Apple Device with >96GB Anybody with a Ryzen AI 395 Device with >96GB Anybody with a DGX Spark Even people with RTX 6000 Pros or 4x3090s or other…

31
r/LocalLLaMA community 12d ago

Lemonade v10.8: auto memory management, cloud offload, Omni improvements, and call your local models as MCP tools

v10.8 is out, so here's a project update on what landed. This was a 20-contributor release in just 7 days! Smarter memory and context management Dynamic VRAM management now auto-unloads idle models and downsizes their KV-cache to reclaim GPU memory on the fly, plus model pinning…

27
r/LocalLLaMA community 12d ago

US holds off blacklisting China's DeepSeek, more than 100 firms deemed security risks, sources say

  submitted by   /u/zxyzyxz [link]   [comments]

15
r/LocalLLaMA community 12d ago

PSA: unsloth/GLM-5.2-GGUF is uploading

Went to check Unsloth's HF to see if they uploaded GLM-5.2 GGUFs, and found the repo was created half an hour ago. It only has the readme for now. I suspect GGUFs are uploading   submitted by   /u/FullstackSensei [link]   [comments]

15
r/LocalLLaMA community 12d ago

i post-trained a model to reliably roll a die

lots of talk about agi, asi, rsi but ask any frontier LLM to roll a die and it will almost always say "4." claude, gpt, kimi - doesn't matter, 4.4.4.4. that sounds silly, but I think it’s actually a nice toy problem for one of the most interesting issues in rl: getting a model…

38
r/LocalLLaMA community 12d ago

llama.cpp - how to free up even more space on your GPU

For the past week or two, llama.cpp has been working much better from the RAM usage prespective. I no longer see any memory leaks, and everything fits nicely on the GPU - my defaults are --n-gpu-layers 99 --no-mmap --mlock to avoid using the regular RAM, since I use my 3090 with…

34
r/LocalLLaMA community 12d ago

My GLM-5.2-FP8 HGX-H200 SGLang docker deploy config

Halo lads. Name says it all. Right now, after 1-2 hours of experimenting, this is maximum i could squeeze out current hardware No, im not rich. Its my companies GPUs, just sharing my experience docker run -d \ --name glm-5.2-sglang \ --restart unless-stopped \ --gpus all \…

23
r/LocalLLaMA community 12d ago

Multilingual-Multimodal-NLP/LoopCoder-V2 · Hugging Face

GitHub : https://github.com/CSJianYang/LoopCoder arXiv : https://arxiv.org/abs/2606.18023 Full Paper PDF : https://arxiv.org/pdf/2606.18023 LoopCoder-V2 LoopCoder-v2 is a 7B instruction-tuned code model based on the Parallel Loop Transformer (PLT). The model studies test-time…

37

EvoTensile: Evolutionary algorithms for AMD Tensile GEMM kernel tuning

GLM-5.2 can now run locally in llama.cpp and Unsloth Studio.

Little late thank you to the DeepSeek team!

[NEW MODEL] SupraLabs just released SupraVL-Nano-900k, a Vision-Language Model built entirely from scratch!

SETI @ Home aka distributed LLM inference engine. Does this exist and if not, should we make one?

2 weeks since the release of Gemma 4 12b Unified, how are we feeling about it?

GLM-5.2 is above GPT-5.5 in AA-Briefcase, Artificial Analysis' new agentic knowledge work eval

GLM-5.2 (744B, 2-bit) at 7.3 tok/s on 4×3090 + 192GB — and why IQ1_M wasn't any faster

LQ50/LQ50-24GB cost around $1200

DiffusionGemma 26b on a 4090 at up to 475t/s... and some thoughts...

Giving GLM-5.2 a spin locally on CPU only! (poor man's rig for big models)

I have an old multi-GPU node lying around at work...

Local Qwen isn't a worse Opus, it's a different tool

Updates on North Mini Code: 4 bit quant + Ollama + OpenRouter

LFM2.5-Embedding-350M & LFM2.5-ColBERT-350M

poolside/Laguna-M.1 · Hugging Face - 225B-A23B

Cutting LLM Token Costs with rtk, headroom, and caveman - savings measured on real workloads

My suitcase robot gets high now off a real gas sensor wired straight into the LLM sampler. Smoke raises temperature/top_p/top_k live, so his speech genuinely gets loopier and never repeats.

GLM-5.2 inference is free on Hugging Face for the next 6 hours

GLM-5.2 Is The Best Open Weight Creative Writing Model

unsloth GLM-5.2-GGUF , including 2bit at 238GB

What's the best place to sell a barely-used RTX PRO 6000 Blackwell Max-Q (96GB)?

the power of intelligence is better in the hands of the people than in the board rooms of tycoons.

GLM's founder says GLM-fable before the end of the year?!

OSS models decisively overtook Proprietary models in market share (based on the last 3 months of OpenRouter data)

NVFP4 kv cache quantization on sm120 will make 32GB VRAM systems very capable

gave my local llm agent mcp tools for local image + video gen, so it just generates when i ask (fully offline+free)

Le Chaton Fat Flash local when?

Kwai-Keye/Keye-VL-2.0-30B-A3B-GGUF · Hugging Face

GLM-5.2 Flash when? (joke)

Does anyone have enough compute to make a distillation dataset out of GLM5.2?

I have a M5 Max MacBook Pro with 128gb of ram, what models should I run on it?

LocalLLaMA crowdsourced coding dataset

Quick thoughts on GLM-5.2 (Bonus: Censorship question answers)

price rising effect is wild..

CEOs of Anthropic and Google DeepMind call for U.S.-led AI coalition in meeting at G7

What have you been working on lately?

Leaked financial docs show OpenAI is losing billions of dollars a year

llama.cpp now supports model management (downloading etc) via API

I released Inflect-Nano, an ultra-extreme tiny 4.63m parameter TTS model.

Lin Junyang AI Lab Closes Round at $2B Valuation

GLM 5.2 Release Video [Made with GLM 5.2]

We need a 80-160B model urgently. The unified memory device market needs more Models.

Lemonade v10.8: auto memory management, cloud offload, Omni improvements, and call your local models as MCP tools

US holds off blacklisting China's DeepSeek, more than 100 firms deemed security risks, sources say

PSA: unsloth/GLM-5.2-GGUF is uploading

i post-trained a model to reliably roll a die

llama.cpp - how to free up even more space on your GPU

My GLM-5.2-FP8 HGX-H200 SGLang docker deploy config

Multilingual-Multimodal-NLP/LoopCoder-V2 · Hugging Face