r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 1mo ago

Be careful fam - New RTX 4090 fakes hit the market, with GPUs and memory almost indistinguishable from real chips

Be careful out there of deals now. If it’s too good to be true…   submitted by   /u/chemist_slime [link]   [comments]

23
r/LocalLLaMA community 1mo ago

AMD Reportedly Raises GDDR6 Prices for Board Partners

Is an AMD price incoming as well? Nvidia just discontinued the 5090 in favor of RTX Pros   submitted by   /u/chemist_slime [link]   [comments]

11
r/LocalLLaMA community 1mo ago

M5 Ultra Mac 256GB Studio - 64 core vs. 80 core version, is the inference speed difference worth the extra 10 weeks wait time?

I’m seeing 6-7 week delivery wait time for M5 Ultra Mac Studio 64 core version versus 15-17 weeks for 80 core version. They both Ultra and have the same memory bandwidth. Is the speed difference that noticeable for prompt processing and token generation? Is it worth the $1200…

38
r/LocalLLaMA community 1mo ago

Building a 4x R9700 setup for a 10 person startup

Just wanted to share a build I am doing for a client. $18k. Specs: Threadripper 9970x 128gb DDR5 ECC 5600 4x AMD Radeon R9700 AI Pro 32gb(128gb total VRAM) 1600W PSU(GPUs are undervolted and under 210w each) Engine : using a fork of Radiance to serve Qwen 3.8 Next flash/27b and…

22
r/LocalLLaMA community 1mo ago

OMG! If you have a Mac with 64GB, try Qwen3.8-Flash-Next-oQ4e-mtp with oMLX!

I'm genuinely shocked! I was able to run Qwen3.8-Flash-Next-oQ4e-mtp on M3Max 64GB with oMLX! Even a couple of weeks ago, I wasn't able to get it to run. I just tried the latest commit for fun, and it worked! It feels like some kind of sorcery to be able to run a 100GB model…

9
r/LocalLLaMA community 1mo ago

Best small model + setup together with qwen3.8-flash-next

hello, I currently use swift qwen3.8 gsq rco in iq3 xxs in strata. The model is only used for hermes agent. I have 64GB of ram + 32GB R9700 where the model runs on. But I also have an rx7800xt with 16GB. For hermes agent i want to run a small but intelligent enough model as an…

34
r/LocalLLaMA community 1mo ago

Qwen3.8-Flash-Next Strata pi beat Claude-code Opus 5.5 on a 5090

EDIT to address that the title is misleading: Qwen3.8-Flash-Next Strata pi beat Claude-code Opus 5.5 on a 5090 on a benchmark dedicated to compare model for everyday tasks. not to discriminate frontier model. --- Original post: I'm using Qwen3.8-Flash-Next on Strata with pi as…

16
r/LocalLLaMA community 1mo ago

[Model] Support MiniCPM-V 4.7 by tc-mb · Pull Request #29416 · ggml-org/llama.cpp

Now all we need is a new model. Let me remind you that MiniCPM-V-4.7-35B-A3B was spotted on r/LocalLLaMA a few days ago (but the model was later hidden on HF).   submitted by   /u/jacek2023 [link]   [comments]

17
r/LocalLLaMA community 1mo ago

Benefits of using bigger models than Qwen 3.8 flash next?

I recently started using local LLM for mostly coding tasks for my personal projects. I also have found local models responding much better for topics such as health, fitness, ergonomics etc. compared to the quality I had observed with ChatGPT Go (which is still a basic level)…

8
r/LocalLLaMA community 1mo ago

Goodbye Coding Plans

I’d say we’re a year-ish away from local + good harness being enough to write 90% of code. Looking at you, Qwen. Edit: damn, sounds like you all think Qwen3.8 Next Flash is there already… going to have to give it a spin…   submitted by   /u/Purple-Programmer-7 [link]…

34
r/LocalLLaMA community 1mo ago

Engineer / developer observations of Gemma4-31B, Qwen3.8-27B, and 6.1-Sol for software engineering work

For what it's worth. I'm a mechanical engineer by education and software engineer in practice over the past ~20 years. Playing around with local models has been a recent hobby. Today I decided to do a few tests on things that I'd consider analogous to "real" work that I do,…

38
r/LocalLLaMA community 1mo ago

SLM community telenovela: what do you guys think about this arguement?

If there is something I like about the indie SLM community is there is always some drama every week about something. I mean, look, I don't like to disrespect others or their work, but this is some crazy statements made by banaxi-tech. here is the link to the HF discussion:…

10
r/LocalLLaMA community 1mo ago

Is anyone running Qwen3.8 Flash Next with a 1M context?

So, while doing a search to read the model card again, and because I didn't memorize the huggingface URL, I saw an AI "answer" at the top of the search, stating that while it natively supports a ~244k context, it could go to 1M using YaRN. That's the first I have heared of both,…

5
r/LocalLLaMA community 1mo ago

48Gb VRAM speed AND quality ! (Qwen 3.8 27B Swift 1.5 W8A16)

Hi, Because sometimes you need both speed AND quality, I made my own Qwen 3.8 27B Swift 1.5 quant. It's an 8 bits AWQ + GPTQ quant. Made with love and precision (read the model card) on my 4 RTX 3090, I believe it's the best compromise you can actually get on 48Gb of VRAM.…

16
r/LocalLLaMA community 1mo ago

No more RTX 5090

Nvidia reportedly halts GeForce RTX 5090 production in favor of AI data center and professional GPUs — impending supply drought expected to drive up prices, RTX 5080 24GB rumored as new gaming flagship…

4
r/LocalLLaMA community 1mo ago

Is there a better option than llama.cpp for 4GB VRAM for Higher tokens/sec?

I love the idea of running local models on consumer hardware. I currently use `llama.cpp`, but I’ve been looking for a "better" alternative for a while now. I want something that utilizes my system resources more efficiently—for instance, by managing my RTX with 4GB VRAM more…

32
r/LocalLLaMA community 1mo ago

Qwen3.8-27B on a single 3090: 140 tok/s on code with a custom megakernel

I've been using Claude Opus 5.5 to speed up Qwen3.8-27B on my PC (rtx 3090), it wrote a CUDA megakernel that is 1.4-1.9x faster than llama.cpp depending on the task/context length. The results and code are below: Results (same hardware, my megakernel vs llama.cpp with MTP): -…

16
r/LocalLLaMA community 1mo ago

Open-source Mac app that runs EmbeddingGemma 2 locally to search your files by what’s in them

DigUp is a free Mac app that runs Google DeepMind’s new EmbeddingGemma 2 locally over your own files. The model puts text, images, audio and video in one space, so you describe what you remember and land on it: “zebra in a video” opens the clip at the moment it shows up “where…

10
r/LocalLLaMA community 1mo ago

Improve token per second without touching quant

Spent the past month tweaking and experimenting with many different numbers to achieve 30tps. Hardware: -Rx6700xt 12gb vram (AMD) -2x16 ddr4 3200 ram -r5 5600x -llama.cpp vulkan sdk -window11 (no wsl switching since im not used to the environment) Im looking for any improvements…

6
r/LocalLLaMA community 1mo ago

Fully local conversational AI: Whisper + Hermes 8B + Kokoro, zero cloud, running inside a plush toy

A conversational AI plush toy where nothing leaves my network. Everything runs on my LAN: STT: Whisper LLM: Hermes 3 Llama 3.1 8B (Q8_0) TTS: Kokoro Memory: per-person (LangGraph). It knows who it's talking to and remembers past conversations Vision: face identity (dlib) +…

18
r/LocalLLaMA community 1mo ago

Typesafe ai raised 870m $ on hype (jev)

https://typesafe.ai/blog/series-ai Meanwhile one of the reddit members apparently developed the same architecture a year ago: https://www.reddit.com/r/LocalLLaMA/s/8zQXqCNDlb Looks like ai valuation now on the peak of hype. Who yelled louder got most of the invest and money.…

22
r/LocalLLaMA community 1mo ago

NVIDIA reportedly discontinuing RTX 5090, GB202 GPUs to be reserved for RTX PRO series

No.....   submitted by   /u/chemist_slime [link]   [comments]

30
r/LocalLLaMA community 1mo ago

Strata with Qwen3.8 Flash Next UD-Q4_K_XL

Been seeing quite a few posts about Strata lately, so I figured I'd give it a shot on my RTX PRO 6000 and I am very impressed. Most of the benchmarks I've seen are using IQ2 or IQ3 quants, so I wanted to see how Unsloth's UD-Q4_K_XL performs instead. Setup: GPU: 1x RTX PRO 6000…

35
r/LocalLLaMA community 1mo ago

want me to get you some?🤣

Local flea market...and it's about to rain...   submitted by   /u/OttoRenner [link]   [comments]

32
r/LocalLLaMA community 1mo ago

What are people doing with decisions Jev-like models?

Let's say I run a local Jev-like decision model on my machine. What are the use cases? I already use it for deep eval testing but what are people usually doing with them? It a bit of "solution in search of a problem" but I'm exploring and want to be sure that I'm not missing…

32
r/LocalLLaMA community 1mo ago

Sharing my latest project: strata-mlx

Strata is impressive: it runs Qwen3.8-Flash-Next, a 125B-parameter mixture-of-experts model, on a 12 GB graphics card. But its authors have said macOS is out of scope. I still wanted it on the Mac. So I spent a day and a half and wrote strata-mlx. It is an unofficial MLX engine,…

35
r/LocalLLaMA community 1mo ago

big or small?

what size do you want? tell them on X: https://x.com/QwenDevs/status/2108764909798641737   submitted by   /u/jacek2023 [link]   [comments]

25
r/LocalLLaMA community 1mo ago

Created laya : Now Introducing a new 800 Million Param physics-based typed decision model with 73k context and image support

Around 3 weeks ago, I posted on this subreddit about how Jev used my one-year-old architecture, og post: https://www.reddit.com/r/LocalLLaMA/s/VpuMJt577S Then, just after that, I introduced Laya, the first open-source version of JEV, and it has reached a very large scale, thanks…

16
r/LocalLLaMA community 1mo ago

microsoft doubles down on local ai with nvidia, but with a cost

https://preview.redd.it/uyjw2lcrvkuh1.png?width=1536&format=png&auto=webp&s=ce991be22faec3de5cbaf39119c14782b6ed0af3 link to article TLDR; new microsoft surface and nvidia rtx spark laptops are starting at 2.6k and hitting nearly 7k for the 128gb unified memory models. local llm…

5
r/LocalLLaMA community 1mo ago

Qwen 3.6 35B A3B: 131K context + vision on 6GB VRAM

Running HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_M with llama.cpp at ~600 tok/s prefill and 23 tok/s decode, 131k context window, Q8 KV cache - on an RTX 2060 6GB + 32GB DDR4 RAM. Speeds start at ~600 tok/s prefill / 23 tok/s decode on an empty KV cache. As…

21
r/LocalLLaMA community 1mo ago

Been a while since I posted about my harness OpenLumara.. I only post when a huge new feature releases. I think this is one of them!

This is my new meditation module for my harness OpenLumara . It started out as a demonstration of just how powerful its new webui extensions system is, but has grown to be quite the meditation app in its own right. As you will hear in the video, it's a full audiovisual…

29
r/LocalLLaMA community 1mo ago

3.66B formal logic model built on Granite 4.2 3B · Hugging Face

webAI released TwIL LM3 Pro on September 30. I haven't seen it posted here yet, so I went through the model card, and their comparison chart is attached. It's a 3.66B model built on IBM's Granite 4.2 3B and tuned only for formal logic. That means things like checking whether a…

5
r/LocalLLaMA community 1mo ago

I have an ESC4000 G3 with 8x T4s in it - what is the fastest way I can deploy Qwen3.5-9B for about 10-15 users concurrently: currently using llama.cpp

Hi All: I have an Asus ESC 4000 G3 with 128 GB DDR4 RAM - I tried putting in my V620s but couldn’t put more than 2. Sadly, pivoted to T4s, these are 72 watt passive cards and I thought I could use them like how I use the Mi50 32GB - but I was very wrong. It has no support from…

18
r/LocalLLaMA community 1mo ago

Basalt: Flash-Next at 665 tok/s structured, 354 prose on a 5090 + 5060 Ti (2.6x Strata)

Basalt: Blackwell inference engine Hi all! So over the last week I've been working on a Strata fork that's heavily tuned and can achieve throughput up to 2.6x what Strata usually does on the same weights. It's designed as a specialized engine that only supports Qwen3.8…

7
r/LocalLLaMA community 1mo ago

PR merged: `/v1/systemone` support in vllm

  submitted by   /u/413205 [link]   [comments]

21
r/LocalLLaMA community 1mo ago

Qwen Flash Next @ 137 tok/s & 3,497 tok/s Prefill w/ 512k context on a 5090, 192gb ram, Windows Build, comparing Strata and Infernix

Strata vs Infernix on the same model, RTX 5090 — A/B at 262k and 512k context Hi folks, another inference engine post for your feed. I was blown away by Strata, but saw someone else post about Infernix with some wild claims, and it doesn't seem so well known, so I've spent a day…

19
r/LocalLLaMA community 1mo ago

What does LocalLLaMA think of REA?

https://github.com/morluto/rea hit #1 a couple of days ago on github   submitted by   /u/Sufficient-Scar4172 [link]   [comments]

37
r/LocalLLaMA community 1mo ago

SlopSoup TV. A live, never-ending 24/7 pixel-art TV network inspired by old-school late night Adult Swim. No human makes any of it.

SlopSoup TV is a live, never-ending pixel-art TV network. Every script, character, voice, camera cut and schedule decision is made by models. It's made to be very absurd, dark and strange. Hardware: one Hugging Face Space, 16 vCPU, no GPU. Everything below runs on CPU next to a…

31
r/LocalLLaMA community 1mo ago

My "Anthropic/ChatGPT in a box" is... now open source and publicly hosted on Github.

It's been a long time coming, and I should have done this in the first place. But LumaBrowser is now open source. Since I first released it a year ago, it's had several thousand installs and an absolutely tremendous amount of feedback. And it's come a long way. I don't call this…

17
r/LocalLLaMA community 1mo ago

Qwen3.8-27B UD-IQ4_XS Heretic + MTP on a 16 GB card with 55-68 tok/s (24gb and 12gb versions available too)

I wanted the uncensored Qwen3.8-27B (llmfan46's Heretic build, MTP head preserved) on my RTX 4080 with MTP on (because I've been testing TONS of configs, and realized that would be the one) And none of the published quants were built for that: the good IQ4_XS doesn't leave room…

4
r/LocalLLaMA community 1mo ago

Strix halo + CMP 170HX setup

A bit of backstory : i got lucky in August and was able to grab a CMP 170 HX 8GB for around 500$ on alibaba. I tried later to get a second one for around 1k$ but my order got cancelled and prices went to the moon... I unlocked it with https://github.com/amoghmunikote/cmpunlocker…

5
r/LocalLLaMA community 1mo ago

Qwen3.8 27b with 200K ctx + MTP on 12GB Ampere Cards

Hello all! Thank you to everyone that contributed their feedback and notes for LlamAmpere the last time I posted. I've continued to chip away at improvements for the 3090 crowd -- this release is modestly faster (3-4%), with a few hundred MB smaller runtime (if you're using…

5
r/LocalLLaMA community 1mo ago

Is there a music embedding model?

Has anybody built/released a music embedding model trained on a large library of diverse music?   submitted by   /u/unraveleverything [link]   [comments]

29
r/LocalLLaMA community 1mo ago

H2O-Lightning-4B: Apache-2.0 4B Decision model, official #1 open model on JevBench (above Jev)

Disclosure: I work at H2O.ai. We released H2O-Lightning-4B, an open-weight (Apache-2.0) model for the "decisions API" style of inference that Jev made popular: you send a state plus typed questions (pick one / yes-no / score), and get calibrated probabilities back from a single…

6
r/LocalLLaMA community 1mo ago

Just another purely open-weight models benchmark

Live results, coding benchmarks included, agentic benchmarks included, domains and other classifications filters included. (Spoiler: DeepSeek-V4-Vision-Exp rules, but other models have their rule areas): https://beta.locallm.top Evaluated by domain experts (my friends mostly;…

14
r/LocalLLaMA community 1mo ago

Qwen 3.8 Flash Next-GSQ-RCO-IQ2_XS at ~21 tok/s on just an RTX 3060 12GB + 16GB DDR4 RAM(No gate pruning, 100% bit-exact)

(Don't judge by the screenshot, the cache is cold. It hits 24+ tok/s with a warm cache!) About two months ago, I made a post here asking whether predicting which MoE experts would be used on the next token could actually help speed up CPU/GPU offloading. Original post: Tried…

11
r/LocalLLaMA community 1mo ago

LlamaTale v0.43.0 - MCP Server for story creation

In the dawn of the LLM craze, when wild Llama2's roamed free and 4k tokens was considered a fairly long context for local models, I forked an interactive fiction and MUD library called Tale to experiment with LLM generated content. The idea being a text based adventure with…

23
r/LocalLLaMA community 1mo ago

GLM 5.3 Flash opensource @ the top of Artificial Analysis Cyber Index over Claude.

It seems Dario's "too powerful for you users" strategy is paying off: we have two open models at the top of the leaderboard, surpassing every single model from Anthropic. Open source prevails. Even Mistral Large 4 is better!   submitted by   /u/LegacyRemaster [link]…

8
r/LocalLLaMA community 1mo ago

OpenAI's math findings built on stolen user data

  submitted by   /u/CuTe_M0nitor [link]   [comments]

28
r/LocalLLaMA community 1mo ago

GitHub - google-ai-edge/ml-drift: GPU-Accelerated AI/ML Inference

Blog Post : ML Drift: Next-Gen GPU AI/ML Inference at the Edge - Google Developers Blog The Google AI Edge Team is excited to announce the open-source release of ML Drift , our high-performance, cross-platform, on-device GPU compute engine specifically built for on-device AI/ML…

29

Be careful fam - New RTX 4090 fakes hit the market, with GPUs and memory almost indistinguishable from real chips

AMD Reportedly Raises GDDR6 Prices for Board Partners

M5 Ultra Mac 256GB Studio - 64 core vs. 80 core version, is the inference speed difference worth the extra 10 weeks wait time?

Building a 4x R9700 setup for a 10 person startup

OMG! If you have a Mac with 64GB, try Qwen3.8-Flash-Next-oQ4e-mtp with oMLX!

Best small model + setup together with qwen3.8-flash-next

Qwen3.8-Flash-Next Strata pi beat Claude-code Opus 5.5 on a 5090

[Model] Support MiniCPM-V 4.7 by tc-mb · Pull Request #29416 · ggml-org/llama.cpp

Benefits of using bigger models than Qwen 3.8 flash next?

Goodbye Coding Plans

Engineer / developer observations of Gemma4-31B, Qwen3.8-27B, and 6.1-Sol for software engineering work

SLM community telenovela: what do you guys think about this arguement?

Is anyone running Qwen3.8 Flash Next with a 1M context?

48Gb VRAM speed AND quality ! (Qwen 3.8 27B Swift 1.5 W8A16)

No more RTX 5090

Is there a better option than llama.cpp for 4GB VRAM for Higher tokens/sec?

Qwen3.8-27B on a single 3090: 140 tok/s on code with a custom megakernel

Open-source Mac app that runs EmbeddingGemma 2 locally to search your files by what’s in them

Improve token per second without touching quant

Fully local conversational AI: Whisper + Hermes 8B + Kokoro, zero cloud, running inside a plush toy

Typesafe ai raised 870m $ on hype (jev)

NVIDIA reportedly discontinuing RTX 5090, GB202 GPUs to be reserved for RTX PRO series

Strata with Qwen3.8 Flash Next UD-Q4_K_XL

want me to get you some?🤣

What are people doing with decisions Jev-like models?

Sharing my latest project: strata-mlx

big or small?

Created laya : Now Introducing a new 800 Million Param physics-based typed decision model with 73k context and image support

microsoft doubles down on local ai with nvidia, but with a cost

Qwen 3.6 35B A3B: 131K context + vision on 6GB VRAM

Been a while since I posted about my harness OpenLumara.. I only post when a huge new feature releases. I think this is one of them!

3.66B formal logic model built on Granite 4.2 3B · Hugging Face

I have an ESC4000 G3 with 8x T4s in it - what is the fastest way I can deploy Qwen3.5-9B for about 10-15 users concurrently: currently using llama.cpp

Basalt: Flash-Next at 665 tok/s structured, 354 prose on a 5090 + 5060 Ti (2.6x Strata)

PR merged: `/v1/systemone` support in vllm

Qwen Flash Next @ 137 tok/s & 3,497 tok/s Prefill w/ 512k context on a 5090, 192gb ram, Windows Build, comparing Strata and Infernix

What does LocalLLaMA think of REA?

SlopSoup TV. A live, never-ending 24/7 pixel-art TV network inspired by old-school late night Adult Swim. No human makes any of it.

My "Anthropic/ChatGPT in a box" is... now open source and publicly hosted on Github.

Qwen3.8-27B UD-IQ4_XS Heretic + MTP on a 16 GB card with 55-68 tok/s (24gb and 12gb versions available too)

Strix halo + CMP 170HX setup

Qwen3.8 27b with 200K ctx + MTP on 12GB Ampere Cards

Is there a music embedding model?

H2O-Lightning-4B: Apache-2.0 4B Decision model, official #1 open model on JevBench (above Jev)

Just another purely open-weight models benchmark

Qwen 3.8 Flash Next-GSQ-RCO-IQ2_XS at ~21 tok/s on just an RTX 3060 12GB + 16GB DDR4 RAM(No gate pruning, 100% bit-exact)

LlamaTale v0.43.0 - MCP Server for story creation

GLM 5.3 Flash opensource @ the top of Artificial Analysis Cyber Index over Claude.

OpenAI's math findings built on stolen user data

GitHub - google-ai-edge/ml-drift: GPU-Accelerated AI/ML Inference