r/LocalLLaMA

500 articles archived · Visit source ↗ · RSS

r/LocalLLaMA community 4d ago

endless-frontier/BigBang-v1 - qwen 3.5 finetunes

table bench https://huggingface.co/bartowski/endless-frontier_BigBang-v1-GGUF I'm downloading this model only because Bartowski converted it to .gguf, so it might be interesting. Doubts : The headline number is basically meaningless. "Performance between DeepSeek Flash (old one)…

26
r/LocalLLaMA community 4d ago

The Gemma team will host a special event on August 20

Tweet by u/hackerllama Could be copium, but I would love to see Gemma 4.1 there with unified audio input for all model sizes perhaps even up to 120B, much improved tool calling (even with the latest template there are still bugs ), higher precision QAT from the start and…

20
r/LocalLLaMA community 4d ago

[2606.05682] Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation

Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments. Quantization-aware distillation (QAD) helps recover accuracy lost under low bit…

32
r/LocalLLaMA community 4d ago

Speculative decoding in a tools call

paper : https://arxiv.org/html/2608.00814v1 source : https://x.com/i/status/2086505517640540587   submitted by   /u/Illustrious-Swim9663 [link]   [comments]

13
r/LocalLLaMA community 4d ago

Open Model: Google Weather Next 2

I am not a meteorologist, but I just read a very interesting article: https://arstechnica.com/science/2026/08/deepminds-hurricane-model-bought-forecasters-an-extra-day/ In a paper published on Thursday in Nature, researchers show that the WeatherNext AI model can predict…

26
r/LocalLLaMA community 4d ago

Pathway's BDH(post-transformer arch) matches GPT2 scaling from 10M to 1B params trained from scratch. runs on Normal GPUs

  submitted by   /u/Candid-Tackle-9061 [link]   [comments]

36
r/LocalLLaMA community 4d ago

Add a amd 9700 ai pro to a 3x5090 system vs buy a 5070ti for general useage.

Hey guys I have this OCD im trying to decide about, I was lucky enough to buy 3 5090 before all the crazy ai stuff started and while that system works fine. Id also bought a razor core egpu that stopped working a little ago and so got sent off for repair, well its come back now…

35
r/LocalLLaMA community 4d ago

DeepSeek-V4-Flash-0731 Q8_K_XL sometimes stops mid-task in OpenCode - anyone else seeing this?

Hey everyone, I've been experimenting with the new DeepSeek-V4-Flash-0731 release locally using the Unsloth Studio Q8_K_XL GGUF with OpenCode. Overall, it's been working really well, but I've noticed a strange behavior during longer agentic coding sessions. Once the context gets…

38
r/LocalLLaMA community 4d ago

small model for text-to-regex

Anyone have a good local model for this? i feel like this should be a near solved problem.   submitted by   /u/minaminotenmangu [link]   [comments]

23
r/LocalLLaMA community 4d ago

Open-weight video gen that actually delivers. Five days with MiniMax H3 on local hardware.

H3 weights went live on HuggingFace August 3rd and I started pulling them immediately. An omni-modal video model with native stereo audio in the same forward pass, where audio can actually drive the video generation? On open weights? I had to try it. Five days in, the quality is…

31
r/LocalLLaMA community 4d ago

Two flags took the official Ling-3.0-flash INT4 from 20.8 to 38.7 tok/s on one DGX Spark

The official INT4 does load on a single DGX Spark. The naive config just leaves most of its speed on the floor, 20.8 tok/s. Two changes take it to 38.7. Quick context on where this comes from: I work on Ling at inclusionAI, and none of these numbers are mine. sudoingX on X ran…

27
r/LocalLLaMA community 4d ago

Lophius: A workbench for language model research, from the creator of Heretic

Hi folks, I hate slop as much as you do, so instead of starting with "The Problem", I'll just cut to the chase: I just published Lophius, which is the culmination of more than two years of fighting with Jupyter and Transformers. It's a hybrid code/GUI research system that runs…

8
r/LocalLLaMA community 4d ago

DeepSeek v4 Flash 0731 locally on CPU

After seeing the benchmark results for the full release of DS v4 Flash 0731, I replaced my 2 x 16GB DDR4 ram sticks with 2 x 32GB DDR4 ram sticks to get a max supported of 128 GB RAM, in hope to be able to run GLM 5.2 equivalent model locally i.e. DS v4 Flash 0731 I also have…

25
r/LocalLLaMA community 4d ago

Underestimated budget solution: radeon 780m iGPU

There are so many posts where people complaining about high prices and asking for solution <= 1000 EUR. So, there is one solution to consider: PC/mini PC/laptop on Ryzen 7 260 / Ryzen 9 8945HX /etc CPU with 780m iGPU and 64 Gb of DDR5 RAM. Barebone mini PC costs around 300-400,…

6
r/LocalLLaMA community 5d ago

Tencent announce WorldClaw

https://tencent-hunyuan.github.io/Hunyuan3D-WorldClaw/ Looks impressive from that site, hopefully they open weight this so we can all play with it.   submitted by   /u/Uncle___Marty [link]   [comments]

34
r/LocalLLaMA community 5d ago

What a deal. Thanks newegg

The gods have blessed me. Card came in. brand new. I couldn’t believe it.   submitted by   /u/Street-Buyer-2428 [link]   [comments]

26
r/LocalLLaMA community 5d ago

AMD llama.cpp: reducing MTP buffer overhead gave me 64K → 149K context for Qwen 27B

Available context length with and without the patch: Model: QWEN 27B ROCm stock patched Vulkan stock patched IQ4_XS Pure, single 16GB GPU 19.456 76.032 68,352 78,592 Q6_K_L on 16GB + 12GB 64,256 149,248 68,864 151,296 The issue is that llama.cpp overestimates the memory needed…

6
r/LocalLLaMA community 5d ago

300b on 32gb MoE-streaming findings + optimisations

The past week I've been running DSv4 inference on my laptop by keeping everything RAM-resident except the MXFP4-experts (since expert pool is ~147GB and won't fit) TL;DR - read speed is the limiter more than the kernels; repacking to enable sequential reads rather than random…

34
r/LocalLLaMA community 5d ago

DeepSeek V4 Flash 0731 hits 82.7% on Terminal-Bench 2.1 in an independent public-harness run (445 trials)

Disclosure: I’m the author of Ante. DeepSeek recently reported an 82.7% score on Terminal-Bench 2.1 for DeepSeek V4 Flash 0731. Its evaluation used “DeepSeek Harness minimal mode,” which hasn’t been released yet. We wanted to see whether the reported result could be…

14
r/LocalLLaMA community 5d ago

Best Embedding + Reranking Model

What Local Embedding + Reranking Models are you guys running for RAG? I went down this rabbit hole because I wanted a Embedding Model + Reranker for a Translation Memory Server. Essentially, given X phrase/word/sentence/paragraph in English, can it find a similar…

26
r/LocalLLaMA community 5d ago

Updated benchmark: Deepseek V4 Flash on SlopCodeBench (local)

Howdy - I posted a benchmark here - https://www.reddit.com/r/LocalLLaMA/comments/1vbtiy7/deepseek_v4_flash_on_slopcodebench/ This was using the hosted API - since then I've been playing around with quants Here is the lastest benchmark -…

20
r/LocalLLaMA community 5d ago

Trustfactor in training data?

Would it be possible and make sense to add metadata to training data e.g. a trustfactor (0.0 - 1.0)? For example: the older data is the less trustworthy it is. And data after 2022 gets less trustworthy over time (because of ai). With the right rules it would maybe be possible to…

27
r/LocalLLaMA community 5d ago

LFM 2.6B is a lot of fun.

I’m reaching speeds of 260T/s tg and 20k pp on my 3090s lol, because this model is small and meant to run on phones. From what I‘ve been trying it’s surprisingly great for incredibly quick things like “read this massive thing and tell me if it mentions x” or “what’s the summary…

15
r/LocalLLaMA community 5d ago

Memory Bandwidth problems with Intel Sapphire Rapids

I have a Xeon w7-3465 and 4 sticks of RDIMM DDR5-4800 with a theoretical max bandwidth of 153GB/s. I am trying to run DeepSeek-V4-Flash-0731 as it is an MoE and the weights are in MXFP4, so I should roughly process 150GB/6.5GB tokens per second as it has 13B Experts. The problem…

11
r/LocalLLaMA community 5d ago

ds4 flash 0731 UD-IQ2_M wrote a custom metal kernal for kimi k2 IQ1_0 in about 50 minutes

as a programming ignoramus this kind of thing seems extremely impressive to me... maybe others can shed light on whether this is expected from this level model at q2. DS4 IQ2_M just spent about 50 minutes writing a custom metal kernel after it could not find one on github for…

15
r/LocalLLaMA community 5d ago

RTX 5090 96GB spotted on Alibaba?

  submitted by   /u/panchovix [link]   [comments]

35
r/LocalLLaMA community 5d ago

Intel Optane - Potential?

Intel's Optane technology as we know it has been discontinued. But while that's the case, I see it being highly promising still due to its role (or where it was meant to be) right in between system RAM and NVME SSDs. As far as I understand, it sits much closer to the RAM in…

21
r/LocalLLaMA community 5d ago

No wonder Qwen and Gemma are so different

Pasted the same HTML/JS code (330 lines) into Qwen 35B A3B and Gemma 26B A4B. Qwen: tokenized the input to 1609 tokens Gemma: tokenized the input to 4258 tokens. Damn. I've never noticed this before and I haven't seen people mention it. That alone helps explain why Qwen is…

4
r/LocalLLaMA community 5d ago

Kimi K3 (Unsloth) IQ2-XXS from 711GB down to 478GB!!! Only Multi-language was removed to trim the size

Firstly a big thanks to the poster "hellohazine", he basically only removed the multi-lingual fat of the model and just kept the English language intact. It is the exact model, and the rest of the model still intact with all of its high intelligence. I think that was a brilliant…

29
r/LocalLLaMA community 5d ago

Extremely slow DSpark draft model performance (1-2 t/s) with DeepSeek-V4-Flash on llama-server compared to MTP?

Hey everyone, I could use some advice on setting up speculative decoding correctly with llama-server . My Hardware: GPUs: RTX 4090 + RTX 6000 Pro (120GB total VRAM) RAM: 32GB I am currently testing the DeepSeek-V4-Flash-0731 unsloth model using the 137GB Q4 UD-IQ4-NL quant. When…

17
r/LocalLLaMA community 5d ago

Is Microsoft-Phi dead?

Phi was one of my favorite models with a bit of a mixed reputation with some claiming it's benchmaxxed and others seeing its potential and usecases. I was a big fan of Phi but the last major release was in december 2024 with every other release being a Phi 4 iteration (like Phi…

9
r/LocalLLaMA community 5d ago

enabling PCI-E p2p for consumer Nvidia cards will yield you more than you think

Disclaimer - no LLM was used to write this post/note As larger post about my setup will come later, want to give heads-up to folks who use VLLM and >= 2 GPUs. So I have pretty meaty server (8 channel AMD EPYC, ~150GB/s RAM bw) and 4x5060Ti 16GB in PCI-E 4.0 8x mode. Considering…

29
r/LocalLLaMA community 5d ago

Comparing 4bit quants for MLX

Curious what people think are the ideal 4-bit quantization types on MLX These quants seem to be the most popular, at least for Gemma4 and Qwen3.6: - OptiQ 4bit ( mlx-community/Qwen3.6-27B-OptiQ-4bit ) - Unsloth dynamic 2.0 MLX ( unsloth/Qwen3.6-27B-UD-MLX-4bit ) - oQ (…

4
r/LocalLLaMA community 5d ago

any reasonably fast public benchmarks I should run quants of deepseek flash 0731 on?

I have various quants of this model and am curious how they perform. can anyone recommend which benchmark would be a good test case for quantization effects? Maybe that can be completed with about 1 million tokens?   submitted by   /u/nomorebuttsplz [link]  …

35
r/LocalLLaMA community 5d ago

Building a budget 32GB → 48GB VRAM home AI server: 2-3x RX 9060 XT 16GB vs RTX 5060 Ti 16GB, AM5 vs used EPYC?

I’m planning a dedicated home AI server, mainly for local LLM inference, agents/tool use, Docker services, and eventually larger MoE models with CPU offload. My plan is to start with 2x 16GB GPUs = 32GB VRAM , but I want to build the platform from day one knowing that I’ll…

37
r/LocalLLaMA community 5d ago

I built a local realtime voice stack for Ollama: Parakeet STT → Qwen 2.5 7B → Qwen3-TTS

  submitted by   /u/InternationalGap3698 [link]   [comments]

6
r/LocalLLaMA community 5d ago

Repeated generation is worth it and self-evaluation is effective

I made gemma4 12B write timestamp-anchored summaries of youtube video transcripts. I tested if the summaries have significant qualitative variance and if the SLM can pick the best one by itself. Below is the prompt texts I used. "{{[INPUT]}}<attachement name='original'>",…

18
r/LocalLLaMA community 5d ago

Building a zero-dependency C inference engine for BitNet (1.58-bit) - lessons from hitting 36 tok/s on a Xeon CPU

Over the past few months I have been building a CPU-first inference engine from scratch in pure C99 (no Python, no CUDA, no BLAS, just GCC and make). The focus has been running 1.58-bit ternary models natively without heavy runtime overhead. Currently it hits 36.25 tok/s on…

30
r/LocalLLaMA community 5d ago

Showoff Saturday: Local 4x 6000 Pro (multi-year progression)

Not the biggest or shiniest, but it's mine From gaming machine inference on the original llama models, to a 4x RTX 6000 Pro Max Q + 4x 3090s local AI cluster. Pictures are in reverse chronological order! With the pricing apocalypse meaning less builds shared here recently,…

38
r/LocalLLaMA community 5d ago

My first run of Kimi K3 locally.

Running across 2 clusters using llama.cpp over RPC too. Both clusters are not enough to hold everything in memory, so main cluster still partially offloads to run. Goal will be to get all the GPUs in one system and without RPC, I should probably see 2-3x faster speed. Running…

7
r/LocalLLaMA community 5d ago

128GB vs 256gb of ram

Imagine you have 128gb of VRAM. what accompanying ram capacity you would choose (DDR4 8channel)? For example Deepseek v4 flash in q8 takes around 170GB + 12GB Dflash + ~10GB per 1m context so it’s under 200gb. so 128 + 128 should be good But for something like MiMo…

5
r/LocalLLaMA community 5d ago

Tesla V100 Qwen3.6 27B Performance

Looking for V100 users to share your config and it's performance. GPU: Tesla V100 PCIE 32Gb Qwen3.6 27B Q4_K_M + Q8_0 MTP 128K context length Pi coding agent llama.cpp model preset: [*] spec-default = 1 ctx-size = 131072 mmap = 1 kv-unified = 1 n-gpu-layers = 999 threads = 18…

14
r/LocalLLaMA community 5d ago

Claude Code in 9 lines python

I was wondering what a minimal coding agent implementation would look like that can be used like Claude Code or Codex Not feature-by-feature of course but basically stripping everything out that is not needed here is what I came up with: 9 lines of python no 3rd party deps…

29
r/LocalLLaMA community 6d ago

MI25 for 80-100€ worth it?

seems to be about as good as a vega 56 with 16Gb of VRAM, is it worth it? (don’t want to deal with NVIDIA drivers on Linux, already have an rx6650xt and might simply use vulkan for llamacpp inference)   submitted by   /u/Kidplayer_666 [link]   [comments]

8
r/LocalLLaMA community 6d ago

Has anyone here fiddled with TPUs for inference ?

I discovered recently that Google uses their own TPUs, like tiny ASIC cards like the toy ones that existed for bitcoin. And while it sounds inefficient the fact they use thousands of them because...they can...means at scale they aren't so bad. Has no one here given them a try? I…

26
r/LocalLLaMA community 6d ago

A local code index for coding agents that resolves imports without a language server (Rust, MIT, runs offline)

Last time I posted about this here someone asked the right question: why not just use rust-analyzer, and how do you build a call graph without real semantics? I did not have a good answer then. I do now, and it comes with a concession. The concession first. This is not…

20
r/LocalLLaMA community 6d ago

2027 Memory Capacity Is Reportedly Sold Out

  submitted by   /u/johnnyApplePRNG [link]   [comments]

27
r/LocalLLaMA community 6d ago

Qwen3.6 27B + 35B on vLLM, single R9700 (gfx1201)

I've been tuning my new Radeon AI Pro R9700, and figured that this would be useful information for people who are trying to optimise their setups. I'm pretty happy with these results and looking forward to Qwen3.8.. Summary below provided by Claude (which helped me configure it…

18
r/LocalLLaMA community 6d ago

model: support Longcat-Flash (need testing) by ngxson · Pull Request #19182 · ggml-org/llama.cpp

This PR should be ready for testing now . I tested with a very small (8B params) sub-model extracted from the original one. Appreciate if someone can test with the bigger model. GGUF(for testing) from PR : (Please check latest comments at bottom for updated GGUFs )…

33
r/LocalLLaMA community 6d ago

DeepSeek V4 Flash 0731 appreciation post

I’m running DSV4F 0731 on dual spark, and honestly… wow. It’s an absolute workhorse, and the benchmarks are real. Everyday tasks with Hermes agent? Effortless. Coding tasks with OpenCode? I’m genuinely amazed at what it can handle. I can throw a two-hour coding session at it,…

12

endless-frontier/BigBang-v1 - qwen 3.5 finetunes

The Gemma team will host a special event on August 20

[2606.05682] Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation

Speculative decoding in a tools call

Open Model: Google Weather Next 2

Pathway's BDH(post-transformer arch) matches GPT2 scaling from 10M to 1B params trained from scratch. runs on Normal GPUs

Add a amd 9700 ai pro to a 3x5090 system vs buy a 5070ti for general useage.

DeepSeek-V4-Flash-0731 Q8_K_XL sometimes stops mid-task in OpenCode - anyone else seeing this?

small model for text-to-regex

Open-weight video gen that actually delivers. Five days with MiniMax H3 on local hardware.

Two flags took the official Ling-3.0-flash INT4 from 20.8 to 38.7 tok/s on one DGX Spark

Lophius: A workbench for language model research, from the creator of Heretic

DeepSeek v4 Flash 0731 locally on CPU

Underestimated budget solution: radeon 780m iGPU

Tencent announce WorldClaw

What a deal. Thanks newegg

AMD llama.cpp: reducing MTP buffer overhead gave me 64K → 149K context for Qwen 27B

300b on 32gb MoE-streaming findings + optimisations

DeepSeek V4 Flash 0731 hits 82.7% on Terminal-Bench 2.1 in an independent public-harness run (445 trials)

Best Embedding + Reranking Model

Updated benchmark: Deepseek V4 Flash on SlopCodeBench (local)

Trustfactor in training data?

LFM 2.6B is a lot of fun.

Memory Bandwidth problems with Intel Sapphire Rapids

ds4 flash 0731 UD-IQ2_M wrote a custom metal kernal for kimi k2 IQ1_0 in about 50 minutes

RTX 5090 96GB spotted on Alibaba?

Intel Optane - Potential?

No wonder Qwen and Gemma are so different

Kimi K3 (Unsloth) IQ2-XXS from 711GB down to 478GB!!! Only Multi-language was removed to trim the size

Extremely slow DSpark draft model performance (1-2 t/s) with DeepSeek-V4-Flash on llama-server compared to MTP?

Is Microsoft-Phi dead?

enabling PCI-E p2p for consumer Nvidia cards will yield you more than you think

Comparing 4bit quants for MLX

any reasonably fast public benchmarks I should run quants of deepseek flash 0731 on?

Building a budget 32GB → 48GB VRAM home AI server: 2-3x RX 9060 XT 16GB vs RTX 5060 Ti 16GB, AM5 vs used EPYC?

I built a local realtime voice stack for Ollama: Parakeet STT → Qwen 2.5 7B → Qwen3-TTS

Repeated generation is worth it and self-evaluation is effective

Building a zero-dependency C inference engine for BitNet (1.58-bit) - lessons from hitting 36 tok/s on a Xeon CPU

Showoff Saturday: Local 4x 6000 Pro (multi-year progression)

My first run of Kimi K3 locally.

128GB vs 256gb of ram

Tesla V100 Qwen3.6 27B Performance

Claude Code in 9 lines python

MI25 for 80-100€ worth it?

Has anyone here fiddled with TPUs for inference ?

A local code index for coding agents that resolves imports without a language server (Rust, MIT, runs offline)

2027 Memory Capacity Is Reportedly Sold Out

Qwen3.6 27B + 35B on vLLM, single R9700 (gfx1201)

model: support Longcat-Flash (need testing) by ngxson · Pull Request #19182 · ggml-org/llama.cpp

DeepSeek V4 Flash 0731 appreciation post