News / #model-release Tag Model releases 500 articles archived under #model-release · RSS Sign in to follow r/LocalLLaMA community 5d ago Comparing 4bit quants for MLX Curious what people think are the ideal 4-bit quantization types on MLX These quants seem to be the most popular, at least for Gemma4 and Qwen3.6: - OptiQ 4bit ( mlx-community/Qwen3.6-27B-OptiQ-4bit ) - Unsloth dynamic 2.0 MLX ( unsloth/Qwen3.6-27B-UD-MLX-4bit ) - oQ (… 4 r/LocalLLaMA community 5d ago any reasonably fast public benchmarks I should run quants of deepseek flash 0731 on? I have various quants of this model and am curious how they perform. can anyone recommend which benchmark would be a good test case for quantization effects? Maybe that can be completed with about 1 million tokens?   submitted by   /u/nomorebuttsplz [link]  … 35 r/LocalLLaMA community 5d ago I built a local realtime voice stack for Ollama: Parakeet STT → Qwen 2.5 7B → Qwen3-TTS   submitted by   /u/InternationalGap3698 [link]   [comments] 6 r/LocalLLaMA community 5d ago Showoff Saturday: Local 4x 6000 Pro (multi-year progression) Not the biggest or shiniest, but it's mine From gaming machine inference on the original llama models, to a 4x RTX 6000 Pro Max Q + 4x 3090s local AI cluster. Pictures are in reverse chronological order! With the pricing apocalypse meaning less builds shared here recently,… 38 r/LocalLLaMA community 5d ago 128GB vs 256gb of ram Imagine you have 128gb of VRAM. what accompanying ram capacity you would choose (DDR4 8channel)? For example Deepseek v4 flash in q8 takes around 170GB + 12GB Dflash + ~10GB per 1m context so it’s under 200gb. so 128 + 128 should be good But for something like MiMo… 5 r/LocalLLaMA community 5d ago Tesla V100 Qwen3.6 27B Performance Looking for V100 users to share your config and it's performance. GPU: Tesla V100 PCIE 32Gb Qwen3.6 27B Q4_K_M + Q8_0 MTP 128K context length Pi coding agent llama.cpp model preset: [*] spec-default = 1 ctx-size = 131072 mmap = 1 kv-unified = 1 n-gpu-layers = 999 threads = 18… 14 r/LocalLLaMA community 5d ago Claude Code in 9 lines python I was wondering what a minimal coding agent implementation would look like that can be used like Claude Code or Codex Not feature-by-feature of course but basically stripping everything out that is not needed here is what I came up with: 9 lines of python no 3rd party deps… 29 r/LocalLLaMA community 6d ago Qwen3.6 27B + 35B on vLLM, single R9700 (gfx1201) I've been tuning my new Radeon AI Pro R9700, and figured that this would be useful information for people who are trying to optimise their setups. I'm pretty happy with these results and looking forward to Qwen3.8.. Summary below provided by Claude (which helped me configure it… 18 llama.cpp releases dev-tools 6d ago b10327 CUDA: fix thread/block count in quantized cpy kernel launches ( #26731 ) CUDA: fix thread/block count in quantized cpy kernel launches tests: add uneven block count cpy case Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI… 31 r/LocalLLaMA community 6d ago DeepSeek V4 Flash 0731 appreciation post I’m running DSV4F 0731 on dual spark, and honestly… wow. It’s an absolute workhorse, and the benchmarks are real. Everyday tasks with Hermes agent? Effortless. Coding tasks with OpenCode? I’m genuinely amazed at what it can handle. I can throw a two-hour coding session at it,… 12 r/LocalLLaMA community 6d ago Anyone else amped up over Qwen 3.8? I’ve been using 3.6 27B Q4, and that quant is fast on an M5. The code has been average, but consistently “good enough.” And, after a year, I can see home LLMs being served at home much like streaming music was introduced. A simple browser extension and all your queries go… 14 r/LocalLLaMA community 6d ago Qwen 35B-A3B MoE vs 27B dense in local coding tests: ~4× faster, much smaller quality gap than I expected I compared Qwen 35B-A3B MoE against Qwen 27B dense on a series of local coding-maintenance tasks. On my R9700/llama.cpp setup, the MoE model generated about 3.9× faster (~116 vs ~30 tok/s) , but the coding-quality difference was much smaller than I expected. Both usually handled… 8 r/LocalLLaMA community 6d ago BeeLLama issues Tried to use Beellama , and using the kvarn6 flag, i notice that its in llama-server --help but its not working. I must be doing something wrong. trying to run the following: llama-server.exe ^ --model "C:\llamaROCM\models\Qwen3.6-35B-A3B-IQ4_XS-3.53bpw.gguf" ^ --spec-type… 11 r/LocalLLaMA community 6d ago Is anyone else finding DeepSeek-V4-Flash unreliable for non-coding tasks? (I am not a native speaker, written by myself, so please bear with me) I really want to like DeepSeek-V4-Flash-0731. But it has serious flaws that don't align with the high score on intelligence benchmarks. And those flaws render it useless unfortunetely for anything else… 21 r/LocalLLaMA community 6d ago U.S. Department of Energy Launches the Genesis Open Models Initiative and, with Arcee, Unveils Genesis-Science-1 — Its First Open-Weight Model for Scientific Research   submitted by   /u/johnnyApplePRNG [link]   [comments] 18 Vercel — AI dev-tools 6d ago Grok Imagine Image 2.0 now available on Vercel AI Gateway Grok Imagine Image 2.0 Preview from xAI is now available on AI Gateway. The model follows detailed instructions closely and plans typography and layout together, so dense, multi-part visuals like infographics, posters, and title screens hold their structure and small text stays… 24 r/LocalLLaMA community 6d ago Serving Deepseek v4 Flash 0731 on 2x DGX Spark — 5-7 GB OS headroom, what would you do to lower VRAM usage and increase OS available RAM? Hey all, I'm serving DSv4Flash 0731 on a cluster of 2x DGX Sparks but am running into constant issues with having almost no RAM (unified memory) left for the OS/cache and I'd love to hear the community feedback on what I could do to get more RAM for headroom. The DGX has an… 32 Hacker News — AI on Front Page community 6d ago U.S. Department of Energy Launches the Genesis Open Models Initiative Article URL: https://genesisopenmodels.anl.gov/ Comments URL: https://news.ycombinator.com/item?id=49216946 Points: 209 # Comments: 68 36 r/LocalLLaMA community 6d ago Qwen 3.6 27B flags/settings in llama.cpp I run the following on a 5090 and have been okay with its performance, it does most things somewhere 80-100 t/s, though that can slow down at full 262k context - more like 40 t/s at times. I use it primarily in appdev tasks. This just barely fits in the 5090, no vision, with… 31 Hacker News — AI on Front Page community 6d ago Lost my phone at the office. Claude suggested tracking Bluetooth signal strength https://xcancel.com/un1c0rnioz/status/2084686552299634805 Comments URL: https://news.ycombinator.com/item?id=49215786 Points: 257 # Comments: 186 31 Simon Willison community 6d ago Moonlight & Mayhem (Raccoon Heist by Codex + GPT-5.6 Sol Ultra) Moonlight & Mayhem (Raccoon Heist by Codex + GPT-5.6 Sol Ultra) On Wednesday I wrote about One-shotting a Raccoon Heist game using Claude Fable 5 , where I had Claude Fable 5 build a full working game from a premise I generated with GPT-3 and DALL-E four years ago . I decided to… 33 Simon Willison community 6d ago Moonlight & Mayhem (Raccoon Heist by Codex + GPT-5.6 Sol Ultra) Moonlight & Mayhem (Raccoon Heist by Codex + GPT-5.6 Sol Ultra) On Wednesday I wrote about One-shotting a Raccoon Heist game using Claude Fable 5 , where I had Claude Fable 5 build a full working game from a premise I generated with GPT-3 and DALL-E four years ago . I decided to… 20 r/LocalLLaMA community 6d ago DeepSeek V4 Flash 0731 - ARC-AGI Results   submitted by   /u/johnnyApplePRNG [link]   [comments] 20 Vercel — AI dev-tools 6d ago Vercel AI Gateway and Vercel Sandbox now available on Hermes Agent Hermes Agent can now use Vercel AI Gateway as its inference layer and run its agent commands in an isolated Vercel Sandbox microVM. Access 200+ models through AI Gateway at no markup on tokens, and every request appears in your AI Gateway dashboard with the rest of your usage… 36 r/LocalLLaMA community 6d ago DeepSeek v4 Flash 0731 on H100 node Hi, I've been playing with recent DeepSeek Flash updated model and it seems excellent. Like talking to Claude 4.8. I'm currently deploying it on VLLm and I'm in progres of tuning performance. I seem to hit the point where I trade decode with prefill by juggling batching knobs.… 21 Hacker News — AI on Front Page community 6d ago DeepSeek V4 Flash 0731 Article URL: https://arcprize.org/results/deepseek-v4-flash-0731 Comments URL: https://news.ycombinator.com/item?id=49214008 Points: 314 # Comments: 189 30 TechCrunch — AI news-outlet 6d ago Cloudflare launches Kitesurf, a browser built for AI agents Cloudflare has introduced Kitesurf, a cloud-hosted browser designed for AI agents instead of people. The company says the browser uses less computing power than Chromium for common automation tasks, helping developers build browser-based AI agents more efficiently. 15 llama.cpp releases dev-tools 6d ago b10311 mtmd: stop feeding the text stream again during Qwen3-TTS generation ( #26706 ) The reference implementation has two mutually exclusive prompt layouts. In non streaming mode the prefill carries the whole utterance text plus tts_eos summed with codec_pad, and the trailing text… 29 LangChain releases dev-tools 6d ago langchain-openai==1.4.2 Changes since langchain-openai==1.4.1 release(openai): 1.4.2 ( #39322 ) fix(openai): handle ContextWindowExceededError ( #39300 ) chore: bump the minor-and-patch group across 3 directories with 7 updates ( #39187 ) fix(openai): filter langchain-generated content block IDs (… 34 r/LocalLLaMA community 7d ago Anyone running DeepSeek-V4-Flash-0731 on MI325X with vLLM? Mine is behaving completely broken Is anyone here successfully running DeepSeek-V4-Flash-0731 locally with vLLM , especially on AMD MI325X? My setup: GPU: 1x AMD Instinct MI325X Model: deepseek-ai/DeepSeek-V4-Flash-0731 vLLM: 0.26.0 ROCm image --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4… 20 r/LocalLLaMA community 7d ago IS GLM 5.2, Kimi 2.7 still worth it? Since now we have kimi k3 and next week we are getting Qwen 3.8 Max and also soon V4 pro Deepseek. I am curious if the old power house like Kimi 2.6/7 code and GLM.5.2 are all that relevant. especially for long hours of coding   submitted by   /u/Hannibalj2ca [link]… 22 r/LocalLLaMA community 7d ago what will be the future of LocalLLaMA? For a long time now, the most popular posts on LocalLLaMA have been either about using LLM in the cloud or about politics. I suspect that people using local models are about 10% now. You can say that this is very good, because now it is an inclusive sub, without gatekeeping. But… 28 r/LocalLLaMA community 7d ago EschaLabs/Qwen3.6-35B-A3B-Escha-W2 · Hugging Face Hey peeps. I know you're tired of low quants giving hard to believe numbers. I'm quite skeptical too and from what I tried I'm often left with the impression that the claims fall short. So this model popped up on Twitter for me. Tried it and was lowkey surprised it held its own.… 26 arXiv — Machine Learning research 7d ago LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm arXiv:2608.06135v1 Announce Type: new Abstract: Large Language Models (LLMs) such as ChatGPT and Claude are widely used for information retrieval and problem-solving. Recent work has focused on improving scheduling algorithms to boost throughput while maintaining low latency.… 12 arXiv — NLP / Computation & Language research 7d ago The em-dash em-beds in Congress: A population-level rise in em-dash frequency in U.S. congressional press releases at the dawn of the large-language-model era, 2021-2025 arXiv:2608.05889v1 Announce Type: cross Abstract: Large language models (LLMs) can leave small stylistic traces in text written with their help. The most discussed is the em-dash (U+2014), especially the unspaced form word---word, which is normal in typeset English prose but… 25 r/LocalLLaMA community 7d ago My issue with Artificial Analysis's 'intelligence index' I swear AA is not the bipartisan they so claim. An open source mode (Qwen 3.8 max) was number 1 on the agentic index, then they just so happen to launch "v4.1.1" of their index in which they just adjusted the weights of the gdpval and t3 banking so that it would be lower than… 13 r/LocalLLaMA community 7d ago Dual 3090 setup: 400 pp t/s to 1600 pp t/s on Qwen 3.6 27B... with slightly lower tps. First of all, my setup: Ryzen 9 5950x DDR4 3200Mhz 64gb (2x32) Dual 3090s, no NVLINK Runtime: llama.cpp Nvidia Drivers 610 Windows 11 25H2 Qwen 3.6 27B Q8 I've been using llama-server with --split-mode tensor for a couple months now, since it gave a pretty nice 10%-20% boost in… 35 Simon Willison community 7d ago datasette-auth-tokens 0.4a13 Release: datasette-auth-tokens 0.4a13 Upgraded for compatibility with `sqlite-utils 4. Tags: datasette 24 Simon Willison community 7d ago datasette-auth-tokens 0.4a13 Release: datasette-auth-tokens 0.4a13 Upgraded for compatibility with `sqlite-utils 4. Tags: datasette 22 Vercel — AI dev-tools 7d ago Give every agent in Herdr its own Vercel Sandbox Terminal coding agents like Claude Code, Codex, and OpenCode can now each run in their own isolated Vercel Sandbox , orchestrated from Herdr , a tmux-style manager that runs them side by side in panes. Nothing an agent runs or edits touches your machine. When you start an agent,… 9 r/LocalLLaMA community 7d ago I thought Deepseek was the answer since I cannot afford GPU for local LLM   submitted by   /u/HsSekhon [link]   [comments] 18 Hacker News — AI on Front Page community 7d ago AMD acquires Taalas to boost inference performance by etching models in silicon https://ir.amd.com/news-events/press-releases/detail/1296/am... https://chatjimmy.ai/ Comments URL: https://news.ycombinator.com/item?id=49201970 Points: 213 # Comments: 151 13 Ars Technica — AI news-outlet 7d ago Anthropic will design its own hardware to power Claude Anthropic and OpenAI are racing to scale up while reducing dependence on Nvidia. 35 r/LocalLLaMA community 7d ago Qwen 3.8 Max now ranked as best overall model ahead of Opus 5 by Artificial Analysis agentic index   submitted by   /u/anderspitman [link]   [comments] 29 r/LocalLLaMA community 7d ago New model release: Ling-3.0-tiny: 7.9B total parameters, with only 1.3B active per token- free for a week A native hybrid reasoning model built for real-world tasks, math, instruction following, and resource-sensitive deployment.   submitted by   /u/niacolhealth [link]   [comments] 8 Hacker News — AI on Front Page community 7d ago Qwen3.8 Max now ranked as the best overall model by agentic index Article URL: https://artificialanalysis.ai/?intelligence=agentic-index Comments URL: https://news.ycombinator.com/item?id=49200652 Points: 267 # Comments: 139 13 Simon Willison community 7d ago datasette 1.0a38 Release: datasette 1.0a38 This release fixes a SQL injection security issue that affects Datasette instances that serve a mixture of public and private tables in the same database, with access configured using the Datasette permissions system . Site administrators who serve… 8 Simon Willison community 7d ago datasette 1.0a38 Release: datasette 1.0a38 This release fixes a SQL injection security issue that affects Datasette instances that serve a mixture of public and private tables in the same database, with access configured using the Datasette permissions system . Site administrators who serve… 20 Simon Willison community 7d ago datasette 0.65.3 Release: datasette 0.65.3 Back-ported the SQL Injection security fix from 1.0a38 . Tags: datasette 29 Simon Willison community 7d ago datasette 0.65.3 Release: datasette 0.65.3 Back-ported the SQL Injection security fix from 1.0a38 . Tags: datasette 16 Page 5 of 10 · 500 articles ← Newer Older →
