News / #edge Tag Edge 373 articles archived under #edge · RSS Sign in to follow r/LocalLLaMA community 2mo ago LiquidAI/LFM2.5-8B-A1B · Hugging Face looks like you can run it on any potato (A1B)! https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF from LiquidAI: LFM2.5 is a new family of hybrid models designed for on-device deployment. It builds on the LFM2 architecture with extended pre-training and reinforcement learning.… 22 r/LocalLLaMA community 2mo ago Qwen3.6 35B - TXT vs Markdown vs HTML vs HTML+CSS Theres been talk of late about using HTML rather than markdown in Claude Code. I was curious how this worked with a local model so loaded up Qwen3.6 35B A3B at Q8 and F16 KV cache. Then I gave it the same prompt write a detailed explanation of the Blazor render cycle first… 31 The Information — AI news-outlet 2mo ago Apple to Renew Push for AI That Runs on Devices, Instead of the Cloud At Apple’s annual developer conference next month, the star of the show will be a series of long-delayed artificial intelligence upgrades to the iPhone. But the company is also expected to emphasize what could be an underrated asset in its efforts to catch up in AI: Its ability… 26 r/LocalLLaMA community 2mo ago Heterogeneous GPU Weighting & Layer Splitting This is what I worked on today. With local LLM of course. So if I didn't write the code, did I really work on it? Who cares. It was my idea and I simply asked it to implement it. I basically downloaded /main/ branch, which is totally broken for Windows by the way (i had to… 21 arXiv — Machine Learning research 2mo ago The Energy Blind Spot: NVIDIA's Flagship Edge AI Hardware Cannot Support Process-Level Energy Attribution arXiv:2605.27599v1 Announce Type: new Abstract: Agentic AI workloads - where a single user goal triggers multi-step orchestration, tool calls, retries, and failure recovery - are being targeted for edge deployment, with NVIDIA, Dell, HP, ASUS, MSI, Acer, and Gigabyte all… 37 r/LocalLLaMA community 2mo ago Local LLMs on Refurb M4 Max vs new M5 Max Hoping the community can guide me on this one. I'm on the fence about the following purchase: Refurbished 16-inch MacBook Pro Apple M4 Max Chip with 16‑Core CPU and 40‑Core GPU, 64gb ram, 1Tb Drv for $3,479.00 vs The new 16-inch MacBook Pro Apple M5 Max Chip with 18‑core CPU,… 30 r/LocalLLaMA community 2mo ago CrankGPT by Squeez Labs - hand-cranked edge AI - talk about local AI!!! I met Katrin from Squeez Labs at an event hosted by Pathway AI (the team behind Baby Dragon Hatchling) where she told me about CrankGPT, a literally hand-cranked device for running local LLMs. It's apparently real. It's appearently launched. It's apparently glorious. Check it… 15 r/LocalLLaMA community 2mo ago Qwen3.6 huge quality gain from Q4 to Q6 for coding agent So, last week I tried to update my unused local LLM setup. I had to stop using it because quality was too low and deepseek was too cheap. First thing I stopped using Ollama and now I only use llama.cpp built in server that works really great. The quality improvement from Q4 to… 34 arXiv — Machine Learning research 2mo ago The Constraint Tax: Measuring Validity-Correctness Tradeoffs in Structured Outputs for Small Language Models arXiv:2605.26128v1 Announce Type: new Abstract: Production LLM systems increasingly require machine-readable outputs: JSON objects, typed traces, regex-constrained fields, and tool-call schemas. This paper targets on-device and low-cost small language model (SLM) deployments,… 24 arXiv — Machine Learning research 2mo ago Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling arXiv:2605.26496v1 Announce Type: new Abstract: The Mixture of Experts MoE architecture is highly promising for resource constrained on device deployments yet training these models from scratch incurs prohibitive costs Current methods attempt to alleviate this by upcycling dense… 32 Hugging Face Daily Papers research 2mo ago MobileMoE: Scaling On-Device Mixture of Experts Abstract MobileMoE introduces efficient on-device Mixture-of-Experts language models with sub-billion parameters that achieve better performance and efficiency compared to dense baselines and existing MoE models. AI-generated summary Mixture-of-Experts (MoE) has become the de… 17 arXiv — Machine Learning research 2mo ago Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning arXiv:2605.24058v1 Announce Type: new Abstract: On-device adaptation of large language models commonly keeps a quantized base model frozen while training and deploying a small, task-specific LoRA adapter. In the unmerged adapter-mode setting, however, the adapter is more than a… 28 r/LocalLLaMA community 2mo ago New local model reaching near frontier on PII removal at 9 ms CPU inference Hi all, I've been working on this model to strip sensitive information from computer use data and would love some feedback!   submitted by   /u/louis3195 [link]   [comments] 34 r/LocalLLaMA community 2mo ago Using Local LLMs for Generating Custom Interactive Recursive Textbooks on the Fly   submitted by   /u/Ryoiki-Tokuiten [link]   [comments] 28 llama.cpp releases dev-tools 2mo ago b9315 llama : document that only one on-device state can be saved per sequence ( #23520 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64… 13 r/MachineLearning community 2mo ago Is AI inference platform really that saturated now? [D] I’m thinking of expanding an on-device inference SDk into a full blown AI inference platform and seeing more and more inference platform popping out. Been talking with a VC from Seattle/NY. Is this space really that saturated?   submitted by   /u/kampak212 [link]  … 35 r/LocalLLaMA community 2mo ago RAG for developer docs so local llm can code using latest library? I was wondering if it would make local llm better at coding if it has access to the latest documentation available through a RAG. I'm specifically interested in python. But then this might lead ingesting and embedding a very large number of documents. Or I could just focus on… 28 r/LocalLLaMA community 2mo ago server: fix checkpoints creation by jacekpoplawski · Pull Request #22929 · ggml-org/llama.cpp Imagine you are using a local model for agentic coding. You discuss the idea (50k tokens), then say “implement it”. The agent reads files, writes files, runs commands, produces another 20k tokens and the code is ready. Then your next prompt is just “thank you”, and... nothing… 6 r/LocalLLaMA community 2mo ago llama.cpp has a clever trick for speeding up KV cache decode So, I use llama-server as my endpoint to run local models and connect them to Open-WebUI, Hermes, and OpenCode. But since llama.cpp's webUI has been receiving a lot of updates, I took a look at its settings and noticed a particular one under developer options. This is the… 23 r/LocalLLaMA community 2mo ago Is NVIDIA still the default best choice for local LLMs in 2026?   submitted by   /u/pmv143 [link]   [comments] 9 r/LocalLLaMA community 2mo ago Local model doing accounting tasks So I've been using qwen 3.6 27b for monthly closes, bank recs, payable and receivables. Built a simple sql lite database it manages. Anyhow, wanted to post I integrated Claude skills and the https://github.com/anthropics/financial-services repo. It works well. Just wanted to… 22 r/LocalLLaMA community 2mo ago club-rdna16: practical 16GB AMD/Radeon local LLM testing repo Following on from club-5060ti, I’ve been doing some testing with my desktop AMD GPU and wanted to make a similar repo for 16GB Radeon cards. Repo: https://github.com/5p00kyy/club-rdna16 Pages/results: https://5p00kyy.github.io/club-rdna16/ The first test machine is an RX 6900 XT… 24 r/LocalLLaMA community 2mo ago Gmail tie-ins hey folks. I’m looking to setup a way to give a local LLM access to google cloud SDK for Gmail functions. The goal is to be able to have an LLM once daily check a spreadsheet, and based on criteria send an email that will be structured exactly the same way each time, simply as a… 14 arXiv — Machine Learning research 2mo ago Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization arXiv:2605.20295v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed on mobile devices, where Neural Processing Units (NPUs) necessitate fully static quantization for optimal inference efficiency. However, existing post-training quantization… 36 arXiv — NLP / Computation & Language research 2mo ago GraphRAG on Consumer Hardware: Benchmarking Local LLMs for Healthcare EHR Schema Retrieval arXiv:2605.20815v1 Announce Type: new Abstract: Graph-based Retrieval Augmented Generation (GraphRAG) extends retrieval-augmented generation to support structured reasoning over complex corpora, but its reliability under resource-constrained, privacy-sensitive deployments… 32 r/LocalLLaMA community 2mo ago 24GB M4 Mac - is Qwen 9B only option while system is running? I have mac at work that I want to use local model for prototyping and basic prompts that needs to stay on device. What sort of model I can run that I can fit at least 64k context ? Any setups share or guides welcome. I need to have firefox open with one tab at minium. Problem I… 6 TechCrunch — AI news-outlet 2mo ago Stability AI releases a new audio model that can create six-minute songs Stability Audio 3.0 small model can run on-device and generate two-minute long tracks 21 r/LocalLLaMA community 2mo ago How accurate can “whichllm” be? Hello people I think the question is clear but I wanted to add some context: I work on internal tools in my job and some of the tools are for us developers (most tools are for marketing and factory production). I am currently working on a small cli tool that uses a local model… 12 r/LocalLLaMA community 2mo ago what non-coding tasks have you gotten a local model to do autonomously? coding agents are everywhere right now but i'm more interested in models that actually take actions autonomously. we built a small vlm for desktop gui automation. i mostly use it for moving data between apps that don't have apis, saves me a lot of copy pasting. still kinda janky… 11 r/LocalLLaMA community 2mo ago Audio upscaling, cleanup, or improvement models? I never see this type of model talked about. Are there many open models in the category? I do a lot of audio cleanup and end up using auphonic but would like to be using a local model. Edit: e.g like voice recovery, reverb removal, auto-EQ type stuff   submitted by  … 5 arXiv — Machine Learning research 2mo ago R2V Agent: Teaching SLMs When to Ask for Help arXiv:2605.16604v1 Announce Type: new Abstract: Efficient agentic systems should incur expensive frontier-model costs only on decisions where a cheaper local model is likely to fail. Existing LLM cascades usually route whole queries before execution, but task difficulty shifts… 18 arXiv — NLP / Computation & Language research 2mo ago Language Acquisition Device in Large Language Models arXiv:2605.16758v1 Announce Type: new Abstract: Large Language Models (LLMs) remain substantially less data-efficient than humans. Pre-pretraining (PPT) on synthetic languages has been proposed to close this gap, with prior work emphasizing highly expressive formal languages… 32 arXiv — NLP / Computation & Language research 2mo ago From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG arXiv:2605.18271v1 Announce Type: new Abstract: With the rapid emergence of personal AI agents based on Large Language Models (LLMs), implementing them on-device has become essential for privacy and responsiveness. To handle the inherently personal and context-dependent nature… 16 r/LocalLLaMA community 2mo ago What’s your current local LLM setup in 2026? Hey all — I’ve been trying to get a better sense of what people are actually running locally these days. Curious about your setup: GPU (or CPU if you’re brave ) RAM / VRAM Models you use the most Main use case (coding, chat, agents, etc.) Also — what’s the biggest bottleneck… 24 r/LocalLLaMA community 2mo ago club-5060ti follow-up: cleaner RTX 5060 Ti local LLM recipes, benchmark explorer, and CUDA GPU compatibility notes I posted earlier about RTX 5060 Ti local LLM testing, and I have cleaned the repo up quite a bit since then. The project is now a more structured benchmark/recipe repo rather than scattered notes. It has a static results explorer, schema-validated benchmark JSON, clearer… 34 Zed Editor dev-tools 2mo ago Why and How to Run Local Models in Zed You can run local AI models in Zed to get better performance and control over your data. Here's how. 33 r/LocalLLaMA community 2mo ago favorite Agentic Coding Harness So far, I’ve tried Codex CLI, Claude Code, Gemini CLI, OpenCode, and recently, Pi with local models. Pi is the leanest of them all, with just four tools: read, write, edit, and bash. Its system prompt is only under 2K tokens, and it's perfect for local models. I've been trying… 29 The Information — AI news-outlet 2mo ago Edge Inference Chip Startup SiMa.ai Raising at $1.4 Billion Valuation Nvidia might be on a tear, but some investors are still convinced that there’s demand for another kind of specialized chips. And they’re putting their money where their mouth is. For example: San Jose, Calif.-based SiMa.ai , which develops chips that work on devices such as… 14 r/LocalLLaMA community 2mo ago What happens to local LLM if/when LLMs are no longer released for free? I’m thinking about where this might wind up in 3-5+ years. As others have noted there’s no guarantee that Qwen, Google, and others will continue to release models in the future. Suppose the supply of new LLM models dries up overnight. Whatever is available today, May 2026, is… 6 r/LocalLLaMA community 2mo ago Is anyone prioritizing code quality checks via a small local model? Sorry if the title is confusing. What I'm trying to say is that since coding agents can write a lot of code very quickly and it can kinda get messy overtime if unchecked frequently. Shouldn't there be a tiny local model with a TESTING(dot)md or a QUALITY(dot)md which describes… 14 r/LocalLLaMA community 2mo ago I built a coding agent that gets 87% on benchmarks with a 4B parameter model, here's how I was frustrated that every coding agent (OpenCode, Cursor, Claude Code) assumes you're running GPT-5.4 or Claude Opus. If you try them with a local model like Gemma or Qwen they fall apart. I find that often tool calls fail, context overflows, multi-step tasks collapse. So I… 12 r/LocalLLaMA community 2mo ago The power of structured workflows and small local models A month ago, I experimented with a very basic home-rolled agent loop with a handful of tools and found it worked surprisingly well in spite of how crude it was: https://www.reddit.com/r/LocalLLaMA/comments/1sl7f8e/homerolled_loop_agent_is_surprisingly_effective/ Later, I wrote… 15 r/LocalLLaMA community 2mo ago Made a simple template manager and GUI for llama.cpp so I don't have to keep memorizing CLI flags. Introducing Hexllama Hey, I’ve always found llama-server to be more than enough for testing out local models, mostly because it guarantees you always have the absolute latest llama.cpp features and architecture support. But keeping track of different CLI commands, context sizes,… 19 r/LocalLLaMA community 2mo ago Using Local LLMs for research Hey there. I am an undergrad who has been doing mostly SWE, but will be doing ML research under my professor over the summer. So I am new to research - I ask not to be judged too harshly. Generally, we will be working on Physics-Informed Neural Networks. I have seen some… 9 r/LocalLLaMA community 2mo ago LLM Phone Home: Reliable Apps that can deliver inference from local backend Hello all, I’m wondering what suggestions there are for an ios app that can serve an openai compatible endpoint. I am using 3sparks which works GREAT for that specific use, BUT, there is no mcp, no web search, etc. I want to show people that a local model with web search on your… 25 r/LocalLLaMA community 3mo ago What’s are the best abliterated or uncensored local models that allow financial advice-related questions? Not trying to get rich quick or anything, but I’m just tired of models refusing to answer questions related to their opinions on money matters or having them be wishy-washy about financial decision making advice. Seems like this can be a blocker with both frontier closed source… 32 r/LocalLLaMA community 3mo ago I built a self-hosted open-source MCP server that gives any local LLM real financial data — SEC filings, 13F, insider & congressional trades, short data, FRED One thing missing when running local models as agents: real, current data. So I built Equibles — a self-hosted MCP server that scrapes and serves public U.S. financial data and exposes it as MCP tools, so any MCP-capable client (Claude Code/Desktop, Cursor, or your own… 30 r/LocalLLaMA community 3mo ago how would you set up a local llm server for a business of 7 people? Okay so i've been stalking this sub for some time and i run the occasional small 2-8b model on my laptop (not the best) for fun but say my role at a company is to set up a local LLM since we obviously don't want confidential data going to other companies etc / main use case… 16 r/LocalLLaMA community 3mo ago Are the rich RAM /poor GPU people wrong here? Hello Guys, I know everyone has his definition of local models, but for me i see 2 "reasonable" type of frontier local models. a dense one that barely fit in a 32GB ou 24GB of gpu for the most "reasonable" GPU wealthy guys and a MOE in the 100B params, the 100ish B billion… 21 r/LocalLLaMA community 3mo ago Gemma 4 + LiteRT-LM on mobile: much better memory/perf than my llama.cpp setup Hi r/LocalLLaMA - I've been paying close attention to the edge AI ecosystem because it's an area where i see huge potential and where I truly believe AI will become more useful for day to day tasks. Around the gemma 4 release I was already experimenting with local AI but the… 18 Page 7 of 8 · 373 articles ← Newer Older →
