News / #edge Tag Edge 373 articles archived under #edge · RSS Sign in to follow r/LocalLLaMA community 14d ago Smallest model (& tips) for intelligent computer use via Hermes? Hello, I have a friend who's using various local LLM's like qwen3.6 27B, 35b-a3b, North Mini Code, and qwen2.5-vl-7b (just for vision). They have a use case where they're trying to have an LLM drive an actual machine via hermes' computer_use tool and cua_driver to click through… 7 r/LocalLLaMA community 14d ago Mechanistic interpretability streamlined for everyday users like us😎 🧠 Context: I want to give the community an Open Research (well open under Apache 2.0 clause) - tool that allows everyday users like us to look deeper into the local models we use consistently. Mechanistic interpretability streamlined into a more visible work-flow. Easy to read for… 26 arXiv — NLP / Computation & Language research 15d ago Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation arXiv:2607.26286v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as general-purpose translation systems, but their behavior is usually evaluated under a single prompt shape: translate one source sentence into one target language. In practice,… 29 r/LocalLLaMA community 15d ago I tested proven orchestration techniques on small local models. 90% failed. The 10% that survived roughly doubled task completion. Hey localllama brochacos, what's up? I'm u/raydestar , long time local llm fan. SWE with about 10 years xp, and I have been cranking hard trying to skill up with agentic AI recently. Since open weights got good, it's just blown my mind. What's been hard to understand is "Why… 26 llama.cpp releases dev-tools 15d ago b10181 ggml-cuda : disable MMQ on devices with less than 48 KiB shared memory ( #26141 ) ggml_cuda_should_use_mmq() selects MMQ purely from the quantization type. The current MMQ configurations are designed and maintained against a minimum of 48 KiB per-block shared memory, the limit… 36 Hacker News — AI on Front Page community 15d ago Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal. I have always adored on-device AI. It feels like magic that you can run a powerful NN on… 29 r/LocalLLaMA community 15d ago A slide deck you can edit with a local model or in Chrome — the whole deck is a JSON block in one HTML file (~640KB with editor and viewer included) Over the past few months, our team has been building more and more slidedecks using web frontend technologies with coding harnesses, but a common complaint is to make even small edits we need to edit the code either manually or via the harness. To avoid this loop, I ended up… 17 r/MachineLearning community 16d ago Vendor-agnostic ML inference on production edge devices [R] I work on PostSlate, a video editing tool, and this comes out of our own work. We run ML models on-device, face detection and embedding among other things, which means we can't assume anything about the user's GPU. NVIDIA discrete, AMD, Intel integrated, Apple Silicon, all of… 7 Hugging Face Daily Papers research 16d ago UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models Abstract Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision… 11 r/LocalLLaMA community 17d ago What’s the maximum physical amount of intelligence we can fit into small models? I was talking to my friend the other day, who is a really avid supporter of local models. He thinks we might be able to get something as intelligent (not smart in terms of how much it knows) as Claude Fable in inside a model which is like, 20 billion parameters, even if we’d get… 29 r/MachineLearning community 17d ago Mix local LLMs, Claude Code, Codex, Gemini and more in one SDLC pipeline (open source) [P] A lot of AI coding tools assume one model should do everything: understand the task, write the code, review it, and decide whether it is correct. I ended up building something around the opposite idea. Instead of one model doing the whole software development lifecycle, every… 22 arXiv — NLP / Computation & Language research 17d ago From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference arXiv:2607.24585v1 Announce Type: new Abstract: We present ELMOD - Efficient Language Model for On-Device Deployment - a compact (2.7B) German language model designed for efficient inference on resource-constrained hardware. ELMOD was trained on a limited computational budget… 30 r/LocalLLaMA community 18d ago What local model do you still use after the hype wore off? Every time a new model is released, I tend to check it out. The benchmarks, readme, or whatever seem pretty convincing, so I download it, test it for a few hours, and then I just go back to the same couple of ones I already had. Curious what models people here have actually kept… 20 arXiv — Machine Learning research 18d ago FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs arXiv:2607.21624v1 Announce Type: cross Abstract: Transformer-based models have enabled unprecedented capabilities across language, vision, and multimodal tasks. On-device fine-tuning of transformer models offers a privacy-preserving path to personalized AI, yet remains… 20 r/LocalLLaMA community 18d ago Unexpected use of local llm I was refreshing my youtube and found out my favourite reviewer uploaded a battery test of 78 smartphones: https://youtu.be/MpgUFrsIWSQ the author said they started using robotic arm to simulate a person using the phone but they wanted to further enhance it by using agentic ai.… 19 r/LocalLLaMA community 18d ago [OSS] Use case only possible with local inference at its core: an on-device LLM understands your entire life, then proactively offers to get your work done through computer use! Open-source & free :D Hey r/LocalLLaMA ! :D I wanna share a really cool fully OSS thing I've been building that's only possible with local models: truly proactive AI! All your existing LLM systems waits for a prompt. Truly proactive AI has to read your entire life, every single day (every file,… 19 r/LocalLLaMA community 18d ago Do people building local LLM rigs track RTX Ada/workstation card prices, or just consumer cards like the 5090? curious how people here approach buying high-end/workstation cards (RTX 6000 Ada, 5000 Ada, etc) for local LLM work, do you actively watch pricing/timing on these specifically, or is the consumer 5090 usually enough for most builds? also wondering if price alerts/tracking tools… 15 r/LocalLLaMA community 19d ago We open-sourced Logue — a privacy-first macOS meeting-notes + writing app that runs on-device (MLX, Apple Silicon) entirely At Bitwize, we've been building Logue, a native macOS app for AI meeting notes and writing, and we just open-sourced it (MIT). We're sharing it here because the whole point is that it runs 100% on-device — we wanted something that could transcribe and summarize meetings without… 27 r/LocalLLaMA community 19d ago Is it worth getting 128GB MacBook Pro? Will it ever be comparable to today’s frontier models for coding? I am a long time iOS app developer. In the last year I have been using Cursor+Claude/others to assist with app development. I am concerned that the current low pricing will disappear eventually. I am pricing out a new laptop with the intention of using local models instead. New… 35 r/LocalLLaMA community 19d ago How much are you actually using your local models these days? Which ones do you reach for the most? I started tracking my local model usage about four weeks ago and was wondering if anyone else here keeps track of how much they use them. I’ve also been running some tests with the cheapest SOTA open-weight Chinese models via OpenRouter. Apart from that, I’m mainly using GPT-5.5… 4 r/LocalLLaMA community 20d ago Who ONLY use local models? Please be honest. I would love to hear about guys really dedicated to local AI and who really reject subscriptions (especially to openai and anthropic). What do you use your model for?   submitted by   /u/takoulseum [link]   [comments] 37 Hacker News — AI on Front Page community 20d ago Android May Soon Restrict On-Device ADB Article URL: https://kitsumed.github.io/blog/posts/android-may-soon-restrict-on-device-adb/ Comments URL: https://news.ycombinator.com/item?id=49045159 Points: 262 # Comments: 140 38 r/LocalLLaMA community 20d ago DKV: Open-source KV-cache compression framework for local LLM inference (CLI + technical report) Hi everyone! Over the past five months I've been working on DKV (DifferentialKV), an open-source project exploring KV-cache compression for long-context local LLM inference. The goal is to reduce KV-cache memory requirements through anchor-based representations, joint low-rank… 21 arXiv — Machine Learning research 21d ago Information-Theoretically Secure Aggregation for Lightweight Federated Learning: Resilient to Dropouts and Adversaries arXiv:2607.20890v1 Announce Type: new Abstract: On-device federated learning (FL) enables privacy-preserving and personalized model training on resource-constrained devices such as smartphones and IoT nodes. To reduce communication cost, sign-based methods (e.g., signSGD)… 20 r/LocalLLaMA community 21d ago I compared local models and different quants / config on a subset of swe-verified bench And gathered a lot of data. you can see them for yourself And For the most curious, there are additional details here In this graph, I regrouped the finetunes under their base models. but you can see the details in the page. The python code to generate those pages is obviously… 18 r/LocalLLaMA community 21d ago CPU-only inference on a Celeron N5095 SBC: 6 models from 0.6B to 8B, benchmarked I wanted to know how cheap you can go and still run local models, so I ran Ollama CPU-only on a Youyeetoo X1S. It's a single-board x86 machine with a Celeron N5095 (Jasper Lake, 4C/4T, 15W), 16GB of RAM, and a 128GB NVMe, running Kali 2025.4. Base configs of this board go for… 28 r/LocalLLaMA community 22d ago Cactus Hybrid: We taught Gemma 4 to know when it's wrong Hey HN, Henry & Roman here from Cactus. A small, on-device model is fast and private, but sometimes wrong, but frontier models are getting expensive pretty fast. So, we post-trained Gemma 4 E2B post-trained to know when it's wrong. Every response comes with a confidence score… 28 r/LocalLLaMA community 22d ago MindControl - llama.cpp fork to guide the reasoning process via injection during sampling The primary driver of this project is that I'd become frustrated with the reasoning behavior of smaller local models such as Qwen3.6-27B (i believe particularly at lower temperatures, and where system prompts are highly specific), their reasoning process is highly unreliable and… 7 r/LocalLLaMA community 22d ago We built NeuTTS-2E, an open-source on-device TTS model with 7 controllable emotions We’re open sourcing an alpha release of NeuTTS-2E : an on-device TTS model with 125M active parameters and 7 controllable emotions. The goal was simple: when you select “angry,” “fearful,” or “happy,” the delivery should follow that instruction rather than whatever emotion the… 14 arXiv — Machine Learning research 23d ago QScheduler: Adaptive Gradient Sampling for Zeroth-Order On-Device Training on INT8 NPUs arXiv:2607.18802v1 Announce Type: new Abstract: Zeroth-Order (ZO) optimization enables On-Device Learning (ODL) on NPU-equipped microcontrollers by estimating gradients through forward passes alone, bypassing the need for backpropagation primitives and reducing memory… 37 r/LocalLLaMA community 23d ago Anthropic claims local models are stealing from it, meanwhile it pays $1.5B for theft $1.5 billion settlement largest known payout in U.S. copyright case Case part of a wave of lawsuits from copyright holders against AI companies Some authors and publishers opted out and continue separate cases against Anthropic   submitted by   /u/Terminator857 [link]… 18 r/LocalLLaMA community 24d ago Today I learnt the power of LocalLlama DISCLAIMER: No Ai was prompted in the creation of this post. Today I had an experience that complely blew my mind, I just had to write it down. As a bit of background I have been dabbling prompting local models using LM studio for the better part of 18 months now, keeping up to… 38 arXiv — Machine Learning research 24d ago Fully-sensorized smart-eyewear platform for on-device Machine Learning arXiv:2607.16222v1 Announce Type: new Abstract: This paper presents ARGO, a smart eyewear platform designed to bridge ergonomic comfort, high computational throughput, and energy efficiency. Unlike cloud-dependent solutions, ARGO leverages the STM32N6 microcontroller and its… 11 r/LocalLLaMA community 24d ago Google has disappeared completely from the top 15 Google hasn't shipped a model recently that is capable of competing with Sol or Fable. The previous models were pretty disappointing and unreliable, it seems the more time goes on that they might have different strategies: - They might be going all-in on on-device inference for… 25 Hugging Face Daily Papers research 24d ago Benchmarking Sensor Robustness in Plasma Diagnostic Models: A Systematic Evaluation on TokaMark Abstract Plasma diagnostic models for tokamak fusion devices are almost universally evaluated on clean, complete sensor data. In practice, fusion diagnostics fail regularly: acquisition systems start late, individual sensors die, and signal dropouts cluster precisely when a… 13 r/LocalLLaMA community 24d ago No-coding model? Mostly all of the local models these days are competing for coding benchmarks. Is there any lab that just focuses all of their attention on making a better creative writing model?   submitted by   /u/Lost_Care7289 [link]   [comments] 11 r/LocalLLaMA community 24d ago Is it possible to run a local model focused solely on "intelligence" and outsource its "knowledge" to web searches? I'm looking to run a very lightweight local model that acts as the brain, handling the logic and comprehension, while hooking it up to a web search tool to act as its memory and knowledge base.   submitted by   /u/chucrutcito [link]   [comments] 26 arXiv — Machine Learning research 25d ago From Feasibility to Desirability: Plan, Learn, Adapt (PLA) Framework for Personalized On-Device Itinerary Generation arXiv:2607.15552v1 Announce Type: new Abstract: Generating personalized trip itineraries is a complex planning task and involves a tension between hard combinatorial feasibility and soft latent desirability. Classical optimization enforces constraints but fails to capture… 15 r/LocalLLaMA community 26d ago What’s your favorite underrated local model? What’s your favorite underrated local model that you actually use every day? I’m not talking about the mainstream choices like Qwen 3.6 or Gemma 4. I’m looking for the hidden gems that deserve more attention. What do you use it for, and what hardware are you running it on? I’m… 20 r/LocalLLaMA community 26d ago Qwen vs Gemma Hi! Been doing some local LLM stuff, and I can't help but notice: despite vastly-superior benchmark scores, Qwen 3.6 35a3B feels... substantially less intelligent than Gemma 4 26a4B (QAT). In terms of prompt adherence, output coherence, and just general "sanity", Gemma seems… 4 r/LocalLLaMA community 26d ago Sharing MiniBot v2, this is what I'm currently using I gave it a major update so I thought I'd share. I make things that do work for me, always have... and this is the latest. It a single file 20k lines ;P https://github.com/illsk1lls/MiniBot (i previously posted v1 of this, which was not WPF) This is for local models only. Although it is OpenAI compatible. Have your favorite AI scan it to make sure its safe enough for you... Tools are enabled as… 28 r/LocalLLaMA community 26d ago TUI app building on Rust How are people building nice looking TUIs with their local models? So far I’ve had zero luck, and my TUIs in Rust that local models build look like shiet   submitted by   /u/Infinite-Ad4512 [link]   [comments] 27 r/LocalLLaMA community 27d ago Local LLM project Is it worth running local models on this old beast? Dell PowerEdge R710 (2009-12 era) Dual Xenon 5500 (I'm pretty sure) 48Gb DDR3-1066   submitted by   /u/Motor-Independent572 [link]   [comments] 18 r/LocalLLaMA community 27d ago A year ago you told me my open-source screen-watching app was flaky. You were right, so I spent the year fixing it with your feedback. Thank you r/LocalLLaMA c: !! TL;DR: This post is part update, mostly thank you for your support :)) Observer is an open-source app that lets local LLMs watch your screen and notify you (WhatsApp/SMS/email/Discord) when something happens. A year of your feedback later , setup went from "flaky and very… 31 arXiv — Machine Learning research 28d ago PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference arXiv:2607.14618v1 Announce Type: new Abstract: CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs. We… 23 r/LocalLLaMA community 28d ago I’m taking a break I’ve just formatted my MacBook Pro with an M2 Max and 32GB of RAM. I’m taking a break from local LLMs. I’ve spent the last few months tinkering with local models, MTP, quants, harnesses, Gemma, Qwen… It’s time to take a break. I noticed this was becoming less of a hobby and more… 28 r/LocalLLaMA community 28d ago I just got my first GPU that can actually run an LLM (Laptop 5090 24 GB) what do I play with first? I figure I can run 30b or even 72b models on it, but this is my first time running my own local LLM. I want to play with prompt engineering and see the most complex things I can get it to do. Any hot tips? I know it all moves fast and this sub is wired in I have 32GB ram also… 24 Hugging Face Daily Papers research 29d ago PalmClaw: A Native On-Device Agent Framework for Mobile Phones Abstract Large Language Model (LLM) agents have moved beyond generating responses to executing multi-step tasks by calling tools, observing the results, and iteratively deciding the next action. Most agent systems run on desktops or servers, which support tool use and task… 19 r/LocalLLaMA community 1mo ago If you had a 384GB (4x Blackwell), what model would you put on it and why? Hey guys, Company I work for is actually very interested in spending the money to host our own local model for the team. We expect probably 2-3 super users and at the worst case 10-20 concurrent users. The LLM would be mostly used for internal company policies/data management… 10 arXiv — Machine Learning research 1mo ago Benchmarking Sensor Robustness in Plasma Diagnostic Models: A Systematic Evaluation on TokaMark arXiv:2607.11915v1 Announce Type: cross Abstract: Plasma diagnostic models for tokamak fusion devices are almost universally evaluated on clean, complete sensor data. In practice, fusion diagnostics fail regularly: acquisition systems start late, individual sensors die, and… 19 Page 2 of 8 · 373 articles ← Newer Older →
