News / #edge Tag Edge 373 articles archived under #edge · RSS Sign in to follow arXiv — NLP / Computation & Language research 1mo ago PalmClaw: A Native On-Device Agent Framework for Mobile Phones arXiv:2607.13027v1 Announce Type: new Abstract: Large Language Model (LLM) agents have moved beyond generating responses to executing multi-step tasks by calling tools, observing the results, and iteratively deciding the next action. Most agent systems run on desktops or… 25 arXiv — NLP / Computation & Language research 1mo ago On-Device Deep Research at 4B: Exposure Bounds Faithfulness, Retrieval Bounds Coverage arXiv:2607.12257v1 Announce Type: cross Abstract: On-device research agents search a corpus, read sources, and write a cited brief on a personal laptop. Whether their citations are faithful, and at what cost, is unmeasured for a deployable small model. This study fixes one 4B… 13 llama.cpp releases dev-tools 1mo ago b10007 opencl: fix a dp4a bug for devices where cl_khr_integer_dot_product is unavailable ( #25639 ) opencl: do not fail backend init on devices without cl_khr_integer_dot_product opencl: do not call dp4 kernels when dp is unavailable Co-authored-by: Li He [email protected]… 10 r/LocalLLaMA community 1mo ago Good podcasts I'm heading on vacation soon and want to download a few good podcasts about local LLMs, open-weight models, inference, tooling and the broader open-source AI ecosystem. Which podcasts or specific episodes do you genuinely recommend? I'm especially interested in technical… 18 r/LocalLLaMA community 1mo ago Hermes Agent with local LLMs Hey all, I have a 4090 + 5060ti setup with 64gb ram. I have been using opencode with MiMoV2 pro and its great but I don't think spending $20 daily is financially sound decision. I heard that using qwen3.6-27b with hermes goes into loops, have any of you successfully used any… 30 arXiv — NLP / Computation & Language research 1mo ago Workload-Driven Optimization for On-Device Real-Time Subtitle Translation arXiv:2607.09957v1 Announce Type: new Abstract: This report studies on-device English-to-Traditional-Chinese subtitle translation for Taiwan under short inputs, short outputs, batch-size-one inference, low latency, and privacy constraints. These conditions limit the value of… 23 r/LocalLLaMA community 1mo ago Excel work - best model I’ve been testing various local models for excel tasks related to my job. So far, Deepseek v4 Flash using DS4 has been best. Gets 30-40 tk/s with medium context and generally produces good results. However, looking to see what others may have had good experiences with. Need to… 23 r/LocalLLaMA community 1mo ago This is why we need local models and opensource harnesses   submitted by   /u/Comfortable-Rock-498 [link]   [comments] 19 arXiv — Machine Learning research 1mo ago On-Device Adaptive Battery Power Prediction for Electric Vehicles arXiv:2607.09400v1 Announce Type: new Abstract: Adaptive power management in Electric Vehicles (EVs) requires accurate power prediction. Although deep learning models have emerged as highly effective for time-series forecasting in this domain, their performance is prone to… 17 r/LocalLLaMA community 1mo ago How is Codex as a harness for local models? I was surprised to see that Codex is actually open source, so to my understanding, if you use a local model, it works fully locally. How does it compare to the other popular harnesses like Pi Code and Open Code?   submitted by   /u/A_Wild_Entei [link]   [comments] 4 llama.cpp releases dev-tools 1mo ago b9974 cuda: Don't crash when querying memory on device with no free memory. ( #25157 ) If a Cuda device has no or limited available memory, the actual call to cudaMemGetInfo() itself can cause a fatal crash due to a cuda out of memory error (there is not enough memory to actually… 38 r/MachineLearning community 1mo ago Zer0Fit: I took Google's new TabFM & TimesFM ML foundation models and made them available as an MCP server for zero-shot ML tasks (forecasts / classifications / regressions). 100% local. [P] TL:DR: I’m a grad student in AI, I saw that Google released TabFM and TimesFM last week, I built an MCP wrapper to serve both transformer models in a single Docker container so you can connect their new ML transformer models to a local LLM via Open WebUI, Claude Code, or Codex… 10 r/LocalLLaMA community 1mo ago Working around Qwen3.6-27B's tool-call failures and looping Let's start a discussion about what can be done to make local models more reliable. I've been using Qwen3.6-27B a lot lately, and have noticed the same thing that many others talk about here, which is the tool-call failures and looping that really gets in the way of being able… 33 r/LocalLLaMA community 1mo ago Zer0Fit: I took Google's new TabFM & TimesFM ML foundation models and made them available as an MCP server for zero-shot ML tasks (forecasts / classifications / regressions). 100% local. TL:DR: I’m a grad student in AI. I saw that Google released TabFM and TimesFM last week. I built an MCP wrapper to serve both transformer models in a single Docker container so you can connect their new ML transformer models to a local LLM via Open WebUI, Claude Code, or Codex… 18 r/LocalLLaMA community 1mo ago Opencode Agents vs Claude Code I’ve been playing around with Opencode and realized how 70% of the capability of my model comes from the agents I can use rather than the model size or parameters. So now obviously I have a question… is there a way to use Claude Code but have it pointing at my local model… 7 r/LocalLLaMA community 1mo ago Vellium v1.0.0 released: security hardening, wallpaper-based themes, JSON chat export and a major desktop stability pass Vellium has reached v1.0.0. It is a local-first desktop workspace for writing, roleplay, character creation, lorebooks and knowledge management with local LLMs. This release promotes the previous v1.0.0-beta build to the first stable version. The main focus was security and… 13 r/LocalLLaMA community 1mo ago MIT LLM Serve Dashboard I am making open source A single-file, dependency-free live dashboard for your local LLM serving box — GPU utilization, per-model throughput, KV/context fill, and system stats for llama.cpp and vLLM , in one green terminal-styled page. No framework, no build step, no external requests. The frontend is… 31 r/LocalLLaMA community 1mo ago A game about Simulation Theory that includes an LLM. I made a game called "Simulation Simulator" . It's a freeform conversation game where you try to convince your AI best friend that reality is a simulation and you're inside a video game. The game has a local LLM packaged inside of it that you can run entirely offline. Been about… 37 r/LocalLLaMA community 1mo ago Should I get a 5090 or a strix halo? tl:dr; RTX5090 for 3400€ or Bosgame M5 AI for 2500€? I have a fairly new computer with a 5080 and 64Gb of RAM. I've been having loads of fun with local LLMs. In the end I find myself using thu free Claude and Deepseek V4 Pro through their API because it's so fucking cheap and… 9 r/LocalLLaMA community 1mo ago Has anyone created a "Local LLM Survival Kit"? Here's what I'm thinking about: A USB thumb drive that you can plug into any PC or laptop, and immediately get a usable knowledge base powered by an LLM, without requiring an Internet connection. I believe the technology for this should be ready. Rough architecture: llama.cpp… 21 r/LocalLLaMA community 1mo ago I built barebrowse: give a local-model agent a browser without Playwright — pruned ARIA snapshots instead of raw HTML (far fewer tokens) Author here, sharing something I built. If you run agents on a local model, feeding a whole page as raw HTML burns your context fast. barebrowse turns a URL into a pruned ARIA snapshot — the semantic tree with nav/ads/boilerplate stripped — so each page is a fraction of the… 10 Hugging Face Daily Papers research 1mo ago A Quantized Native Runtime for On-Device Semantic Audio Generation Abstract A dependency-free runtime enables efficient text-to-music generation on embedded devices through quantization and activation steering while maintaining audio quality. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Semantic audio applications increasingly require… 22 r/LocalLLaMA community 1mo ago Neuralwatt Pricing will Double From 07/16 - Got This Email Welp, the age of cheap tokens coming to an end. Many here use this service to access GLM 5.2. Looks like only sustainable way moving forward to local LLM.   submitted by   /u/BoogerheadCult [link]   [comments] 26 arXiv — Machine Learning research 1mo ago Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment arXiv:2607.08029v1 Announce Type: new Abstract: The emergence of vision language models with fewer than 3 billion parameters has accelerated the implementation of on-device multimodal intelligence. However, a detailed understanding of component-wise quantization remains a… 12 arXiv — NLP / Computation & Language research 1mo ago Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment arXiv:2607.08268v1 Announce Type: cross Abstract: High-volume structured extraction pays a large model's latency on every item, so distilling the task into a small on-device model is attractive: comparable output at a fraction of the time and cost. We measure what that… 11 Hugging Face Daily Papers research 1mo ago CineMobile: On-Device Image-to-Video Diffusion for Cinematic Camera Motion Generation Abstract CineMobile enables efficient image-to-video generation on mobile devices through distillation-guided pruning, diffusion distillation, and hybrid quantization techniques while maintaining visual quality and achieving significant speedup. Generated by… 6 r/LocalLLaMA community 1mo ago If You Already Pay for an LLM Service, Running Local Embeddings and Rerankers Feels More Useful Than Running Local LLMs https://preview.redd.it/v0xtn3jdu9ch1.png?width=2047&format=png&auto=webp&s=628a6a541fe5f097d0f771ae0ba3b7f44126198f https://preview.redd.it/vjxiucsdu9ch1.png?width=2047&format=png&auto=webp&s=74f7a18a5a30276e206e2bfb5a0c529826ce86e4 This post was originally written in Korean,… 33 arXiv — Machine Learning research 1mo ago Robust Federated Learning Under Real-World Client Churn arXiv:2607.06979v1 Announce Type: new Abstract: Federated Learning (FL) enables training shared models on private, on-device data, but production deployments remain constrained to slow, multi-day refresh cycles due to the complexity of coordinating massive client populations.… 9 arXiv — NLP / Computation & Language research 1mo ago Billions of Sketches Reveal Hidden Cultural Variation in Human Concepts arXiv:2607.07267v1 Announce Type: cross Abstract: Claims about the universality of human concepts have been predominantly assessed through linguistic similarity across languages and cultures. However, words are effective as communication devices because they compress rich… 34 r/LocalLLaMA community 1mo ago Can you explain the concept behind each of the main size ranges of LLM models, as in, what hardware setups the different size niches are meant to fit into (~30b, ~70b, ~120b, ~230b, etc). Like is it mainly based on pro hardware sizing for 8-bit, or consumer GPU vram for ~Q4, or… I am curious about intended sizings of the main size niches of the popular local LLM models. As in, we can see there is a major niche at 26b-35b, then hardly anything from 36 through 69b, then (formerly) another major niche at ~70b-72b, then another niche at ~120b-123b, then… 22 r/LocalLLaMA community 1mo ago What GUI-first coding tool tool are you pairing your local LLMs with? Opencode isn't it for me. I've grown very frustrated with OpenCode. The web GUI and desktop app ideas are good, but the execution not so much. The GUI is lacking so many basic features. It's clear that the TUI is more important to the devs. Is there anything free that provides a more feature-rich GUI?… 14 r/LocalLLaMA community 1mo ago Complete local model asset generation pipeline So I figured I'd update the community given I just shipped a nice little feature set and feel like sharing it finally :) In the past few weeks, I've been test-coding an isometric RPG game/engine in Three.js, as part of my research into how LLMs work at scale in higher quality… 20 r/LocalLLaMA community 1mo ago The standard free ChatGPT LLM you get after a few messages HAS to be some sub-20b model with online search enabled, no other way to explain how awful it is Recently bought into the local LLM hype by buying a 32gb vram gpu and holy shit, gemma 4 31b at 5bits blows the standard ChatGPT model out of the fucking water. I just can't unsee the quality difference now that I've experienced it. Does Openai just cut costs by serving their… 34 r/LocalLLaMA community 1mo ago Can you trust local models to answer accurately? My goal is to improve as a developer, thus I needed to know if local llms can answer technical questions accurately The conclusion is that without rag they don't do too well, but with rag they are very good. Thinking didn't really help, and took so long I only got the scores for… 29 r/LocalLLaMA community 1mo ago I made a tool that chains a small local model into a big coding model and auto-unloads VRAM between them A couple weeks ago I shared PromptChain here a small Streamlit app that chains two models: a little Prompter that rewrites your rough idea into a proper prompt, then a larger Coder that turns that prompt into code. The whole point is that on an 8–16 GB card you can usually only… 18 r/LocalLLaMA community 1mo ago Are there any local ASR models that surpass Whisper right now? Hey everyone, I'm currently using faster-whisper(medium/large turbo) for local speech recognition, running it on an 8GB VRAM GPU. It works great, but I was wondering if there are any new open-source/local models that outright beat Whisper at this point? Here is exactly what I'm… 18 arXiv — Machine Learning research 1mo ago STELLA: Efficient Sensor-to-LLM Translation for On-Device Human Activity Recognition arXiv:2607.03089v1 Announce Type: new Abstract: HAR is increasingly expected to run continuously on edge devices, yet recent LLM-based methods remain hard to deploy: raw sensor prompts are long, cloud inference adds latency and privacy risk, and fine-tuned LLM pipelines turn… 33 arXiv — Machine Learning research 1mo ago Decentralised Federated Learning over Temporal Networks: The Role of Heterogeneities arXiv:2607.03171v1 Announce Type: new Abstract: Decentralised federated learning, based on peer-to-peer communication, is increasingly proposed for on-device training of machine learning models, promising a privacy-preserving, communication-efficient training process with no… 30 r/LocalLLaMA community 1mo ago UPDATE: I built a tool to turn your Claude Code sessions into fine-tuning data for local models (You can now convert your Codex and Pi sessions) A few days ago I shared this resource I created to convert your Claude Code sessions into training data (Thank you so much for all the support :D ): Original Post Today I'm sharing that I just released version 1.5.0, which now supports converting your Claude Code, Codex, and Pi… 13 r/LocalLLaMA community 1mo ago Running Gemini nano locally. You know that chrome lately downloads a local model (Gemini Nano). Which probably is a gemma vision quantized model. I tested it inside the browser, but I wonder, how could I load the "weights.bin" file from a linux shell? Both tensorflow and llama.cpp fail to recognize it. It… 10 r/LocalLLaMA community 1mo ago Local models + big context = slow. How are you orchestrating "map-reduce" style agent workflows? I tried running local models (qwen3.6*, ds4 flash, gemma4*, etc) on my mbp pro m5 with 128Gb of unified memory and concluded the bottleneck is context size. The moment a conversation gets long (16k is already the bottleneck), inference slows to a crawl. If you work with Hermes… 35 r/LocalLLaMA community 1mo ago Prefill vs. decoding and local LLM ROI: is prefill underrated? I'm trying to understand why, when people discuss the ROI of running LLMs locally, they almost always focus on output speed (decoding) and rarely on input speed (prefill), which seems like it could have a significant impact on hardware ROI. Yesterday I saw a post on X where… 7 r/LocalLLaMA community 1mo ago OpenComputer | An Open Source Computer Built For Agents. Open Computer running in an isolated VM with inference running M4 Pro via LM Studio Gemma 4 13B QAT Hey everyone, Tim from AnythingLLM , where we have been building productive an on-device agent and AI assistant experience for the past 2.5 years now. I want to talk about a new… 15 r/MachineLearning community 1mo ago Edge AI ASL Recognition on Raspberry Pi 5 – Looking for Feedback on My System Design [P] Body:   submitted by   /u/Unlikely_Let_9147 [link]   [comments] 6 r/LocalLLaMA community 1mo ago <15B local models to test on medical benchmark? HI, in a few weeks the medical exam for italian residents will drop, and I would like to compare small models that run on my laptop against it. It would not be a standard benchmark for llms but it would give a sense on how much better does it perform compared to the average… 7 r/LocalLLaMA community 1mo ago Is DeepSeek v4 (Flash) really extremely cheap to run? If yes, how? Hi. I don't have a GPU. So my biggest "local LLM" experience has been running ~26B models with single-digits tps values. However, the "serving economy" of DSv4 models look like a riddle to me. The Flash model has 284B parameters, but providers (e.g. OpenRouter) charge so little… 20 r/LocalLLaMA community 1mo ago Local LLM for legal-document adaptation keeps hallucinating citations with total confidence — grounding/model/pipeline ideas? ## Why local is non-negotiable I'm a criminal defense lawyer and the documents I'd feed this contain **client-confidential case material**. Sending real case content to a cloud API isn't an option for me, so this has to run on my own hardware. Frontier models are clearly better… 13 r/LocalLLaMA community 1mo ago What do you guys use local models for? I previously built a telegram bot that chatted with people. But it was never put in production. Besides that I often find that even the cheap tier premium models like ChatGPT 4.1mini are a little bit too stupid to do things. They just don't seem to grasp the job as well as the… 27 r/LocalLLaMA community 1mo ago Who Has The “Jankiest” Local LLM Setup? | Non-Official | Fun Contest | No Prizes Had an idea for a fun no prize/non official competition to see who has the “Jankiest” local LLM setup. NOTE: This is NOT an official competition. There are NO prizes. This is just for fun . Rules: 1. One Submission via comment per person 2. Has to be your current setup or your… 23 r/LocalLLaMA community 1mo ago I built an opensource AI notepad alternative to Granola, need help to improve it Hey, I built a project called steno. Steno is an AI notepad for confidential conversations. It runs fully locally on your device with local llms like Gemma 4 quantised. The quality has gotten pretty good now so wanted to share to the communities like LocalLLM that helped me… 26 Page 3 of 8 · 373 articles ← Newer Older →
