News / #edge Tag Edge 373 articles archived under #edge · RSS Sign in to follow r/LocalLLaMA community 1mo ago Built Kivarro, an all-in-one local inference workbench. Looking for brutal feedback from people who actually run models locally. I’ve been building Kivarro , a desktop app for local model inference. The idea is simple: Most local LLM tools solve one slice of the workflow. One app for chat. One app for model files. One script for llama.cpp flags. One dashboard for memory. One terminal for logs. One random… 36 r/LocalLLaMA community 1mo ago I built Kivarro, an open-source Rust/Tauri local LLM inference workbench. Looking for r/LocalLLaMA testers. Hey r/LocalLLaMA , I’m the creator of Kivarro. It’s an open-source local model inference workbench built with Rust/Tauri, focused on running GGUF models locally with llama.cpp, profile switching, model registry, runtime controls, local API status, logs, benchmarks, and a… 22 r/LocalLLaMA community 1mo ago A planetary test for local models Here is a fun test prompt: Imagine a date in the next 1000 years where the Sun, along with its gravity, suddenly disappeared. When that happens, all planets in our solar system would stop orbiting and carry in a straight line. Is there a date that would then cause two planets… 34 r/LocalLLaMA community 1mo ago I benchmarked 13 models at 65K-128K context to find out what actually matters for agentic workloads I benchmarked 13 models at 65K-128K context to find out what actually matters for agentic workloads — prefill dominates everything, and KV head count beats parameter count I've been running local LLMs for agentic workflows (tool use, coding agents, RAG) and kept seeing people… 36 r/LocalLLaMA community 1mo ago Ran a classic(medival europe) fantasy RP/agentic benchmark across 8 local models Qwen3.6-27B held up better than its size suggests Threw together a benchmark suite (quest completion, scene endings, item/time tracking, character detection, storytelling, drafting) and ran it across 8 models people talk about a lot on here. Judged with an external LLM grader, N varies per category (shown on the chart). Overall… 32 r/LocalLLaMA community 1mo ago Using local models with Hermes vs Claude code Today I saw this in StepFun’s blog for their Step 3.7 Flash model. Running the model with CC performed better results vs Hermes. Curious why?   submitted by   /u/GreatMammad [link]   [comments] 34 r/LocalLLaMA community 1mo ago A fully local, self-hosted repo index for coding agents (Rust, MIT, runs offline) If you run local models for coding, feeding them repo context without blowing the context window is half the battle. I built basemind to index a repo locally and serve it over MCP: a code map across 300+ languages, git history and blame, and document RAG over 90+ formats, all on… 15 r/LocalLLaMA community 1mo ago Qwen3.6-27b-mtp-q8 successfully created an A* pathfinding implementation on a test game built in Java from scratch. I used Claude Code locally running this model. Took a surprisingly long amount of time to pull off. This game is just a test game my friend is making and we're both vibecoding. He uses Codex due to lack of hardware for local models to do the graphics, UI and lighting, while I… 9 r/MachineLearning community 1mo ago Small Language Model SLM [D] Hi, I am supposed to prepare for SLM and its software part for an on campus internship, i've worked with local models like ollama generally,in my projects and also with open claw so can anyone guide me the last 2-3 days tips on what should i go through for this internship prep??… 19 Hugging Face Daily Papers research 1mo ago DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation Abstract DuoMem is a dual-space distillation framework that transfers procedural problem-solving from large language models to compact student models through context-space and parameter-space distillation, achieving high performance with minimal additional parameters and… 31 r/LocalLLaMA community 1mo ago Mapping Local Nodes - Mildlyinteresting I've been working on mapping (with tags) and steering local models based on their activation path in specific context to questioning during a/b testing. There is no insight or "how to" here, no benchmarks or improvement suggestions, no products. I just think that the activation… 23 r/LocalLLaMA community 1mo ago Follow-up: DeepSeek V4 Flash on 2x RTX PRO 6000 finishes real coding tasks faster than Sonnet and Opus, at about Sonnet quality This is a follow-up to post about which local models stay fast deep into long context and I learned a lot from people here. I kept measuring after that and it turned into a proper indie coding bench. With DeepSeek V4 Flash running on vLLM it lands around Sonnet quality and it… 24 r/LocalLLaMA community 1mo ago Claude Code and China: The mechanism is activated when the user sets the ANTHROPIC_BASE_URL environment variable (used for local models) The list of suspicious hostnames is not stored in plaintext within the code; instead, it is Base64-encoded and then encrypted using a simple XOR operation with a key of 91. Once decoded, it reveals domains belonging to Chinese companies, keywords related to artificial… 38 r/LocalLLaMA community 1mo ago I made a local model that gives you a multimedia role playing experience (and you can too!) https://huggingface.co/lemonade-sdk/RPG-HaloTales-V1   submitted by   /u/jfowers_amd [link]   [comments] 17 r/LocalLLaMA community 1mo ago I built a local LLM NPC backend focused on NPC-to-NPC conversations I just released a research project I did last year as open source. It is a fully local speech-to-speech backend for LLM NPCs. So speech-to-text, local LLM, text-to-speech, no cloud needed. The main focus was NPCs talking to each other, not just answering the player, and my study… 17 r/LocalLLaMA community 1mo ago Palantir CEO rages against closed models For context, this week they struck a deal to buy Nvidia chips and run local models for their enterprise clients. So in this video he is railing against Anthropic and OpenAI saying they are ripping everyone off while stealing their data too. Always a special moment when the enemy… 30 r/LocalLLaMA community 1mo ago My reasons to run local models I can finetune any model on any dataset I want. I can use techniques like speculative decoding and other sota approaches to get the max tps The llm provides like anthropic and openai are not getting access to my data The hardware is reusable for vision text speech, and I can run… 10 r/LocalLLaMA community 1mo ago Open benchmark: how well can multimodal LLMs read a calendar week-view from a screenshot? Humans ~99%, Q4 local models..... Some backstory I've been working on my local agent (openclaw), and I wanted to give it the skill to reconstruct calendar entries from a photo of the screen. I couldn't get at the calendar through an API (long story), so a photo was the only low-friction way to export the data.… 16 r/LocalLLaMA community 1mo ago I mapped which local LLMs actually fit each RAM tier, 8 to 128GB (open dataset) I kept answering the same question for friends ("I've got a 16GB MacBook / a 3060, what can I actually run?") and got tired of guessing, so I started a spreadsheet. It grew into a real dataset, so I put it on GitHub under CC BY for anyone to use or fix. Rule of thumb I landed… 28 r/LocalLLaMA community 1mo ago LokalBot - fully local macOS app: meetings, autocomplete, and day tracking that all run on your machine with a user friendly UI Been lurking here a while, this sub is basically why LokalBot exists. It's a Mac app that records + summarizes your meetings, autocompletes your typing in any app, and tracks where your day went, with every model running on-device . No cloud, no account, no API keys. Most of the… 15 r/LocalLLaMA community 1mo ago I built a desktop AI that scrubs your PII locally before it hits the cloud — here's every feature with real screenshots Been building this for a few months. It's called Primnox. The core thing: before ANY message leaves your machine, a local DeBERTa NER model runs on-device, finds names/emails/addresses/phone numbers, swaps them for stable placeholders (FIRSTNAME, EMAIL etc), sends the tokens to… 37 r/LocalLLaMA community 1mo ago Ketch - Best Search Tool for local models recently I wrote a blog post, to find which search tool will be best for the pi coding agent paired with local models (currently I use Qwen3.6 35B) Before that I were using firecrawl or brave-search, but found them very decent, so I went to SearXNG, which is fine, but lacks some… 38 Hugging Face Daily Papers research 1mo ago Little Brains, Big Feats: Exploring Compact Language Models Abstract Small language models can effectively perform retrieval-augmented generation tasks directly on-device without GPU acceleration. Generated by Qwen/Qwen2.5-Coder-32B-Instruct While large language models have been dominating the research landscape recently, small language… 13 r/LocalLLaMA community 1mo ago I benchmarked full tool catalog vs ranked catalog on a local model: 8% → 77% accuracy Been running agents locally for a while and kept hitting the same issue: the more tools I added, the worse the model got at picking the right one.. So I finally benchmarked it properly.. Setup: qwen3.5-class model on an M4 MacBook, 100 tools in the catalog. One run with the full… 23 r/LocalLLaMA community 1mo ago Tesla V100 16GB local LLMs, single and dual NVLink benchmarks Picked up a couple of Tesla V100-SXM2-16GB modules a while back to run local models and drive Claude Code fully offline, figured the actual numbers and the traps might save someone else the pain. They've come right down in price and the 16GB of HBM2 at ~900 GB/s still holds up… 33 arXiv — NLP / Computation & Language research 1mo ago MAM-AI: An On-Device Medical Retrieval-Augmented Generation System for Nurses and Midwives in Zanzibar arXiv:2606.29580v1 Announce Type: new Abstract: Maternal and newborn mortality remain among the highest in sub-Saharan Africa, where midwifery care is often delivered by nurses who lack midwifery training to international standards, and consulting authoritative guidance at the… 7 r/LocalLLaMA community 1mo ago How I'm using local models from real-world coding Just want to share since after many attempts over the past year, I finally have a setup I kinda like and does useful work for me. I only have 32GB of RAM and a 4070 8GB (laptop), just very ordinary hardware. I found that Qwen3.6-35B-A3B runs reliably at about 15 tokens per… 25 r/LocalLLaMA community 1mo ago I Hate Dario Amodei, and everything he stands for. I am so incredibly sick of this guy‘s fear mongering about open source while fundamentally misunderstanding how it actually works. He recently dropped some arguments that are so completely detached from reality, it honestly feels like he’s never even touched a local model in his… 31 r/LocalLLaMA community 1mo ago Anyone else end up building a web access layer for local AI agents? I've been running local models for most of my experiments, and I kept running into the same issue. The model lives locally, but everything it needs to interact with doesn't. Every new agent ended up with another GitHub client, another Reddit integration, another documentation… 10 r/LocalLLaMA community 1mo ago NASA testing local LLM inference for future space missions Red Hat published a blog post last week about an initiative I supported with NASA researchers at Johnson Space Center building a medical AI assistant. It's called the Crew Medical Officer Digital Assistant (CMO-DA) and the system runs LLMs and other models on local hardware with… 34 r/LocalLLaMA community 1mo ago I built an agent Harness for Small Models. I got Qwen 3.5 4b managing servers. This is something I've been working on, I like playing around with smaller local models but found most agent harness's not well suited for them. The failure modes across different model family's tend to be the same: Failed tool calls Poor varication of environment variables Poor… 12 r/LocalLLaMA community 1mo ago NPC Engine Using Local Models I’ve been working on a game-agnostic NPC engine/backend based pretty heavily on SillyTavern-style architecture, and with smaller local models getting better and better, I honestly think this kind of thing could be the future of RPGs. Right now I’m using NVIDIA Parakeet 0.6 for… 22 r/LocalLLaMA community 1mo ago Best case for dual RTX 3090 (250W each) on Crosshair VIII Hero? I'm building a local LLM workstation and would appreciate some advice from people already running 2×3090s. Current hardware: ASUS Crosshair VIII Hero (X570) One Gainward Phoenix RTX 3090 Looking for a second used 3090 (not necessarily the same model) Both GPUs will be… 9 r/LocalLLaMA community 1mo ago I built a tool to turn your Claude Code sessions into fine-tuning data for local models If you use Claude Code, every session is already sitting on disk as a .jsonl file under ~/.claude/projects/ . It has real coding conversations: multi-turn edits, tool calls, reasoning traces. That's training data you already generated for free. The problem is the format is not… 36 r/LocalLLaMA community 1mo ago Mythos was the first, now GPT-5.6 https://techcrunch.com/2026/06/26/openai-limits-gpt-5-6-rollout-after-government-request-says-restrictions-shouldnt-be-the-norm/ Either a hype before IPO, or they have just shot themselves in a foot. This is pretty much it for more advanced online models. Local LLM is one of the… 17 r/LocalLLaMA community 1mo ago What’s the latest on agent browser use? What is the latest and greatest agent browser use framework? I remember trying browser use a few months back and it was ok but would fall apart after long workflows. Has there been improvements to agents controlling browsers and following a predefined workflow? Can local models… 32 r/LocalLLaMA community 1mo ago Dear poor people of this subreddit I see people with multi-gpu setups but I'm sure there's a potato LLM runner out there somewhere. I have an old macbook pro (i5 8th gen, 8GB RAM) that I want to turn into a homelab. I want to run a small local model for experimenting and if possible, agentic tasks (like say… 22 r/LocalLLaMA community 1mo ago Local LLM Peeps I am 80% done with a harness that works for local and API but is local first. The harness has some interesting logic around multiple agents which I’m holding back on until it is open source on GitHub. I have been local for 6 months and built out EVERYTHING I could think of to… 28 r/LocalLLaMA community 1mo ago Streaming medical STT running locally on a MacBook Quick teaser of what I’ve been working on over the last few weeks: a streaming medical speech-to-text model that runs fully on-device. This demo is running locally on a MacBook through MLX. Still doing more evals, but planning to release the open weights next week.  … 22 r/LocalLLaMA community 1mo ago Getting real work out of a 4B local model: the distill-on-idle pipeline behind an on-device "memory" assistant https://preview.redd.it/iiiqwt96tn9h1.png?width=3004&format=png&auto=webp&s=f02fba9f64e27ac91b2ae4cd478842106b294366 https://preview.redd.it/47cb5u96tn9h1.png?width=3024&format=png&auto=webp&s=b1cee93477970b8b0a636c37be657fecd38ba968… 7 r/LocalLLaMA community 1mo ago What's one local AI workflow you wish you'd discovered sooner? There are a lot of posts about the models and benchmarks, but I am more interested in the workflows that people use. What is one workflow that really saved you time or made your local LLM more useful? It could be anything—RAG, MCP, coding agents, organizing prompt, document… 23 r/LocalLLaMA community 1mo ago Help optimizing llama.cpp + Qwen 27B on RTX PRO 6000 Blackwell for coding agents Our company recently acquired a workstation with an RTX PRO 6000 Blackwell , and we're experimenting with local LLMs to reduce part of our Claude token usage. Right now we’re running Qwen3.6 27B MTP Q8_K_XL with llama.cpp on Windows 11 . I've been using both Claude Opus and… 13 arXiv — Machine Learning research 1mo ago Dot-Flik: A Scalable Edge AI Architecture for Distributed Insect Monitoring arXiv:2606.26121v1 Announce Type: cross Abstract: Global insect population declines necessitate scalable, continuous monitoring systems, yet existing vision-based solutions remain constrained by high hardware costs, energy demands, and reliance on centralized processing or cloud… 11 arXiv — NLP / Computation & Language research 1mo ago AnySimLite: A Lightweight Few-Shot Similarity Encoder for On-Device Speech-Adjacent Classification arXiv:2606.26452v1 Announce Type: new Abstract: To minimize privacy concerns and inference latency on edge devices like smartphones, lightweight on-device models remain important for end-user applications. Many of these applications involve natural language classification, but… 31 arXiv — NLP / Computation & Language research 1mo ago Cascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT arXiv:2606.26861v1 Announce Type: new Abstract: Deploying large language models (LLMs) on Industrial Internet of Things (IIoT) edge devices demands extreme compression, yet existing structured pruning methods collapse at high compression ratios due to one-shot importance… 27 r/LocalLLaMA community 1mo ago Good YouTube channels for local LLM news and development? Sometimes I'd prefer chilling on the couch and learning instead of reading. I've searched on YouTube and most seem like clickbait and slop. Thanks   submitted by   /u/6jarjar6 [link]   [comments] 5 r/LocalLLaMA community 1mo ago Built an open source local first Kanban workflow for running AI coding agents without babysitting every step I’ve been building BatonBot, a local first app for running AI coding workflows with less babysitting. The problem I kept running into, especially with local models, is that coding agents can be useful but the workflow gets slow: start task → wait → check output → fix next issue… 10 r/LocalLLaMA community 1mo ago Prices of graphic cards are going crazy, should I buy a second card though? A few months ago, I bought a RX 7900 XTX 24g to start toying with local LLM, at 900€ new. Little I knew that now I want to add a second card to my rig, but prices have gone insane! Adding a new 7900 XTX would cost me 1200€ as new now, used price is around 900€ now, and the last… 38 r/LocalLLaMA community 1mo ago Fast medical RAG API to give your local LLMs access to facts I created a simple RAG API using medical Wikipedia articles that you can point your agent to and use freely. It may be useful in allowing your local LLMs access to medical facts they might not be able to recall from their weights. I'm aiming for subsecond responses but cannot… 7 r/LocalLLaMA community 1mo ago It turns out Bash is All You Need to write a language model REPL (and jq and curl) While working on an self-educational exercise tinkering with local models and trying my hand at setting up agents, I went down a rabbit hole: to see how far I could build a custom agent REPL loop using exclusively command-line building blocks and stripping out dependencies… 20 Page 4 of 8 · 373 articles ← Newer Older →
