News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow r/LocalLLaMA community 12d ago DeepSeek V4 Flash 0731 IQ2_M benchmark for Dual 3060 and 96GB RAM ≈ 3.5 tok/s. Thanks to the community help I finally launched this llm. LM Studio refused to load weight onto second GPU but Unsloth Studio did so everything was done in there. Not a proper benchmark (used PC in parallel as well) but it gives an idea of the performance from dual 3060 with… 16 r/LocalLLaMA community 13d ago Weight-Aware Streaming Tensor Engine: run Kimi K3 using 29 GB of RAM at 0.50 tok/s   submitted by   /u/galapag0 [link]   [comments] 29 r/LocalLLaMA community 13d ago We've gotten some great medium sized models lately (DSV4 Flash 0731, Inkling Small, Laguna S 2.1, Step 3.7 Flash) but does anybody else want to see some new 70-80b contenders? I can run the mediums, but sometimes I want a faster option that's smarter than Qwen 27B/35B. On my hardware I get like 500 to 800 tok/s prefill and 16 to 22 tok/s gen on ~120B class models, which is not the worst but it does get a bit annoying on agentic coding tasks. If we… 8 r/LocalLLaMA community 13d ago DeepSeek-V4-Flash-0731 unsloth gguf on A100 A100 with 40gb VRAM: 162GB Q8_K_XL ~16.1 tok/s generation Only 15.8GB of 40GB VRAM used with all experts on CPU NOTE just tested coding on linux box DeepSeek-V4-Flash-0731 runs losslessly on the single 40GB A100 at 17.7 tok/s with 6 experts loaded into VRAM, with Codex driving… 24 Hacker News — AI on Front Page community 13d ago Run Kimi K3 using 29 GB of RAM at 0.50 tok/s Article URL: https://github.com/sqliteai/waste Comments URL: https://news.ycombinator.com/item?id=49123386 Points: 248 # Comments: 105 21 arXiv — Machine Learning research 14d ago Regularizing modality contribution drift in multimodal continual learning arXiv:2607.27260v1 Announce Type: new Abstract: Multimodal continual learning (MMCL) aims to learn emerging knowledge from multimodal data while preserving knowledge. To mitigate forgetting, current MMCL methods usually focus on cross-modal representation alignment or semantic… 36 arXiv — NLP / Computation & Language research 14d ago DualAnchor: Preserving Language Priors and Improving Lexical Fidelity in Gloss-Free Sign Language Translation arXiv:2607.27614v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have led sign language translation (SLT), the task of converting sign-language videos into spoken-language text, to increasingly adopt LLMs as textual backbones. However, despite… 35 arXiv — NLP / Computation & Language research 14d ago CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising arXiv:2607.28236v1 Announce Type: cross Abstract: Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout.… 19 arXiv — NLP / Computation & Language research 14d ago WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning arXiv:2607.28418v1 Announce Type: cross Abstract: Pruning is a promising approach for improving the efficiency of LLMs. Existing static structured pruning methods are hardware-friendly and can deliver practical throughput gains, but their input-agnostic computation allocation… 27 r/LocalLLaMA community 14d ago Open Source Ternary LLM Engine in Rust/CUDA for Quantization, Serving, and Training of models on consumer GPUs, called Tritium (Apache 2.0) This post was not written by a clanker. Hey guys, I'm a comp sci major who wanted to introduce a cool project I built for quantizing models to ternary (1.58 bit) with as minimal of loss as possible, a process that can provide even more than 10x reductions in VRAM usage and much… 25 r/LocalLLaMA community 14d ago Would extremely high decode tok/s even be useful? If you were able to get an inference machine that could do decode at 1k toks/s or even 10k tok/s, would that even be helpful? Would it unlock any new use cases? Let’s assume that this is for actually useful models and fairly large models like Qwen 3.5 397B, GLM-5.2, etc Or at… 27 NVIDIA Developer Blog official-blog 14d ago NVIDIA Exemplar Cloud: Lessons for Unlocking Full Performance on AI Infrastructure Two AI computing clusters built from identical NVIDIA H100, GB200 NVL72, or GB300 NVL72 systems can deliver materially different training throughput. We... 9 r/LocalLLaMA community 14d ago Turbo-fieldfare: Open-source engine running Gemma 4 26B in 2 GB RAM on Apple Silicon Its a custom Swift/Metal inference engine that runs Gemma 4 26B-A4B-IT on M-series Macs with very low RAM. It uses ~2GB instead of ~14 GB. The result is reportedly 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro. It also includes an OpenAI-compatible… 11 arXiv — Machine Learning research 15d ago From Conceptual Hydrologic Models to Conceptually Interpretable Neural Networks: A Snow-Water Mass-Conserving-Perceptron Framework for Discovering Catchment-Scale Precipitation-Storage-Runoff Representations arXiv:2607.26492v1 Announce Type: new Abstract: The Mass-Conserving Perceptron (MCP) establishes a modeling paradigm in which conceptual hydrologic models can be reformulated as physically constrained, conceptually interpretable neural networks. Here, we develop a snow-water MCP… 35 arXiv — Machine Learning research 15d ago Lilith: Backdoor Generalization under Training-Inference Trigger Shift arXiv:2607.26099v1 Announce Type: cross Abstract: Machine-learning services increasingly rely on public data, third-party providers, and outsourced training, creating opportunities for data-poisoning attacks that implant persistent malicious behavior while preserving benign… 38 arXiv — Machine Learning research 15d ago Lightweight Image Classification of Raptor Species for Edge Devices: Rare-Species Dataset Expansion via Video Frame Extraction, Knowledge Distillation, and TensorRT Deployment arXiv:2607.26238v1 Announce Type: cross Abstract: We investigate lightweight raptor-species classification for real-time edge deployment in wind-turbine collision mitigation. Using DINOv2-L (304M parameters) as a teacher, we distilled three lightweight students (MobileNetV4,… 12 arXiv — NLP / Computation & Language research 15d ago AgentGUI: An Interface for Observing and Steering Long-Running AI Agents arXiv:2607.26300v1 Announce Type: new Abstract: AI agents are increasingly adept at tackling complex, long-running tasks. With the rapid surge of autonomous capabilities, human oversight is systematically lagging behind due to limited human-centered interfacing. Aiming to… 19 r/LocalLLaMA community 15d ago CPU prompt processing leaves CPU and RAM throughput on the table. I've been doing some experimentation with CPU-only processing on mainline llama.cpp and I'm confused about the difference in performance between inference and prefill. To be clear, I'm not complaining that either are slow (they're expected to be), nor am I complaining that they… 37 r/LocalLLaMA community 15d ago 5060ti Chads, vllm updates and nvfp4 Hey y'all! How is it going. Today this will be a short posting for posterity, mostly so the future llm/scraping overlords catch it since they like reddit and also for anyone out there trying this shit. I have also seen a lot of others post about not getting good results out of… 19 r/LocalLLaMA community 15d ago The idea: on a CPU the decode speed depends on the active params per token, not the total. My objective is trying to run a 10B at 100tok/s on a mid level PC (No GPU). On the CPU, batch 1 is memory bandwidth bound. But if token/s = bandwidth / (bytes_per_weight * active_weights_per_token) the total number of parameters doesnt slow down the generation speed. So building the architecture aroud a small batch "active parameters per token" (ternary… 17 Hugging Face Daily Papers research 15d ago CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents Abstract Coding agents repeatedly search, navigate, and retain context from evolving repositories, but disconnected indexes, language servers, and task-local histories force repeated discovery and obscure lifecycle costs. CodeNib builds reusable lexical, dense, and structural… 37 arXiv — NLP / Computation & Language research 16d ago SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies arXiv:2607.25716v1 Announce Type: new Abstract: Federated learning (FL) enables privacy-preserving training of automatic speech recognition (ASR) systems across distributed data sources, yet its application to large-scale speech language models (SpeechLLMs) remains unexplored.… 18 arXiv — NLP / Computation & Language research 16d ago GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference arXiv:2607.24788v1 Announce Type: cross Abstract: As Large Language Models scale to increasingly long contexts, the memory I/O and computational overhead of the Key-Value (KV) cache during decoding emerges as the primary throughput bottleneck. To address this, we propose GLIDE,… 36 r/LocalLLaMA community 16d ago I got Kimi-k3 running..... Results: prompt eval: 40 tokens / 97.5s → 0.41 tok/s eval: 400 tokens / 1769.9s → 0.23 tok/s total: 440 tokens / 1867s (31 min) Prompt: "Write a C++ function that reverses a linked list in place. Explain the pointer manipulation." How I ran it: Using PR#26185 from llama.cpp… 32 r/LocalLLaMA community 16d ago DeepSeek V4 Flash, up to 32 tok/s on AMD Ryzen AI MAX+ 395 Hey fellow llamas. we have something new for Strix Halo owners we thought would be useful to share. i'll keep it short: We were able to fit DeepSeek V4 Flash plus its speculative draft on a single Ryzen AI MAX+ 395 with 128 GB of unified memory, and got it to a usable decode… 34 Hugging Face Daily Papers research 16d ago TRACE: Business Rule-Grounded Reasoning Curriculum for Knowledge-Preserving Parametric Tool Retrieval in Enterprise LLMs Abstract Parametric retrieval enables LLMs to retrieve tools implicitly by assigning each API a unique virtual token and training the model to generate it via constrained beam search. Toolsense shows that this regime has two critical drawbacks: it destroys parametric tool… 32 arXiv — Machine Learning research 17d ago XGRVFL-MV: Residual-Coupled Graph-Embedded Multi-View Random Vector Functional Link Network with FleXi Guardian Loss arXiv:2607.23149v1 Announce Type: new Abstract: Random Vector Functional Link (RVFL) networks provide an efficient randomized learning framework for classification. Existing multi-view RVFL methods utilize complementary information from multiple views. However, preserving… 38 arXiv — Machine Learning research 17d ago Variance-Preserving Orthogonal Selection (VPOS): Greedy Feature Selection via Orthogonal Deflation in PCA Loading Space arXiv:2607.23198v1 Announce Type: new Abstract: We propose Variance-Preserving Orthogonal Selection (VPOS), a greedy framework for unsupervised feature selection that operates in the weighted PCA loading space. After each selection, VPOS projects out the chosen feature's… 20 arXiv — NLP / Computation & Language research 17d ago LoRA for Gender-Inclusive Rewriting and Activation Steering for Counter-Narrative Generation arXiv:2607.23083v1 Announce Type: new Abstract: Gender-inclusive language generation seeks to transform biased text into inclusive alternatives while preserving semantic meaning and contextual coherence. This paper presents the IHLC system for the LT-EDI 2026 Shared Task,… 5 arXiv — NLP / Computation & Language research 17d ago Masked Distillation: Internalizing the Chain-of-Thought in Language Models arXiv:2607.22629v1 Announce Type: cross Abstract: Large Reasoning Models (LRMs) produce long, explicit chains of intermediate steps before generating a final answer at inference time. These intermediate traces dominate latency, memory usage, and serving cost, even though the… 32 Hugging Face Daily Papers research 17d ago dRAE: Representation Autoencoder with Hyper-Spherical Codes Abstract In this work, we aim to discretize the high-dimensional visual representations to bridge the gap with language models - a non-trivial challenge, as existing quantization methods suffer from codebook collapse, failing to scale while preserving semantic coherence. We… 23 r/LocalLLaMA community 17d ago Ling-3.0-flash weights: SGLang says day-0, vLLM says when they land, llama.cpp closed the 2.6 request as not_planned Some Ling-3.0-flash threads here last week ended on the same two questions with no real answer, so I went through the repos. State as of writing, with links so you can check instead of taking my word for it. SGLang publicly committed to day-0 support. Their post on the 23rd says… 17 arXiv — Machine Learning research 18d ago Measuring the Dependency Gap: Diagnosing Inter-Column Fidelity in Tabular Generative Models arXiv:2607.21636v1 Announce Type: new Abstract: Synthetic tabular data is valued for preserving not only each column's marginal distribution but the dependencies between columns -- structure that carries much of the discriminative signal for minority classes in imbalanced… 8 arXiv — Machine Learning research 18d ago MA-DAR: Manifold-Aligned Dynamic Adaptive Routing for Continual Temporal Knowledge Graph Reasoning arXiv:2607.21949v1 Announce Type: new Abstract: Continual temporal knowledge graph (TKG) reasoning aims to continuously incorporate newly emerging facts while preserving previously acquired knowledge. Replay-based continual learning has achieved promising performance by… 36 arXiv — Machine Learning research 18d ago FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs arXiv:2607.21624v1 Announce Type: cross Abstract: Transformer-based models have enabled unprecedented capabilities across language, vision, and multimodal tasks. On-device fine-tuning of transformer models offers a privacy-preserving path to personalized AI, yet remains… 20 arXiv — NLP / Computation & Language research 18d ago Diffusion Models in Medical Image Inpainting: Challenges, Solution Taxonomy, and Future Directions arXiv:2607.21904v1 Announce Type: cross Abstract: Image inpainting aims to reconstruct missing or corrupted regions of an image while preserving as much as possible, visual and semantic consistency. In medical imaging, this task is particularly important because artifacts,… 9 Vercel — AI dev-tools 18d ago Kimi K3 and Kimi K3 Fast with ZDR and US-based providers now on AI Gateway Kimi K3 from Moonshot AI and its faster serving path, Kimi K3 Fast , are now available from US-based providers on AI Gateway, including Baseten and Fireworks. Zero Data Retention (ZDR) is also supported for both models. Running Kimi K3 on US-based providers lets teams with data… 31 Hugging Face Daily Papers research 19d ago VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression Abstract Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference latency and memory overhead. This has motivated extensive research on visual token compression. While training-free strategies rely on heuristic metrics and suffer… 12 r/LocalLLaMA community 19d ago GLM 5.2 and ik_llama.ccp Running GLM-5.2 (the new glm-dsa arch), Unsloth UD-Q4_K_XL, on a 4-socket Xeon E7-8880 v4 box with 1TB RAM and a single RTX 3060 12GB. ik_llama.cpp, experts on CPU (--cpu-moe), 24 attention layers on the GPU. Works great at 8k context — rock solid, ~3.7 tok/s gen. Problem: the… 5 r/LocalLLaMA community 19d ago 4x 3090, 96gb vram what Model to drive Hermes? 3 year lurker, now i finally got my server up and running. dont know which model to choose. llama.cpp or vllm, what makes more sense? mainly single user with maybe 2-3 more additional users in family, if everything checks out. hermes is gonna be used as "ai playground" to… 30 r/LocalLLaMA community 19d ago LFM 2.5 230M running at 1440 tok/s in-browser through a custom backend Everything runs through WebGPU, in-browser or in electron/tauri apps. It's fully portable and supports either Nvidia and Apple Silicon (Metal). The actual kernels are optimized for the specific hardware of the device. The Nvidia kernels are aggressively fused into a multi-pass… 20 r/LocalLLaMA community 19d ago My GX10 died Everything ran fine, I was using UD 3.6 Q6 for 35 and 27B, each 4 concurrent requests at 200K context. I had Dify and Mastra to play around with, Unsloth studio to get around to and vLLM ready for whenever I decided to do some more testing. LLama-swap above lama.cpp and liteLLM… 30 Hacker News — AI on Front Page community 20d ago Watching Go's new garbage collector move through the heap Article URL: https://theconsensus.dev/p/2026/07/19/observing-gos-garbage-collector-old-and-new.html Comments URL: https://news.ycombinator.com/item?id=49045474 Points: 202 # Comments: 21 17 r/LocalLLaMA community 20d ago OrangePi AI Studio Pro - Qwen3.5-122B-A10B https://preview.redd.it/wbq8ullnbafh1.png?width=1409&format=png&auto=webp&s=e6d2fe2b1c87c724bc64003c25f917dcee53260f I finally got round to tweaking this, with a bit of help from GLM5.2. The trick to getting it running with vLLM (which I couldn't get anything really out of… 24 r/LocalLLaMA community 20d ago Agentic Kernel Optimization, visualized. A swarm of GPT 5.6 Sol agents spent over 40 hours optimizing a Kimi K3-like model from 65 to 406 tok/s. This animation follows their collaboration as they discover operator fusions, transform the execution graph, and develop new kernel algorithms. We start with a fully… 12 r/LocalLLaMA community 20d ago AMD Kernel Optimizations in llama.cpp Is it a thing? i use kernel-anvil added to llama.cpp, but wondering if theres others out there ? Heres a link i found for vLLM - GEAK v4   submitted by   /u/uber-linny [link]   [comments] 12 r/LocalLLaMA community 21d ago Extened garlic to run Qwen3.5 35B A3B float8 at 55 tok/s on RTX 5060 Ti In a previous post ( https://www.reddit.com/r/LocalLLaMA/comments/1utefpr/running\_qwen3\_30b\_a3b\_at\_50\_toks\_on\_rtx\_5060\_ti/ ) there seemed to be great demand for bringing in Qwen3.5 35B. Some Gated Delta Network kernels later and here it is. It runs at 55 tok/s (61 when… 15 arXiv — Machine Learning research 21d ago Information-Theoretically Secure Aggregation for Lightweight Federated Learning: Resilient to Dropouts and Adversaries arXiv:2607.20890v1 Announce Type: new Abstract: On-device federated learning (FL) enables privacy-preserving and personalized model training on resource-constrained devices such as smartphones and IoT nodes. To reduce communication cost, sign-based methods (e.g., signSGD)… 20 arXiv — Machine Learning research 21d ago Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearning arXiv:2607.21353v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of specific training data while preserving model utility. Many state-of-the-art approaches pursue this goal by restricting the forgetting update to a subset of parameters selected… 38 arXiv — NLP / Computation & Language research 21d ago Refusal-Gated Decoding: Preserving Refusal Behavior Under High-Temperature Sampling arXiv:2607.20791v1 Announce Type: cross Abstract: High-temperature sampling is one of the primary mechanisms for increasing diversity in LLMs. Recent advances in truncation-based sampling techniques have helped mitigate drawbacks of high-temperature sampling such as neural text… 35 Page 3 of 10 · 500 articles ← Newer Older →
