News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow arXiv — Machine Learning research 1mo ago Training Hybrid Block Diffusion Language Models with Partial Bidirectionality arXiv:2607.02805v1 Announce Type: new Abstract: High-throughput long-context generation is one of the central challenges for large language models. Generation is typically memory-bandwidth-bound rather than compute-bound: each decoding step must stream the accumulated key/value… 32 arXiv — Machine Learning research 1mo ago A Precedent-Guided Co-Scientist for Side-Effect-Aware Drug Redesign arXiv:2607.02944v1 Announce Type: new Abstract: We propose PRECEDE, a precedent-guided co-scientist for side-effect-aware drug redesign that revises a parent compound to mitigate a specified side effect while preserving therapeutic function. Rather than isolated molecular… 6 arXiv — Machine Learning research 1mo ago Decentralised Federated Learning over Temporal Networks: The Role of Heterogeneities arXiv:2607.03171v1 Announce Type: new Abstract: Decentralised federated learning, based on peer-to-peer communication, is increasingly proposed for on-device training of machine learning models, promising a privacy-preserving, communication-efficient training process with no… 30 arXiv — Machine Learning research 1mo ago PhenoNEST: A Neuro-Symbolic Framework for Ontology-Aware Multimodal Plant Phenotyping and Trait Discovery arXiv:2607.03245v1 Announce Type: new Abstract: High-throughput plant phenotyping generates valuable data that often remains trapped in unstructured text and isolated RGB images. To bridge this semantic gap, we propose a framework for constructing a multimodal granular Knowledge… 28 arXiv — Machine Learning research 1mo ago Phase-Preserving Trimodal Transformer for Tropical Forest Biomass Estimation Using Optical and PolInSAR Data arXiv:2607.03663v1 Announce Type: new Abstract: The accurate estimation of Above-Ground Biomass (AGB) in mature tropical forests remains a critical challenge in remote sensing, primarily due to the saturation of Synthetic Aperture Radar (SAR) signals in high-density areas and… 38 arXiv — NLP / Computation & Language research 1mo ago Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs arXiv:2607.04281v1 Announce Type: new Abstract: Semantic caching reduces the latency and cost of retrieval-augmented generation (RAG) by serving cached answers to semantically similar queries, but most existing methods do not model the time-varying freshness of open-web… 17 arXiv — NLP / Computation & Language research 1mo ago LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure arXiv:2607.04733v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities. Standard… 11 r/LocalLLaMA community 1mo ago Qwen 3.6 27B absolutely fails at agentic work I have been running Qwen 3.5 122B at 4 bit for quite a while, and have started running it at 5 bit recently now that Llama.cpp has comparable performance to VLLM. I have also tried, several times, to use Qwen 3.6 27B at 8 bit & 16 bit, as numerous people have claimed that 27B is… 38 r/LocalLLaMA community 1mo ago Got my Ascent GX10 two days ago, ran REAP-pruned NVFP4 DeepSeek-V4-Flash on a single Spark, and it stays consistent at long context Got my Ascent GX10 two days ago and spent the last couple of days pushing a REAP-pruned NVFP4 DeepSeek-V4-Flash setup on a single Spark by patching the eugr/spark-vllm-docker image. Credit where it’s due: the REAPs were done by 0xSero . I’m just the person who wired it up,… 36 r/LocalLLaMA community 1mo ago Is DeepSeek v4 (Flash) really extremely cheap to run? If yes, how? Hi. I don't have a GPU. So my biggest "local LLM" experience has been running ~26B models with single-digits tps values. However, the "serving economy" of DSv4 models look like a riddle to me. The Flash model has 284B parameters, but providers (e.g. OpenRouter) charge so little… 20 Smol AI News news-outlet 1mo ago not much happened today **Tencent** released **Hy3**, a **295B MoE** open-weight model with **21B active parameters**, **192 experts**, and **256K context** supporting **MTP speculative decoding**. It runs natively on **vLLM** with optimizations for **NVIDIA** and **AMD** hardware, achieving up to… 14 r/LocalLLaMA community 1mo ago Qualcomm launches GenieX to run LLMs on their Windows Laptops Qualcomm was behind every major chipmaker so they are playing catchup when it comes to SDKs. https://aihub.qualcomm.com/geniex I was able to get 20 tok/s running Gemma 4 26B A4B 0.5s for first token running on the GPU or NPU 10 tok/s on the GPU for Qwen 3.6 27B MTP To use… 33 r/LocalLLaMA community 1mo ago Qwen 3.6 27B - VLLM Performance Benchmark Results (BF16, FP8, NVFP4) Sharing some testing of Qwen 3.6 27B using VLLM across the popular quants on my development system. I used llama benchy to generate the results, then fed it into an LLM to format it the tables for readibility. While NVFP4 is blazing fast, have had looping issues in copilot that… 14 r/LocalLLaMA community 1mo ago Concurrency plus nvfp4 on Blackwell Parsed from VLLM log file ~2000 tps in aggregate performing bulk captioning on images. Above is parsed from vllm log while a client runs 30 concurrent streams, each concurrent stream has 1 request with an image and prompt, then a 2nd request on the same stream (so 1st Q:A would… 21 r/MachineLearning community 1mo ago We'll benchmark an Open weights LLM on any GPU you choose — drop your model + hardware and we'll run it. [D] We run HexGrid Cloud, a platform for deploying open-source models on GPUs, and we're heads-down optimizing our serving/deployment layer. To pressure-test it we're benchmarking real models under real concurrency — and instead of guessing, we'd rather run what you actually want to… 35 r/LocalLLaMA community 1mo ago Comparing local inference speeds across a few real setups people are running (3090 vs 5090 vs dual 6000) Pulled together token rates from a few different local rigs people have reported running lately, just to get a sense of what's realistic at each hardware tier(source discord group) Qwen3.6 27B on a single 3090 (Q4/Q8 MTP, 128k ctx): ~50 tok/s inference, ~950 tok/s prompt… 29 r/LocalLLaMA community 1mo ago Qwen3.6 27B on a 5090, 6.4k sample tok/s distribution after tuning MTP/cache settings Spent a while tuning llama.cpp for Qwen3.6 27B on a 9800X3D / 64GB / 5090 box and wanted to share the real distribution instead of just a headline number, since averages hide a lot. Ran with q8 KV cache, 192k context, MTP draft=10, spec-draft-p-min=0.5, batch/ubatch 512. Logged… 8 r/LocalLLaMA community 1mo ago Gemma4 with audio input: 16.8 tok/s on Macbook M2 Max 64GB Here's the setup I decided on for embedding gemma4-12b into a Tauri2 desktop app: Native Rust FFI into llama.cpp via llama-cpp-2 (Metal enabled) Model: gemma-4-12b-it-Q5_K_S quantized by Unsloth, Q5_K - Small Audio input is a 607 KB 16-bit mono 16 kHz PCM WAV. Prompt path: Gemma… 38 r/LocalLLaMA community 1mo ago Is dSpark, dflash, MTP, QAT, and similar tech going to increase inference speed enough to where model spillover to disk will be more tolerable? We’re seeing all these performance boosts coming to inference lately with things like dSpark, dllash, MTP, etc. and I know the whole model spillover-to-disk has always been the inflection point where a model would go from maybe a barely acceptable 4 to 5 tokens per second to… 17 r/LocalLLaMA community 1mo ago GLM5.2 performance. I was wondering how fast GLM5.2 (Nvidia’s 460GB nvfp4 checkpoint) is running on your rigs. I have it running at ~1tok/s in the simulation harness. The data extrapolates to 75tok/s on the real Cuda MGPU machine. So I would like to collect data from you how fast it runs for you.… 25 r/LocalLLaMA community 1mo ago Uh.. Honey, how do you feel about takeout? - 2x RTX Pro 6000 Max-Q (96GB) - 8x RTX 3090 (24GB) - 2x RTX 5090 (32GB) - 3 PSUs - 128GB DDR5 SDIMM RAM (4-channel) - Threadripper 9960x - 1x Ryobi Portable Fan - 1x large Uber Eats bill 448GB VRAM Running MiniMax M3 in AWQ-INT4 on VLLM via PP over TP groups of 2. ~30 tp/s per… 7 r/LocalLLaMA community 1mo ago For RAG specifically, prefill speed matters more than decode and why Strix Halo struggles for interactive use Seeing a lot of "what hardware for local RAG" threads lately, and the framing that keeps getting missed is: decode tok/s is not the bottleneck for RAG. Prefill is the bottleneck for RAG. RAG queries stuff thousands of tokens of retrieved context into every prompt. On unified… 13 r/LocalLLaMA community 1mo ago ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving In prefill-decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing decode routers balance only load; for mixture-of-experts (MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step… 16 r/LocalLLaMA community 1mo ago Follow-up: DeepSeek V4 Flash on 2x RTX PRO 6000 finishes real coding tasks faster than Sonnet and Opus, at about Sonnet quality This is a follow-up to post about which local models stay fast deep into long context and I learned a lot from people here. I kept measuring after that and it turned into a proper indie coding bench. With DeepSeek V4 Flash running on vLLM it lands around Sonnet quality and it… 24 r/LocalLLaMA community 1mo ago Follow-up: GLM-5.2 NVFP4 on four DGX Sparks — the MTP mystery is solved, and it's now ~24 tok/s at 128K context Follow-up: GLM-5.2 NVFP4 on four DGX Sparks — the MTP mystery is solved, and it's now ~24 tok/s at 128K context This is a follow-up to my earlier post about running GLM-5.2 NVFP4 on 4x DGX Spark at 128K context. Short version of that post: 128K worked at ~15 tok/s with MTP1, and… 27 r/LocalLLaMA community 1mo ago Pay attention: a few chats waiting in tray reserve 1GB VRAM for themselves. If an application uses a Web-based interface and "hardware acceleration", it constructs its frame in VRAM and sometimes keeps it reserved even if the app is minimised. On my Linux machine, Discord is the worst offender, reserving 450 MB VRAM. Steam takes 200 MB, Telegram 150 MB,… 19 arXiv — NLP / Computation & Language research 1mo ago Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training arXiv:2607.01763v1 Announce Type: cross Abstract: Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as… 34 arXiv — Machine Learning research 1mo ago Privacy-Preserving and Verifiable Approximate Distributed Coded Computing arXiv:2607.02187v1 Announce Type: new Abstract: Distributed machine learning enables collaborative model training without centralizing data, but it also exposes learning processes to privacy leakage and malicious manipulation. Existing defenses typically address these threats in… 13 arXiv — NLP / Computation & Language research 1mo ago Kara: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression arXiv:2607.01237v1 Announce Type: new Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput. To address these issues, KV cache… 21 r/LocalLLaMA community 1mo ago Gemma 4 WebGPU Kernels 255 tok/s by x/@xenovacom We need more of this, 100+ T/s on dense models is the difference between defaulting to Claude/Codex for everything vs having a local private model doing most of the heavy lifting and only reaching for frontier for heavy intelligence work.… 6 r/LocalLLaMA community 1mo ago Software developers appreciation post Im on the bus to work and just felt like i dont see enough grattitude for the men, women, children, and people who contribute thier time and effort on open projects. Just last night i saw ive been sleeping while vllm developers are releasing 3 new major releases, and not only… 31 r/LocalLLaMA community 1mo ago I’m switching to Linux, is Ubuntu the most compatible with local AI? I will definitely use vLLM now (unless there is something faster now) but i want to make sure ggufs + llamacpp works along with comfyui and things of that nature too.   submitted by   /u/XiRw [link]   [comments] 10 Hugging Face Daily Papers research 1mo ago Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation Abstract Vision-language dataset distillation method using rank-aware hyperbolic alignment to optimize synthetic image-text pairs for efficient contrastive model training while preserving modality-specific diversity. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Vision-language… 10 Hugging Face Daily Papers research 1mo ago ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving Abstract ELDR is an expert-locality-aware decode router for prefill-decode disaggregated Mixture-of-Experts serving that improves performance by predicting expert activations and routing requests accordingly. Generated by Qwen/Qwen2.5-Coder-32B-Instruct In prefill-decode (PD)… 30 arXiv — Machine Learning research 1mo ago Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL arXiv:2607.00392v1 Announce Type: new Abstract: Unsupervised Reinforcement Learning (URL) aims to pre-train scalable, skill-conditioned policies without extrinsic rewards, serving as a foundation for downstream control tasks. Despite recent progress, we argue that current… 34 arXiv — Machine Learning research 1mo ago MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression arXiv:2607.00760v1 Announce Type: new Abstract: Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving cost. Because the cache grows linearly with context length, it can exhaust GPU… 9 arXiv — Machine Learning research 1mo ago Staleness-Learning Rate Scaling Laws for Asynchronous RLHF arXiv:2607.01083v1 Announce Type: new Abstract: High-throughput RLHF systems often decouple rollout generation from policy optimization, leading to the use of stale rollouts during learner updates. In this work, we study the effect of such staleness in asynchronous GRPO. We make… 23 arXiv — NLP / Computation & Language research 1mo ago BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal arXiv:2607.00501v1 Announce Type: new Abstract: We present BaseRT, a native Metal inference runtime for large language models (LLMs) on Apple Silicon, and report the highest inference throughput on this hardware to date. Existing runtimes, including llama.cpp and MLX-based… 22 arXiv — NLP / Computation & Language research 1mo ago OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning arXiv:2510.24636v3 Announce Type: replace Abstract: Reward models (RMs) have become essential for aligning large language models (LLMs), serving as scalable proxies for human evaluation in both training and inference. However, existing RMs struggle on knowledge-intensive and… 34 Hugging Face Daily Papers research 1mo ago TurboServe: Serving Streaming Video Generation Efficiently and Economically Abstract TurboServe is a specialized serving system for streaming video generation that addresses session state management and dynamic resource allocation challenges through integrated scheduling, autoscaling, and migration mechanisms. Generated by… 5 r/LocalLLaMA community 1mo ago I added MTP to local SoTA Agentic Coding Model Ornith 35B FP8 E4M3 Just wanted to share that I was looking for an optimal way to run Ornith 35B in FP8 with E4M3 and MTP with vLLM but there was no out-of-the-box model with MTP drafter support. So I grafted this new model! It's 18% faster than without MTP and the drafter acceptance rate is not… 31 r/LocalLLaMA community 1mo ago How to improve RAM offload? I have only 12GB VRAM (RTX3060) but have enough RAM to run Qwen3.6 27B Q4 with offload. Something tells me that it won't achieve maximum performance but why DRAM speed is only around 30GB/s (HWiNFO data) during inference with dual channel 5200 RAM? TG is 3.12 tok/sec with 18K… 38 r/LocalLLaMA community 1mo ago Thinking about grabbing 4x Ascend GX10s Some in this sub have tested GLM5.2 on 4x DGX Sparks (or Ascend GX10) with 400-500 tok/s prompt processing and ~15 tok/s output at 128k context. Not blazing fast, but usable imo, especially with quantization. My thinking: If there's an open-source fable 5 sometime in december or… 20 arXiv — Machine Learning research 1mo ago Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning arXiv:2606.31092v1 Announce Type: new Abstract: Full fine-tuning adapts large language models to new tasks but can erode capabilities they already possess. Existing remedies protect through proxies such as parameter distances, importance penalties, output matching, or dominant… 11 arXiv — Machine Learning research 1mo ago TDGT: A Tabular Data Generation Toolkit supporting adaptive GPU-accelerated Bayesian mixture models, diffusion-based models, and latent-space generative modeling arXiv:2606.31268v1 Announce Type: new Abstract: The growing demand for privacy-preserving data sharing has positioned synthetic data generation as a critical component of responsible AI workflows. Despite notable advances in generative modeling, existing solutions often lack… 29 arXiv — Machine Learning research 1mo ago Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR arXiv:2606.31813v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm. However, their efficacy and behavior under Reinforcement learning with… 24 arXiv — Machine Learning research 1mo ago Criticality-Constrained Iterative Pruning for Energy-Efficient Spiking Neural Networks via Combined Importance Scoring arXiv:2606.30676v1 Announce Type: cross Abstract: Deploying spiking neural networks (SNNs) on neuromorphic hardware demands aggressive synaptic pruning while preserving temporal computation integrity. Existing strategies either neglect neuronal criticality or rely on convex… 5 arXiv — NLP / Computation & Language research 1mo ago UniSAE: Unified Speech Attribute Editing on Speaker, Emotion and Low-Level Content via Discrete Phonetic Posteriorgram Modelling arXiv:2606.31128v1 Announce Type: cross Abstract: Speech editing aims to modify specific portions of an utterance while preserving the remaining speech. Existing approaches primarily focus on word-level content modification and typically treat content, speaker, and emotion… 30 Hugging Face Daily Papers research 1mo ago One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding Abstract InnerZoom addresses GUI grounding challenges by preserving target-region awareness across decoder layers through a single-forward pass that bridges cross-layer evidence, achieving state-of-the-art performance with reduced computational cost. Generated by… 16 r/LocalLLaMA community 1mo ago Devs - you have 64gb of VRAM - which model do you use for coding? I've currently settled on an unsloth version of Qwen 3.5 122b-a10b model (UD-IQ4_NL). With 100k bf16 context window, I only had to load a few layers into CPU/RAM, it runs around 30 tok/sec which is fine for me. I've tested many models, hours of testing but I am currently deeply… 32 Page 6 of 10 · 500 articles ← Newer Older →
