News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow arXiv — Machine Learning research 1mo ago Distance-Preserving Embeddings in Inhomogeneous Random Graphs arXiv:2607.10074v1 Announce Type: new Abstract: Graph machine learning provides powerful tools for understanding complex networks and learning meaningful node representations. A central challenge, however, is designing embeddings with minimal distortion of both local and global… 10 arXiv — NLP / Computation & Language research 1mo ago Large Language Models for Token-Efficient and Semantic-Preserving Opinion Summarization arXiv:2607.10825v1 Announce Type: new Abstract: Opinionated text - spanning product reviews, hotel feedback, and social posts - captures rich signals about user experiences, preferences, and concerns. However, the scale, redundancy, and imbalance of such corpora make it… 21 r/LocalLLaMA community 1mo ago Joined the Dual RTX 6000 club I only spent 2 hours making the bios accept the dual gpus, only 5 hours configuring VLLM to run deepseek v4 flash dspark, but totally worth it. I truly believe in the near future we will have to rely on ourselves.   submitted by   /u/BitXorBit [link]   [comments] 20 r/LocalLLaMA community 1mo ago Production Qwen 3.6-27B VLLM config? Hi everyone, I've spent about four days now trying to find the eight configuration for running Qwen 27b in production using VLLM but have been getting significant decreases in performance with the FP8 safetensors in comparison to the llama.cpp variants, especially under… 28 arXiv — Machine Learning research 1mo ago Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement arXiv:2607.08782v1 Announce Type: new Abstract: Expert parallelism has become the prevailing paradigm to serve Mixture-of-Experts (MoE) models. Its efficiency depends on the communication and computation latencies of the GPUs, which are linked to the placement of experts in the… 21 arXiv — Machine Learning research 1mo ago BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving arXiv:2607.08930v1 Announce Type: new Abstract: Efficient serving of diffusion large language models (dLLMs) is hindered by convergence heterogeneity: when batching multiple requests, different sequences converge at different rates, causing faster requests to stall behind slower… 17 arXiv — NLP / Computation & Language research 1mo ago Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models arXiv:2607.08991v1 Announce Type: cross Abstract: Efficient inference in Large Language Models (LLMs) requires deciding where computation can be reduced while preserving model quality. We study this problem through multilayer perceptron (MLP) activation sparsification and… 6 arXiv — Machine Learning research 1mo ago COAST: Context-Aware Differential Learning for Gene Expression Prediction in Spatial Transcriptomics arXiv:2607.09166v1 Announce Type: new Abstract: Spatial transcriptomics enables profiling of spatial gene expression but is limited by high cost and low throughput, motivating prediction from H&E histopathology images. Existing context-aware methods mainly supervise absolute… 17 arXiv — Machine Learning research 1mo ago Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem arXiv:2607.09236v1 Announce Type: new Abstract: Machine unlearning in LLMs is the targeted removal of specific knowledge while preserving all other capabilities, critical for privacy and safety. Yet existing benchmarks measure it unreliably. They miss knowledge that resurfaces… 27 arXiv — NLP / Computation & Language research 1mo ago AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving arXiv:2512.04013v3 Announce Type: replace Abstract: As augmented large language models (LLMs) with external tools become increasingly popular in web applications, improving augmented LLM inference serving efficiency and optimizing service-level objectives (SLOs) are critical for… 36 Simon Willison community 1mo ago shot-scraper 1.11 Release: shot-scraper 1.11 Some minor improvements, mainly around command option consistency and making the server: mechanism used by both shot-scraper video and shot-scraper multi work if the server takes longer than a second to start serving traffic. server: processes used by… 25 r/LocalLLaMA community 1mo ago Has anyone gotten Llama.cpp (or other) working using Intel iGPU (arrowlake) where it actually improves anything? I Recently did a bunch of tests and wrote them all up on here, but the short version is that Vulkan basically doesn't work (or when it does, it's at 1tok/s at best). SYCL works pretty well, seems to run the Qwen3.6 35b models at around 12tok/s. The prefill part is around 20tok/s… 17 r/LocalLLaMA community 1mo ago Benchmark - 4x 5060 Ti (64GB VRAM) (P2P) - Qwen3.6 27B (INT8 /w bf16 kv cache) @ 8 concurrency with SGLang. SGLang seems to handle higher concurrency better with this setup I recently posted some posts with VLLM showing issues with TTFT and concurrency with 4x 5060 ti's. Wanted to share this benchmark to provide what worked for me so other people that are planning to go the 4x 5060 ti route aren't discouraged. Benchmark Results ============ Serving… 38 Hacker News — AI on Front Page community 1mo ago We scaled PgBouncer to 4x throughput Article URL: https://clickhouse.com/blog/pgbouncer-clickhouse-managed-postgres Comments URL: https://news.ycombinator.com/item?id=48872874 Points: 201 # Comments: 43 6 r/LocalLLaMA community 1mo ago Running Qwen3 30B A3B at 50 tok/s on RTX 5060 Ti Experimented with some custom CUDA and C++ code that can now run a Qwen3-30B-A3B at 50-54 tok/s at float 8 on an RTX 5060 Ti with only 16 GB of VRAM. This speed is roughly 50% improvement to llama.cpp which runs at around 33-34 tok/s (with n-cpu-moe). These speedups come mostly… 17 r/LocalLLaMA community 1mo ago Hy3 (295B MoE) and NVIDIA Nemotron-Labs-Audex-30B-A3B (audio-capable 30B MoE) GGUF quants Sharing two GGUF quant sets, both with the same treatment: imatrix quantization, KLD/PPL measured against BF16 reference logits, llama-bench throughput numbers, and all raw benchmark data included in the repos. No vibes-based "quality tested" claims — every number is… 16 r/LocalLLaMA community 1mo ago MIT LLM Serve Dashboard I am making open source A single-file, dependency-free live dashboard for your local LLM serving box — GPU utilization, per-model throughput, KV/context fill, and system stats for llama.cpp and vLLM , in one green terminal-styled page. No framework, no build step, no external requests. The frontend is… 31 r/LocalLLaMA community 1mo ago Looking for Dual GPU Tips and tricks. Just added a second 5060 16gb to my server for 32gb total VRAM + 80GB of ECC DDR4 It's pcie 3.0 so I think tensor parallel is not going to run well in any config but I get around 3200 tok/s prompt processing and 100 tok/s generation with qwen 3.6 35B. 27b runs at about 600 PP /… 9 NVIDIA Developer Blog official-blog 1mo ago AI Model Co-Design: Hardware-Friendly LLM Design AI performance comes down to three dimensions: Accuracy: How well the model reasons and produces outputs Throughput: How many tokens per second a... 32 arXiv — NLP / Computation & Language research 1mo ago Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization arXiv:2607.08057v1 Announce Type: cross Abstract: Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling… 36 arXiv — Machine Learning research 1mo ago Workload-Preserving Differentially Private Synthetic Data for Causal Inference via Maximum-Entropy Calibration arXiv:2607.08122v1 Announce Type: new Abstract: Workload-based differentially private (DP) synthetic data methods privately measure aggregate queries and post-process the noisy answers into synthetic records. Generic workloads can achieve strong distributional fidelity, but… 20 arXiv — Machine Learning research 1mo ago Predicting Male Fertility Using Machine Learning: A Semen Parameters Based Analysis with the VISEM Dataset arXiv:2607.08429v1 Announce Type: new Abstract: Male infertility is a significant yet often underdiagnosed aspect of reproductive health, with semen analysis serving as the cornerstone of clinical evaluation. To address this problem, this study investigates the use of machine… 8 arXiv — Machine Learning research 1mo ago Federated Deep Learning for Privacy-Preserving Cardiovascular Disease Risk Prediction arXiv:2607.08595v1 Announce Type: new Abstract: Cardiovascular disease risk prediction models often rely on data from a single institution or centrally pooled datasets. Extending these models across institutions could be limited by privacy regulations and constraints on sharing… 16 arXiv — Machine Learning research 1mo ago Resample or Reroute? Budget-Aware Test-Time Model Selection for Large Language Models arXiv:2607.08665v1 Announce Type: new Abstract: Routing among large language models (LLMs) trades response quality against serving cost, motivated by the reported gap between deployed routers and a per-instance oracle. Recent analysis shows that test-time resampling can recover… 21 arXiv — NLP / Computation & Language research 1mo ago Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention arXiv:2607.08027v1 Announce Type: new Abstract: This paper proposes an improved structured pruning method for large language models (LLMs) that addresses key challenges in adapting Adaptive Feature Retention (AFR), an unstructured pruning technique, to structured pruning. When… 28 arXiv — NLP / Computation & Language research 1mo ago Fair Document Valuation in LLM Summaries via Shapley Values arXiv:2505.23842v5 Announce Type: replace Abstract: Large Language Models (LLMs) increasingly power search engines and AI assistants that retrieve and summarize content from many sources. By serving answers directly, these systems obscure the original content creators'… 10 arXiv — NLP / Computation & Language research 1mo ago UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speech arXiv:2508.09767v3 Announce Type: replace-cross Abstract: We propose UtterTune, a lightweight method for adapting a multilingual text-to-speech (TTS) system built on a large language model (LLM). It improves control of pronunciation in the target language while preserving… 25 r/LocalLLaMA community 1mo ago Deepseek V4 Flash on a single RTX 6000 Pro - vLLM-Moet Wow... https://github.com/kacper-daftcode/vLLM-Moet Using this customized vllm provided as a docker, I'm able to run DS V4 Flash on a single RTX 6000 Pro (apparently it also works on a single 5090 - check his readme, but I haven't tried). Apparently this also works with GLM 5.2… 26 r/LocalLLaMA community 1mo ago Exploring FlashAttention-3/4 optimizations on RTX GPUs I was curious whether any of the FA-3/4 optimizations transfer to RTX GPUs. vLLM/SGLang attention falls back to FA-2 on consumer cards (FA-3 and FA-4 are datacenter-only), so I wanted to know if there's any performance left on the table, and I rebuilt the attention kernels from… 21 r/LocalLLaMA community 1mo ago NVIDIA Puzzle-75B-A9B NVFP4 at 132 t/s on 3×3090 — Why is this size category a desert otherwise? TLDR: 75B-total / 9B-active MoE is the perfect shape for multi-24GB rigs, and almost nobody ships it. Qwen 27B is a great model and punches way above its weight-class, it is a frequent fallback for me. Nemotron-3-Puzzle-75B-A9B, NVFP4, vLLM 0.22.1 (the new Marlin fallbacks run… 18 arXiv — Machine Learning research 1mo ago Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata arXiv:2607.07026v1 Announce Type: new Abstract: Constrained decoding is essential for serving LLMs, ensuring that generated outputs follow specific structures such as JSON schema-formatted function calls. Existing systems are designed for autoregressive models and assume… 7 arXiv — Machine Learning research 1mo ago Where to Intervene? Benchmarking Fairness-Aware Learning on Differentially Private Synthetic Tabular Data arXiv:2607.07471v1 Announce Type: new Abstract: Machine learning models are increasingly deployed in high-stakes domains, raising concerns about both privacy and fairness. Differential Privacy (DP) has become a gold standard for privacy-preserving data analysis, while… 19 arXiv — NLP / Computation & Language research 1mo ago $C$-$\Delta\Theta$: Circuit-Restricted Weight Arithmetic for Selective Refusal arXiv:2602.04521v2 Announce Type: replace Abstract: Modern deployments require LLMs to enforce safety policies at scale, yet many controls rely on inference-time interventions that add recurring compute cost and serving complexity. Activation steering is widely used, but it… 38 r/LocalLLaMA community 1mo ago 4 GPUs (MI50) llama.cpp or vLLM? Hi, I've been running vLLM on my MI50 because of tensor-parallel support. It works, but I have some complaints. For one, the quants seem much harder to find than GGUFs. Also, model switching is a pain, especially with the insanely long startup time. I recently discovered that… 28 Hugging Face Daily Papers research 1mo ago JD Oxygen AI Item Center (Oxygen AIIC) V1: An Industrial-Scale LLM/VLM-Centric Solution for Item Understanding, Management, and Applications Abstract A large-scale industrial platform leveraging LLMs and VLMs addresses key challenges in structured item knowledge production for e-commerce, achieving high precision and throughput across billions of products. Generated by Qwen/Qwen2.5-Coder-32B-Instruct JD.com, one of… 17 r/LocalLLaMA community 1mo ago How big of a model do you guys think google overview model is? It's perhaps the model AI model in the world right know, and I just wanted to understand, if you are serving a billion plus requests a day, what size of model could be actually used in the real world? Edit: I don't expert people from Google to say what it is, but if you could… 30 r/LocalLLaMA community 1mo ago The standard free ChatGPT LLM you get after a few messages HAS to be some sub-20b model with online search enabled, no other way to explain how awful it is Recently bought into the local LLM hype by buying a 32gb vram gpu and holy shit, gemma 4 31b at 5bits blows the standard ChatGPT model out of the fucking water. I just can't unsee the quality difference now that I've experienced it. Does Openai just cut costs by serving their… 34 Hugging Face Daily Papers research 1mo ago DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation Abstract DSpark enhances LLM inference speed by combining parallel draft generation with adaptive verification that reduces waste and improves throughput in high-concurrency settings. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Speculative decoding accelerates Large Language… 32 Hugging Face Daily Papers research 1mo ago Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding Abstract Nemotron-Labs-Diffusion is a tri-mode language model that combines autoregressive, diffusion, and self-speculation decoding to achieve superior throughput and efficiency compared to existing models. Generated by Qwen/Qwen2.5-Coder-32B-Instruct We introduce… 23 arXiv — Machine Learning research 1mo ago Intuitionistic Fuzzy Graph Embedded Random Vector Functional Link with Multiview Learning arXiv:2607.05635v1 Announce Type: new Abstract: Random Vector Functional Link (RVFL) networks are popular due to their fast training and universal approximation capabilities. However, RVFL models face challenges in preserving geometric relationships and utilizing multiple… 12 arXiv — Machine Learning research 1mo ago Deep Reinforcement Learning for Dynamic Battery Management of Autonomous Order Pickers arXiv:2607.05683v1 Announce Type: new Abstract: Battery charging of Autonomous Mobile Robots (AMRs) in warehouses is a critical operational challenge that heavily impacts both order processing times and throughput. In this study, we address the dynamic AMR charging problem under… 21 arXiv — Machine Learning research 1mo ago Association Restoration Test: Revealing Restorable Shortcuts after Unlearning arXiv:2607.05726v1 Announce Type: cross Abstract: Association unlearning aims to disable learned label-attribute shortcuts while preserving task performance. Existing evaluations mainly measure output-level robustness or probe whether shortcut attributes remain readable in… 30 arXiv — NLP / Computation & Language research 1mo ago Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving arXiv:2607.05399v1 Announce Type: new Abstract: Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks,… 30 arXiv — NLP / Computation & Language research 1mo ago StateFuse: Deterministic Conflict-Preserving Memory for Multi-Agent Systems arXiv:2607.05844v1 Announce Type: cross Abstract: Agent systems accumulate conflicting observations across branches, retries, and replicas, yet many practical memory layers still collapse disagreement behind overwrite rules that are difficult to inspect or correct. We present… 7 arXiv — NLP / Computation & Language research 1mo ago CMDR: Contextual Multimodal Document Retrieval arXiv:2607.05927v1 Announce Type: cross Abstract: Multimodal document retrieval aims to retrieve relevant pages while preserving both textual and visual content from the original document. However, existing benchmarks primarily evaluate simple lexical or semantic matching, and… 9 Hugging Face official-blog 1mo ago Native-speed vLLM transformers modeling backend Back to Articles a]:hidden"> Native-speed vLLM transformers modeling backend Published July 8, 2026 Update on GitHub Upvote 2 Harry Mellor hmellor Lysandre lysandre TL;DR : The transformers vLLM backend is now as fast (or faster) than custom vLLM implementations for many LLM… 15 r/LocalLLaMA community 1mo ago Gepard : 0.6B streaming TTS built for real-time dialogue - 20× realtime factor, ~50ms time-to-first-audio, vLLM-native, Apache 2.0 We just open-sourced Gepard 1.0 , a TTS model built for real-time conversation. It’s streaming-first: audio starts the moment text arrives, generated frame by frame instead of waiting for a full sentence. - ~555M params : Qwen3.5 0.8B backbone (14 layers) + Nemo NanoCodec (FSQ,… 36 NVIDIA Developer Blog official-blog 1mo ago NVIDIA Vera CPU Boosts AI Factory Throughput to Accelerate Agentic Workloads Agentic systems turn model reasoning into action through multi-step workflows that combine inference, tool use, code execution, retrieval, orchestration, and... 8 Hugging Face Daily Papers research 1mo ago Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval Abstract Object-aware token merging framework SaMer compresses image-side tokens while preserving query-selectable visual evidence, achieving significant storage reduction and improved retrieval performance. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Multi-vector… 19 Hugging Face Daily Papers research 1mo ago GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving Abstract GORGO is a proxy architecture that optimizes LLM inference load balancing by jointly considering network latency, prefill cost, and queueing delay through evolutionary strategy tuning on a new synthetic dataset. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Increasingly,… 17 Page 5 of 10 · 500 articles ← Newer Older →
