News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow r/LocalLLaMA community 21d ago Deepseek V4 Flash ~105 t/s on two Nvidia 4090d 48G (ada) in vLLM TLDR: I (with the help of AI) re-implemented every Blackwell-only kernel (DeepGEMM, FlashInfer sparse-MLA, block-scaled FP8) in Triton, because they simply don't exist for sm89. The performance is 2-3x more for parallel agentic workflows. Benchmark llama-server vs vLLM I was… 27 arXiv — Machine Learning research 22d ago Predictive single cell foundation model for gene regulation and aging with privacy-preserving tabular learning arXiv:2607.19400v1 Announce Type: new Abstract: Pre-trained foundation models (FMs) have begun transforming single-cell genomics, but scaling them raises privacy concerns. Moreover, unlike text data, single-cell data is unordered and exhibits a unique tabular structure that… 17 arXiv — Machine Learning research 22d ago Reproducing Recurrent Transformers: The CoTFormer arXiv:2607.19405v1 Announce Type: new Abstract: The CoTFormer architecture formalizes Chain-of-Thought as a form of recurrent latent computation, preserving intermediate states as attendable representations to mimic explicit reasoning traces. In this work, we evaluate CoTFormer… 32 arXiv — Machine Learning research 22d ago Total Variation Distance Estimation in Autoregressive Models arXiv:2607.19510v1 Announce Type: new Abstract: Modern LLM deployments use a number of implementation choices and inference optimizations (e.g., batching, custom kernels, and quantization) on top of fixed weights, so two engines serving "the same model" can produce meaningfully… 7 arXiv — Machine Learning research 22d ago SynPre-FL: Synthetic data-driven pretraining integrated Federated Learning training framework arXiv:2607.19524v1 Announce Type: new Abstract: Federated learning (FL) offers a promising approach to privacy-preserving clinical risk prediction, but its deployment remains limited by restricted data sharing, client heterogeneity, class imbalance, and the lack of realistic… 12 arXiv — Machine Learning research 22d ago Trustworthy Privacy-Preserving Multimodal Federated Learning for Personalised Breast Cancer Prediction arXiv:2607.19532v1 Announce Type: new Abstract: Federated learning has emerged as a potential solution to privacy concerns associated with using sensitive health data for training predictive models, particularly in personalised cancer care. This research investigates whether… 22 arXiv — Machine Learning research 22d ago An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies arXiv:2607.19771v1 Announce Type: new Abstract: Muon and related matrix-sign optimizers are increasingly used to pre-train large language models, but their effect on the internal geometry of individual weight matrices is not well understood. This preliminary report proposes a… 18 r/LocalLLaMA community 22d ago Built a from-scratch BitNet inference engine in pure C — 1.8× faster than bitnet.cpp on Xeon (36 tok/s), zero dependencies [BitNet & Bonsai CPU testers wanted] Hey [ r/LocalLLM ]( r/LocalLLM ), Built Project Zero — a from-scratch CPU-only LLM inference engine in pure C99. It beats bitnet.cpp by 1.8× on the same hardware. We also fully support Qwen Bonsai-27B on CPU, and we are looking for the community's help to get x86 CPU benchmark… 38 r/LocalLLaMA community 22d ago 16x AMD MI50 32GB: GLM-5.2 Q4 at 12.2 tok/s with llama.cpp RPC GLM-5.2 UD-Q4_K_XL GGUF @ 12.2 tok/s output // 30.9 tok/s input on a real 10.7k-token document using llama.cpp RPC - At 10.7k context: 10.2 tok/s output with coherent long-form generation Two parallel requests: 14.5 tok/s aggregate Context: 2x 16,384-token slots Model size: 436… 37 NVIDIA Developer Blog official-blog 22d ago Make Long-Running NVIDIA TensorRT Engine Builds Observable and Cancelable in Python or C++ A TensorRT engine build can take seconds to many minutes. Large strongly typed models, deep tactic search, and a cold timing cache on a brand-new GPU SKU can... 14 r/LocalLLaMA community 23d ago Today was the perfect day for Poolside to drop Laguna S 2.1 because I just got these in! Finally have a half decent amount of VRAM. 3x V620 = 96 GB. Laguna is the first model I'm trying, Q4_K_M fits with 256K context @ F16. Doing the html flight simulator test now. These cards are getting 400 to 600 tok/s prefill and 16 to 20 tok/s gen so far (I have NOT enabled dflash yet). Not bad at all for the cost. ($350 each) In a Dell… 18 arXiv — Machine Learning research 23d ago On the Limits of Support-Preserving Alignment and Bounded Filtering arXiv:2607.18295v1 Announce Type: new Abstract: We study whether alignment schemes that reshape a base model's output distribution, combined with bounded safety filters, can drive the probability of harmful behavior to zero in modern large language models. Recent research… 11 arXiv — NLP / Computation & Language research 23d ago Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning arXiv:2607.18722v1 Announce Type: cross Abstract: Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but staleness is an inevitable byproduct compounded by policy lag, engine delays, and mixture-of-experts routing. From a… 19 arXiv — Machine Learning research 23d ago KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scale arXiv:2607.18885v1 Announce Type: new Abstract: Kernel-based alignment of CLIP toward a vision centric teacher such as DINOv2 (KUEA) improves CLIP's visual representations while preserving text-encoder compatibility, using a fixed trade-off weight tuned on curated ImageNet-1K.… 35 arXiv — NLP / Computation & Language research 23d ago Operational Hallucination and Safety Drift in AI Agents arXiv:2607.18366v1 Announce Type: cross Abstract: Large language models (LLMs) serving as planners in tool-using autonomous agents introduce dynamic reliability risks in multi-turn execution. While single-turn safety mechanisms are relatively mature, extended interactions reveal… 26 Hugging Face Daily Papers research 23d ago Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning Abstract Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but staleness is an inevitable byproduct compounded by policy lag, engine delays, and mixture-of-experts routing. From a trust-region perspective, this mismatch… 15 arXiv — Machine Learning research 24d ago Fully-sensorized smart-eyewear platform for on-device Machine Learning arXiv:2607.16222v1 Announce Type: new Abstract: This paper presents ARGO, a smart eyewear platform designed to bridge ergonomic comfort, high computational throughput, and energy efficiency. Unlike cloud-dependent solutions, ARGO leverages the STM32N6 microcontroller and its… 11 arXiv — Machine Learning research 24d ago Benchmarking Machine Learning Models for Multi-Omics-Based Breast Cancer Prediction arXiv:2607.16250v1 Announce Type: new Abstract: Estrogen Receptor (ER) status is a critical biomarker in breast cancer diagnosis, prognosis, and treatment selection. Recent advances in high-throughput sequencing technologies have enabled the generation of multi-omics datasets… 18 arXiv — Machine Learning research 24d ago K-IPO: Kendall-constrained Importance Preserving Oversampling for Imbalanced Tabular Data arXiv:2607.16478v1 Announce Type: new Abstract: Oversampling is widely used to address class imbalance in tabular classification, but existing methods can distort the feature importance ranking underlying model explanations. Although recent studies have quantified this… 18 arXiv — NLP / Computation & Language research 24d ago PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning arXiv:2607.18199v1 Announce Type: new Abstract: Not all training samples contribute equally to large language model fine-tuning. Selecting informative training samples can reduce the computational cost while preserving downstream performance. Many existing data selection methods… 25 r/LocalLLaMA community 24d ago Benchmarked every spec-decode method on Qwen3.6-27B across vLLM and SGLang (single RTX PRO 6000 Max-Q) https://preview.redd.it/wluwwp6s4heh1.png?width=1248&format=png&auto=webp&s=6e95d963645c5a0ef750bf81324a6d4dcbc0389e Spent the last few days measuring speculative decoding on Qwen3.6-27B (dense, NVFP4) on one RTX PRO 6000 Max-Q, comparing vLLM and SGLang across MTP, DFlash,… 37 Vercel — AI dev-tools 24d ago Service tiers now available on AI Gateway AI Gateway now supports service tiering. Service tiers let you optimize for latency, throughput, and cost per request to match your use case. Pick a faster tier for interactive workloads (less queueing, higher token throughput), or a lower cost tier for background jobs that can… 13 r/LocalLLaMA community 24d ago 543 tok/s single-request Qwen3.6-35B-A3B on one RTX 5090 over a 65K-token decode An example TL;DR I have open-sourced NInfer , a from-scratch C++/CUDA inference engine currently specialized for two exact Qwen3.6 checkpoints on a single RTX 5090. Both the engine and the converted model artifacts are publicly available: Github :… 26 arXiv — Machine Learning research 25d ago Do Generative Models Keep Time? A Time-Aware Evaluation of Synthetic Sequential Tabular Data arXiv:2607.15606v1 Announce Type: new Abstract: Synthetic sequential tabular data are increasingly used for privacy-preserving data sharing, yet a generator can reproduce every marginal and every foreign-key relationship while emitting timestamps that run backwards or repeat,… 11 arXiv — Machine Learning research 25d ago PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization arXiv:2607.16184v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements… 19 arXiv — Machine Learning research 25d ago Fast and Scalable Caputo Fractional Gradient Descent via Perturbation-Preserving Memory Compression arXiv:2607.15505v1 Announce Type: cross Abstract: Fractional gradient descent (FGD) incorporates long-range memory through Caputo-type operators and has been shown to improve stability in ill-conditioned and nonconvex optimization problems. Despite these advantages, its… 38 r/LocalLLaMA community 26d ago "Basalt Labs" pulling a generationally dumb scam. Incredibly stupid lmao. Claiming 99.44% on HLE with tools. Model they released is based on Qwen2.5-7B-Instruct and the model they're serving on their website is DeepSeek.   submitted by   /u/WithoutReason1729 [link]   [comments] 29 r/LocalLLaMA community 27d ago Serving a fleet of Qwen3.5 122b sessions on a single Mac Studio (96GB) without losing your sanity Hello all Just following up on a post I made last week about my experiment to try minmax my Mac Studio. In particular, I've had quite a lot of success with pushing things even further. Across a 20 minute test with three concurrent sessions, my Mac Studio was offered 789,351… 5 r/LocalLLaMA community 27d ago GPT-OSS-120B, Qwen 30B and Gemma 26B on an Android phone at 1-5 tok/s: +60GB model, 11GB of RAM, CPU only This is a OnePlus 15R with about 11GB of usable RAM. The heaviest model is gpt-oss-120b, Q4_K_M , 60GB on disk . So it's roughly 5x bigger than the memory it's running in, which means keeping it resident isn't a matter of tuning, it just can't happen. It runs anyway: 1.3 tok/s… 18 arXiv — Machine Learning research 28d ago Gate-Zero Growth: A Geometric Framework for Function-Preserving Continual Learning arXiv:2607.14571v1 Announce Type: new Abstract: We introduce \emph{gate-zero growth}, a function-preserving (FP) operator for continual learning that adds new residual blocks through a zero-initialised gate. Under a transversality condition, gate-zero growth induces \emph{rank… 24 r/LocalLLaMA community 28d ago DFlash makes Qwen3.6 27B 2.2x faster with no quality loss We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks were: quicksort, write a Steam library in JSON, solve a logic puzzle and write a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98… 22 Hugging Face Daily Papers research 29d ago PalmClaw: A Native On-Device Agent Framework for Mobile Phones Abstract Large Language Model (LLM) agents have moved beyond generating responses to executing multi-step tasks by calling tools, observing the results, and iteratively deciding the next action. Most agent systems run on desktops or servers, which support tool use and task… 19 r/LocalLLaMA community 29d ago NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B on 2x3090s I managed to get this model working on 2x 3090s with full 262k ctx and N=4, if anyone is interested to try it, thanks to this quant: https://huggingface.co/danielrmay/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-W4A16 Nemotron-Labs-3-Puzzle-75B-A9B (W4A16) on 2× RTX 3090 — vLLM, no CPU… 6 arXiv — Machine Learning research 29d ago Federated Explainable Artificial Intelligence: Roles, Architectures, Evaluation, and Open Challenges arXiv:2607.13045v1 Announce Type: new Abstract: Federated Learning (FL) has emerged as a key paradigm for privacy-preserving collaborative model training across distributed and heterogeneous data sources. By keeping raw data local, FL addresses data confidentiality concerns, yet… 18 arXiv — Machine Learning research 29d ago Accuracy-Preserving Stability Regularization for Large-Scale Retail Demand Forecasting arXiv:2607.13331v1 Announce Type: new Abstract: Retail demand forecasts are reused across replenishment, capacity, labor, and transportation planning cycles. Point-error objectives do not constrain abrupt movement between adjacent forecasts, while post-hoc smoothing acts only… 15 r/LocalLLaMA community 29d ago Qwen3.5 122B-A10B · ROCmFP4 iMatrix Hola Strix and AMD stacker frendios. Read the Lineage and Credits, this uses charlie12345/ROCmFPX, won't work on native llama.cpp yet. 122B total · 10B active · 60.70 GiB · 28.50 tok/s MTP-off · BF16 KLD 0.041366 · Decode 28.505 Decode speed + 36.89% faster Size - 13.47gb… 6 r/LocalLLaMA community 29d ago OvisOCR2 (0.8B): first end-to-end model to top OmniDocBench - I threw 827 real scanned medical docs at it, here's everything I learned What it is: ATH-MaaS/OvisOCR2 - a 0.8B document-parsing VLM post-trained from Qwen3.5-0.8B (SFT + RL + OPD), Apache 2.0, runs on vLLM 0.22.1. One prompt per page image -> complete markdown (HTML tables, LaTeX formulas, bbox stubs for figures). It scores 96.58 on OmniDocBench… 9 arXiv — Machine Learning research 1mo ago Quantum Port-Hamiltonian Neural Networks: Learning Conservative and Dissipative Dynamics via Measurement-Induced Nonlinearity arXiv:2607.12269v1 Announce Type: new Abstract: We introduce Quantum Port-Hamiltonian Neural Networks (Q-pHNNs), a family of parameterised quantum circuits that learn classical dynamics in a structure-preserving manner. The framework relies on the Isomorphic Hamiltonian Mapping… 35 arXiv — NLP / Computation & Language research 1mo ago A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs arXiv:2607.12550v1 Announce Type: cross Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference. It grows with batch size, context length, and depth, and at long context it, rather than the model weights, sets the ceiling on throughput.… 31 arXiv — Machine Learning research 1mo ago AdaPCLA: Adaptive Prior-Calibrated Logit Adjustment for Long-Tailed Longitudinal EHR Generation arXiv:2607.12645v1 Announce Type: new Abstract: Generative modeling of longitudinal Electronic Health Records is increasingly important for privacy-preserving research, yet standard autoregressive models tend to underrepresent the co-occurrence structure of tail events (i.e.,… 9 arXiv — Machine Learning research 1mo ago Contrastive-Collapsed Loss for Flexible and Geometrically Optimal Embeddings and Faster Convergence arXiv:2607.12916v1 Announce Type: new Abstract: In this work, we introduce CoCo, a loss function aimed at learning normalized and well-structured representations. The proposed loss encourages intra-class collapse and inter-class contrast while preserving sufficient flexibility… 18 arXiv — NLP / Computation & Language research 1mo ago Evaluating Health Misinformation in Low-Resource Languages: Integrating Small Language Models with a Culturally-Sensitive Responsible NLP Framework (Bangla as a Case Study) arXiv:2607.12336v1 Announce Type: new Abstract: Artificial Intelligence (AI) technologies, while serving as a foundational enabler for modern social media and digital health services, exert a bivalent effect by simultaneously acting as a combatant against and a spread vector for… 25 arXiv — NLP / Computation & Language research 1mo ago PalmClaw: A Native On-Device Agent Framework for Mobile Phones arXiv:2607.13027v1 Announce Type: new Abstract: Large Language Model (LLM) agents have moved beyond generating responses to executing multi-step tasks by calling tools, observing the results, and iteratively deciding the next action. Most agent systems run on desktops or… 25 arXiv — NLP / Computation & Language research 1mo ago Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models arXiv:2602.02244v3 Announce Type: replace-cross Abstract: The standard post-training recipe for large reasoning models, supervised fine-tuning followed by reinforcement learning (SFT-then-RL), may limit the benefits of the RL stage: while SFT imitates expert demonstrations, it… 6 r/LocalLLaMA community 1mo ago Qwen 3.5 122B Heretic ROCmFP4 iMatrix My first time doing anything like this, and I built it because I wanted it. If anyone wants the non Heretic I'll mosey that out as compute allows. Strix Halo: 122B total · 10B active · 60.70 GiB · 28.45 tok/s · BF16 · KLD 0.100716 · PP 353.3 t/s   submitted by  … 18 r/MachineLearning community 1mo ago Cloud-vLLM Benchmark Differences [R] Does anyone know of any evidence/forum/paper analyzing benchmark result differences between cloud inference platforms (togetherai) and running models locally with vLLM under greedy decoding?   submitted by   /u/No_Cardiologist7609 [link]   [comments] 7 arXiv — Machine Learning research 1mo ago Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls arXiv:2607.09791v1 Announce Type: new Abstract: The multiplicative repetition penalty shipped across the LLM inference ecosystem (HuggingFace, vLLM, llama.cpp, and a dozen further engines) branches on the sign of each raw logit (divide positives by theta, multiply negatives).… 25 arXiv — Machine Learning research 1mo ago Serving the Long Tail: Training-Free LLM Candidate Generation for Vacation Rental Marketplaces arXiv:2607.09877v1 Announce Type: new Abstract: Vacation rental marketplaces face a structural imbalance on the supply side: a small fraction of properties receive most user interactions, while the long tail of new, niche, and seasonal listings generates too little behavioral… 22 arXiv — Machine Learning research 1mo ago Distance-Preserving Embeddings in Inhomogeneous Random Graphs arXiv:2607.10074v1 Announce Type: new Abstract: Graph machine learning provides powerful tools for understanding complex networks and learning meaningful node representations. A central challenge, however, is designing embeddings with minimal distortion of both local and global… 10 arXiv — NLP / Computation & Language research 1mo ago Large Language Models for Token-Efficient and Semantic-Preserving Opinion Summarization arXiv:2607.10825v1 Announce Type: new Abstract: Opinionated text - spanning product reviews, hotel feedback, and social posts - captures rich signals about user experiences, preferences, and concerns. However, the scale, redundancy, and imbalance of such corpora make it… 21 Page 4 of 10 · 500 articles ← Newer Older →
