News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow arXiv — Machine Learning research 3h ago Structure-preserving uncertainty quantification for GENERIC dynamics arXiv:2608.12624v1 Announce Type: new Abstract: Structure-preserving machine learning embeds physical structure directly into model architectures, yet uncertainty quantification (UQ) for such hard-constrained models remains limited because standard UQ methods may violate the… 18 arXiv — NLP / Computation & Language research 3h ago TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes arXiv:2608.13057v1 Announce Type: cross Abstract: In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU. Dispatchers balance token counts (EPLB, LPLB, UltraEP) or activated-expert counts (METRO), assuming expert time is linear in one. Measurements on… 32 r/LocalLLaMA community 13h ago EXPERIMENT: Qwen3.8-2.4T-A95B running locally on an RTX 5090 + RTX 5060 Ti at ~0.80 tok/s I managed to get Qwen3.8-2.4T-A95B running locally with llama.cpp on mu PC just for fun, cause why not. I was using the Unsloth Qwen3.8-2.4T-A95B-UD-Q1_0 GGUF quantization. The full GGUF is about 397 GiB . The model uses 512 routed experts, with 10 active per token. My hardware:… 31 OpenAI official-blog 21h ago Previewing Ultrafast mode: GPT-5.6 Sol at up to 14X the speed Preview Ultrafast, a new OpenAI API service tier that runs GPT-5.6 Sol up to 14× faster. Powered by Cerebras, it delivers up to 750 output tokens per second. 24 arXiv — NLP / Computation & Language research 1d ago Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems arXiv:2608.11879v1 Announce Type: new Abstract: Long-running conversational agents increasingly rely on a memory system to avoid resending the whole conversation each turn, yet how much that costs to serve has received little systematic benchmarking. We compare three memory… 15 arXiv — NLP / Computation & Language research 1d ago QV-PIC: Query-Aware Visual Position-Independent Caching for Efficient RAG Serving arXiv:2608.12121v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) repeatedly prefills identical text chunks across queries, incurring redundant computations. Position-Independent Caching (PIC) mitigates it by reusing precomputed Key-Value (KV) across… 27 Hugging Face Daily Papers research 1d ago NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs Abstract NeuPAT selectively constrains updates to language-sensitive neurons during multimodal tuning to preserve LLM language capabilities while enabling perceptual adaptation. Generated by thinkingmachines/Inkling-Small Multimodal expansion of large language models (LLMs)… 18 r/LocalLLaMA community 1d ago Qwen 3.8 27B — MTP or DFlash? Do we.know whether the 27B model will ship with a DFlash or MTP head? It's super exciting, but since 35B-A3B is my daily driver, 27B will crawl — still excited for it though! I think 3.6 27B with MTP was about 8 tok/s for me (32GB unified memory, 780M)   submitted by  … 25 Hugging Face Daily Papers research 1d ago SkillZip: Evaluation-Free Skill Compression for Self-Evolving Agents by Discovering Reusable Structure Abstract SkillZip compresses self-evolving agent skills by finding a minimal faithful structural explanation that shares repeated rules and procedures while preserving rare exceptions, without requiring evaluation rollouts. Generated by thinkingmachines/Inkling-Small… 26 r/LocalLLaMA community 1d ago Meta's Muse Glimmer 30B now runs up to ~3.3x faster on Mac with mlx-dspark Been tinkering with speculative decoding on Apple Silicon for a while, and this week I got Meta's new Muse Glimmer 30B working in my project mlx-dspark . On my M4 Pro, the 8-bit model goes from 8.2 tok/s to 18-26 tok/s depending on content. Math is the best case at 3.27x, code… 12 r/LocalLLaMA community 1d ago Benched a 124B on one DGX Spark for a week and published all of it — 38.7 tok/s on the fastest path he found, 2.4x DeepSeek V4 Flash on the same box sudoingX on X spent a week on this and posted his wrap-up. The hardware is his. He isn't affiliated with us and we didn't see any of it before he put it out — I work on Ling at inclusionAI. Where he landed on one Spark: 38.7 tok/s on the official INT4 once it's configured right,… 9 Hugging Face Daily Papers research 2d ago UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models Abstract UniMoMo compresses trained recommendation mixture-of-experts models into smaller standard MoE checkpoints via functional similarity grouping and layer-adaptive protection, preserving accuracy while accelerating inference. Generated by thinkingmachines/Inkling-Small… 27 arXiv — Machine Learning research 2d ago Invertible Logits Transformation for Accuracy-Preserving Post-Hoc Uncertainty Calibration arXiv:2608.10372v1 Announce Type: new Abstract: Post-hoc calibration aligns a classifier's predicted confidences with its empirical accuracy without retraining. An ideal calibrator should correct nonlinear miscalibration, scale gracefully to large label spaces, and preserve the… 25 arXiv — Machine Learning research 2d ago Benchmarking Time Series Generation Methods for Privacy-Preserving Forecasting arXiv:2608.10891v1 Announce Type: new Abstract: Time series forecasting in privacy-sensitive domains often requires training models on released data rather than original observations. Synthetic time series generation has been developed primarily for data augmentation, where… 6 arXiv — Machine Learning research 2d ago Two-stage Odd Residual Flows for Mean-Preserving Probabilistic Time Series Forecasting arXiv:2608.11114v1 Announce Type: new Abstract: Probabilistic forecasting plays an essential role in risk-sensitive decision-making, particularly in long-horizon settings. However, existing approaches often face a fundamental trade-off between distributional flexibility and… 31 TechCrunch — AI news-outlet 2d ago Brad Lightcap, OpenAI’s longtime COO, is leaving to ‘start something new’ One of OpenAI's longest serving executives is headed out the door, although the longtime COO told staff that he was "excited to help you all advance the mission from a different vantage point." 10 r/LocalLLaMA community 2d ago Ling-3.0-flash quant ladder on one DGX Spark: the whole thing sits in a 32 to 40 tok/s band The interesting part of this one isn't the top number, it's how little distance there is between the top and the bottom of the ladder. Where it comes from: I work on Ling at inclusionAI, these aren't my numbers. sudoingX on X benched the full community GGUF ladder on his own DGX… 36 r/LocalLLaMA community 2d ago Muse Glimmer 30B running locally in-browser with custom WebGPU kernels at ~25 tok/s on an M4 Max (same speed as llama.cpp)   submitted by   /u/xenovatech [link]   [comments] 23 r/MachineLearning community 2d ago Planning/RL for a stochastic single-player merge puzzle: afterstates, previewed chance events, and long-horizon throughput [D] I am working on an AI for a small single-player merge puzzle and would appreciate pointers to related algorithms, papers, or existing implementations. It resembles 2048 in its action -> afterstate -> random event structure, but has a larger action space, stack constraints, and a… 18 Hugging Face Daily Papers research 3d ago OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching Abstract OasisKV improves LLM inference throughput by storing full KV caches in lower memory tiers and prefetching only relevant entries into HBM using speculative-decoding lookahead predictions. Generated by thinkingmachines/Inkling-Small Large language model (LLM) inference… 6 arXiv — Machine Learning research 3d ago ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling arXiv:2608.07974v1 Announce Type: new Abstract: Large language model (LLM) fine-tuning at the edge adapts the model to scenario-specific data while preserving privacy. Although existing studies proposed pipeline parallelism to address the limited memory and computing resources… 10 arXiv — Machine Learning research 3d ago Efficient Test-Time Scaling for LLM-based Time Series Forecasting arXiv:2608.08675v1 Announce Type: new Abstract: Long-term time series forecasting benefits from preserving global structure such as trends and seasonality. Recent LLM-based forecasters often improve accuracy through test-time scaling (e.g., iterative refinement), but these… 27 arXiv — NLP / Computation & Language research 3d ago NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs arXiv:2608.08107v1 Announce Type: new Abstract: Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective… 8 arXiv — NLP / Computation & Language research 3d ago Tied Trit-Planes: Constraining PTQTP to a Uniform Nine-Level Quantizer, with a Persistent Folded Format for Disk-Streamed Mixture-of-Experts Serving arXiv:2608.08910v1 Announce Type: new Abstract: PTQTP decomposes LLM weight matrices into two ternary (trit) planes with two free per-group scales. Tying the scales to a fixed ratio of three collapses the decomposition into a single uniform nine-level quantizer, a known… 29 r/LocalLLaMA community 3d ago Observations on Muse-Glimmer reasoning traces being noticeably different from qwen / gemma models and questions for you guys Just downloaded the model, UD-Q5_K_XL quant, asked it to generate a long story to test out reasoning and speed with dflash (super fast btw, ~ 90 to 160 tok/s on a 5090 depending on task) and was surprised at the reasoning traces, they are so unlike anything i've seen recently… 5 Hugging Face Daily Papers research 3d ago Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence Abstract Vision-language models are increasingly serving as the reasoning core of embodied agents. Robot execution is inherently iterative: each action reshapes the scene and physical state, continually renewing what must be perceived, reasoned about, and verified. Meeting these… 25 arXiv — Machine Learning research 4d ago CrystalGRPO: Target-Aligned and Coverage-Preserving Reinforcement Learning for Flow-Based Crystal Structure Prediction arXiv:2608.06582v1 Announce Type: new Abstract: Flow-based generative models can efficiently produce candidate structures for crystal structure prediction (CSP), but their pretrained objectives do not directly optimize downstream target recovery. Reinforcement-learning… 33 arXiv — Machine Learning research 4d ago Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors arXiv:2608.06723v1 Announce Type: new Abstract: The rapid scaling of Large Language Models (LLMs) has significantly increased computational cost, energy consumption, and inference latency, making accurate estimation essential for sustainable artificial intelligence deployment… 8 arXiv — Machine Learning research 4d ago CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights arXiv:2608.06763v1 Announce Type: new Abstract: Weight quantization for large-language-model inference must balance adaptive reconstruction levels with representations regular enough for efficient GPU execution. Uniform integers constrain each group to a linear grid. Low-bit… 9 arXiv — Machine Learning research 4d ago MAUPITI: On-Device Prototype-Based Learning on a Smart Infrared Sensor arXiv:2608.07192v1 Announce Type: new Abstract: Low-resolution infrared (IR) array sensors represent an interesting solution for privacy-preserving human sensing in embedded systems. In this letter, we describe a smart multi-pixel IR sensor integrating a 16$\times$16 thermal… 13 arXiv — Machine Learning research 4d ago TOFD: Target-Oriented Feature Decoupling against Poisoning Attacks in Split Federated Learning arXiv:2608.07274v1 Announce Type: new Abstract: Split Federated Learning (SFL) facilitates privacy-preserving collaborative training with reduced client-side overhead. However, its split architecture introduces unique attack surfaces, rendering it vulnerable to diverse poisoning… 22 arXiv — Machine Learning research 4d ago Residual Algebra for Representation-Preserving Learning arXiv:2608.07349v1 Announce Type: new Abstract: Learning from heterogeneous representations is usually reduced to feature concatenation, which erases which representation produced an error. We instead algebraize the residual: a representation is a typed object that owns both a… 14 arXiv — Machine Learning research 4d ago Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving arXiv:2608.06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests. LLM serving platforms today define… 34 arXiv — NLP / Computation & Language research 4d ago Model Confidence Under Answer-Preserving Attacks: An Informativeness-Manipulability Frontier arXiv:2608.06571v1 Announce Type: cross Abstract: Deployed vision-language systems often gate their answers on confidence, making confidence robustness relevant to oversight. We study confidence readouts under white-box, image-only attacks constrained to preserve the generated… 11 r/LocalLLaMA community 4d ago [2606.05682] Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments. Quantization-aware distillation (QAD) helps recover accuracy lost under low bit… 32 r/LocalLLaMA community 4d ago Two flags took the official Ling-3.0-flash INT4 from 20.8 to 38.7 tok/s on one DGX Spark The official INT4 does load on a single DGX Spark. The naive config just leaves most of its speed on the floor, 20.8 tok/s. Two changes take it to 38.7. Quick context on where this comes from: I work on Ling at inclusionAI, and none of these numbers are mine. sudoingX on X ran… 27 r/LocalLLaMA community 5d ago Memory Bandwidth problems with Intel Sapphire Rapids I have a Xeon w7-3465 and 4 sticks of RDIMM DDR5-4800 with a theoretical max bandwidth of 153GB/s. I am trying to run DeepSeek-V4-Flash-0731 as it is an MoE and the weights are in MXFP4, so I should roughly process 150GB/6.5GB tokens per second as it has 13B Experts. The problem… 11 r/LocalLLaMA community 5d ago enabling PCI-E p2p for consumer Nvidia cards will yield you more than you think Disclaimer - no LLM was used to write this post/note As larger post about my setup will come later, want to give heads-up to folks who use VLLM and >= 2 GPUs. So I have pretty meaty server (8 channel AMD EPYC, ~150GB/s RAM bw) and 4x5060Ti 16GB in PCI-E 4.0 8x mode. Considering… 29 r/LocalLLaMA community 5d ago Building a zero-dependency C inference engine for BitNet (1.58-bit) - lessons from hitting 36 tok/s on a Xeon CPU Over the past few months I have been building a CPU-first inference engine from scratch in pure C99 (no Python, no CUDA, no BLAS, just GCC and make). The focus has been running 1.58-bit ternary models natively without heavy runtime overhead. Currently it hits 36.25 tok/s on… 30 r/LocalLLaMA community 5d ago Qwen3.6 27B + 35B on vLLM, single R9700 (gfx1201) I've been tuning my new Radeon AI Pro R9700, and figured that this would be useful information for people who are trying to optimise their setups. I'm pretty happy with these results and looking forward to Qwen3.8.. Summary below provided by Claude (which helped me configure it… 18 r/LocalLLaMA community 6d ago Qwen 35B-A3B MoE vs 27B dense in local coding tests: ~4× faster, much smaller quality gap than I expected I compared Qwen 35B-A3B MoE against Qwen 27B dense on a series of local coding-maintenance tasks. On my R9700/llama.cpp setup, the MoE model generated about 3.9× faster (~116 vs ~30 tok/s) , but the coding-quality difference was much smaller than I expected. Both usually handled… 8 r/LocalLLaMA community 6d ago Serving Deepseek v4 Flash 0731 on 2x DGX Spark — 5-7 GB OS headroom, what would you do to lower VRAM usage and increase OS available RAM? Hey all, I'm serving DSv4Flash 0731 on a cluster of 2x DGX Sparks but am running into constant issues with having almost no RAM (unified memory) left for the OS/cache and I'd love to hear the community feedback on what I could do to get more RAM for headroom. The DGX has an… 32 r/LocalLLaMA community 6d ago DeepSeek v4 Flash 0731 on H100 node Hi, I've been playing with recent DeepSeek Flash updated model and it seems excellent. Like talking to Claude 4.8. I'm currently deploying it on VLLm and I'm in progres of tuning performance. I seem to hit the point where I trade decode with prefill by juggling batching knobs.… 21 r/LocalLLaMA community 6d ago A llama.cpp PR makes Q2_0 3.0–3.6x faster on x86 CPUs, 8B decode goes 2.39 → 8.20 tok/s I was going through the current llama.cpp CPU PRs and #26348 stood out because this isn't the usual +5% kernel optimization. It adds an x86 VNNI implementation for the Q2_0 × Q8_0 dot product, and the author's controlled CPU-only benchmarks show roughly 3–3.6x higher throughput… 24 r/LocalLLaMA community 6d ago Anyone running DeepSeek-V4-Flash-0731 on MI325X with vLLM? Mine is behaving completely broken Is anyone here successfully running DeepSeek-V4-Flash-0731 locally with vLLM , especially on AMD MI325X? My setup: GPU: 1x AMD Instinct MI325X Model: deepseek-ai/DeepSeek-V4-Flash-0731 vLLM: 0.26.0 ROCm image --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4… 20 Hugging Face Daily Papers research 6d ago SkillZip: Contract-Preserving Graph Compression for Scalable Agent Skill Libraries Abstract SkillZip compresses reusable procedural skills into contract-preserving, executable graph units to enable efficient retrieval and expansion under limited context budgets. Generated by thinkingmachines/Inkling-Small Large Language Models (LLMs) increasingly act as agents… 14 arXiv — Machine Learning research 7d ago When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering arXiv:2608.05670v1 Announce Type: new Abstract: A model's agreement across perturbed inputs is used both as a label-free reliability signal and as a self-training target, on the premise that agreement tracks correctness. That coupling is rarely measured directly: natural-image… 8 arXiv — Machine Learning research 7d ago LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm arXiv:2608.06135v1 Announce Type: new Abstract: Large Language Models (LLMs) such as ChatGPT and Claude are widely used for information retrieval and problem-solving. Recent work has focused on improving scheduling algorithms to boost throughput while maintaining low latency.… 12 arXiv — Machine Learning research 7d ago Hypothesis Testing with Conditional Queries: Learnability and the Value of Interaction arXiv:2608.06262v1 Announce Type: new Abstract: Model evaluations may fix all tests before observing any responses or select later tests using earlier responses. We study this choice in a conditional-query model on a finite outcome space $\mathcal{X}$ with $|\mathcal{X}|=N$. We… 17 arXiv — NLP / Computation & Language research 7d ago Example-Guided Prompting for Document-Level Text Simplification arXiv:2608.05447v1 Announce Type: new Abstract: Document-level text simplification requires large language models (LLMs) to rewrite complex documents while preserving meaning, readability, and discourse coherence. Although prompt-based LLMs have shown promising performance, they… 30 Page 1 of 10 · 500 articles Older →
