News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow r/LocalLLaMA community 20h ago Updated my benchmark with a new vLLM based recipe for Qwen 3.8 Flash Next : now up to 98/100 (instead of 91 previously) I was using: weights https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 with the optimized SGLANG (patched) from https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_on_1x_rtx_pro_6000_171_ts_c1_428/ Now I'm using: weights (AWQ W4A16) from:… 11 r/LocalLLaMA community 21h ago You can now run a 90M conversational LLM on the Sony PSP (hardware from 2004). Doesn't get more local than this. Github link: https://github.com/thatblend/LLMPSP I wanted to see what the PSP can theoretically handle and I got my answer - a 90M model is about the max it can do without atrocious inference speeds. It's running around 0.5 - 0.6 tokens per second, which is very slow, but it's… 6 Hugging Face Daily Papers research 1d ago Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration Abstract CORD is a post-fit adapter that repairs calibrated probability vectors to exactly preserve original top-1 predictions while maintaining calibration quality. Generated by thinkingmachines/Inkling-Small Post-hoc calibration corrects reported confidence, yet a multiclass… 14 arXiv — NLP / Computation & Language research 1d ago Privacy-Preserving Heterogeneous Multi-LLM Federated Inference for Cognitive Diagnosis arXiv:2609.02947v1 Announce Type: cross Abstract: Significant challenges remain in AI-driven educational systems in balancing privacy preservation with accurate cognitive diagnosis. To overcome this, we propose a federated inference framework in which several commercial LLM APIs… 36 arXiv — Machine Learning research 1d ago Privacy-Preserving Topology-Guided Safety for LLM-Based Multi-Agent Systems via Federated Graph Learning arXiv:2609.02967v1 Announce Type: cross Abstract: Topology-guided safeguards for LLM-based multi-agent systems (MAS) train a GNN over the inter-agent communication graph to localize risky agents and intervene on the topology---but they assume one operator can pool all labeled… 11 arXiv — Machine Learning research 1d ago Privacy Leakage in Federated Learning: Gradient-Based Client Identity Inference and Defenses for Inertial Sensing in Vehicular Edge Networks arXiv:2609.02971v1 Announce Type: cross Abstract: As vehicular networks move toward 5G/6G edge intelligence, federated learning (FL) is widely promoted as a privacy-preserving way for vehicles and infrastructure to train shared models without exposing raw sensor data. Yet the… 22 arXiv — NLP / Computation & Language research 1d ago Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations arXiv:2609.03511v1 Announce Type: new Abstract: Large Language Models (LLMs) demonstrate strong multilingual reasoning performance, yet their robustness to semantics-preserving structural variation remains underexplored, particularly for relatively free word-order languages. We… 4 r/LocalLLaMA community 1d ago Qwen 3.8 27B Vs. Qwen 3.6 27B on oMLX Quality: 81.1 → 87.7 (+8%) Speed: 35 → 29 tok/s (−16%) Runtime: 8m51s → 44m39s (5x longer) Output tokens: 18K → 78K (🤯) Noticeably better quality, but you're paying for it with tokens and time. Full benchmark results (all hardware, all quants): llm-bench.io Qwen3.6-27B Vs.… 28 r/LocalLLaMA community 1d ago Qwen3.8-Flash-Next MTP merged in ik_llama.cpp (integrated head or separate -md file)... 45 → 90 tok/s on a 5090 + 128GB, works down to a 12GB 4070 ik_llama.cpp merged qwen4exp MTP support yesterday (PR #2369, mine, reviewed and tested by four other people on their own hardware). It's on main now, no fork or patch needed. Posting since the last couple threads had people saying MTP for this model only exists as an unsloth… 34 r/LocalLLaMA community 2d ago My RULE of Thumb of choosing a models This is mostly for setting up for expectation, since personally without LLM i could take 3 days (15 hours of active programming) to debug or implement a feature, but with Qwen 27B (even before Qwen 3.8) it take 4 hours. And yes 0.5 tok/s is human, not accounting of deletion and… 37 arXiv — Machine Learning research 2d ago OutageDiT: A Generative Foundation Model for Power Outage Forecasting and Scenario Simulation arXiv:2609.01896v1 Announce Type: new Abstract: Power-outage planning requires scenarios before an event occurs. These scenarios must represent uncertainty in magnitude, timing, and duration while preserving temporal dependence. However, severe events are rare, and data from any… 14 arXiv — Machine Learning research 2d ago Source-Free Class Relearning: Diagnosing Forgetting in Class Unlearning arXiv:2609.02018v1 Announce Type: new Abstract: Class unlearning aims to remove a model's ability to recognize designated forget classes while preserving performance on retain classes. However, low forget accuracy after unlearning does not necessarily mean the class structure… 16 arXiv — NLP / Computation & Language research 2d ago XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compression arXiv:2609.02083v1 Announce Type: cross Abstract: Removing complete transformer layers preserves a standard serving architecture, but existing depth-compression methods can lose substantial quality, and the loss varies unpredictably across models. We introduce XMerge, a… 10 arXiv — Machine Learning research 2d ago Exact Limits of Random Projections for Preserving Geometry: Distance Recovery, Nearest-Neighbor Rankings, and Covariance Shape in Gaussian Models arXiv:2609.02155v1 Announce Type: new Abstract: The Johnson-Lindenstrauss (JL) lemma guarantees that a random projection of $n$ points to $m=O(\varepsilon^{-2}\log n)$ dimensions preserves pairwise squared distances within relative error $\varepsilon$ with high probability, and… 16 arXiv — Machine Learning research 2d ago Similarity-Aware Personalized Federated Learning in Heterogeneous Environments arXiv:2609.02241v1 Announce Type: new Abstract: Federated Learning (FL) allows decentralized clients to train models collaboratively while preserving data privacy. However, distribution mismatch across clients often leads to poor global generalization and degraded local… 33 arXiv — Machine Learning research 2d ago Online Reinforcement Learning in the Met Office Unified Model through Distributed Model-Agent Coupling arXiv:2609.02566v1 Announce Type: new Abstract: Machine-learnt corrections can complement numerical weather prediction only if they adapt to the evolving model state while preserving dynamical consistency and numerical stability. To test this within a global forecasting model,… 24 arXiv — NLP / Computation & Language research 2d ago HyperStyler: Low-resource Authorship Style Transfer via Context-aware Style Navigation and Hypernetworks arXiv:2609.02772v1 Announce Type: new Abstract: Low-resource authorship style transfer (LAST) aims to rewrite text into the style of an arbitrary target author using only a few reference examples while preserving the original meaning. Existing methods often struggle to achieve… 33 Hugging Face Daily Papers research 2d ago Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation Abstract Influence-Directed Adaptive On-Policy Distillation improves diversity transfer in reasoning model distillation by selectively preserving entropy-expanding updates and replacing entropy-contracting ones with adaptive advantage shrinkage. Generated by… 23 r/LocalLLaMA community 2d ago Quad R9700 AI Pro with vLLM-Radiance easily reaching 17,6k PP https://preview.redd.it/74bmvel9b5nh1.png?width=1602&format=png&auto=webp&s=0d0c1adaa016a486ffd97c4c466e980dc611b139 I've only recently started looking deeper into vLLM after running llama.cpp for a good while. Initially vLLM (official repo) was terribly slow on my four R9700s… 10 Hugging Face Daily Papers research 3d ago From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix Abstract A smaller self-hosted LLM trained with separate GRPO experts merged via SLERP outperforms a much larger baseline on instruction following, function-calling, and internal tasks while serving half of platform traffic at lower cost. Generated by… 31 arXiv — Machine Learning research 3d ago QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization arXiv:2609.00224v1 Announce Type: new Abstract: Weight-only post-training quantization (PTQ) can alleviate the computational burden of serving large language models (LLMs) at scale. However, existing PTQ methods often fail to generalize across models and suffer severe accuracy… 36 arXiv — Machine Learning research 3d ago Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration arXiv:2609.01072v1 Announce Type: new Abstract: Post-hoc calibration corrects reported confidence, yet a multiclass calibrator can also change the associated top-1 prediction. Accuracy captures only the net effect of these changes on correctness, not how often predictions… 6 arXiv — Machine Learning research 3d ago Bandits in Prod: Hyperparameter Optimization at Inference Time arXiv:2609.01335v1 Announce Type: new Abstract: Many production systems can assess a configuration only by using it on live requests and observing noisy feedback. Modern agentic systems are a prominent example, with inference-time choices such as model selection, retrieval… 35 arXiv — NLP / Computation & Language research 3d ago OUTLETS: Output-Length Prediction from Speculative Decoding Backbones arXiv:2609.01068v1 Announce Type: new Abstract: The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing… 17 r/LocalLLaMA community 3d ago Running 104GB Qwen3.8-Flash-Next on 48GB Mac at ~12 tok/s   submitted by   /u/yogthos [link]   [comments] 34 r/LocalLLaMA community 3d ago 2/5 of my CMP 170HX have died after 2 weeks and the 3rd came with defective tensor cores. Current prices DO NOT justify the risk you are taking One GPU drops off immediately when vllm is started and the other throws CUDA errors on start   submitted by   /u/cantgetthistowork [link]   [comments] 22 r/LocalLLaMA community 3d ago Qwen3.8-Flash-Next (104 GB MoE) on a Strix Halo + RTX 3090 Ti eGPU: 22 -> 84 tok/s, and within one HumanEval+ problem of a dual-3090 vLLM box at 0.4x the wall time Follow-up to my Qwen3.8-27B post. This time the target is Qwen3.8-Flash-Next: 512 experts per layer, 36 layers of gated DeltaNet, 12 layers of top-k sparse attention, a 26.8 GiB n-gram table and a built-in MTP draft head. unsloth UD-Q4_K_XL, 103.69 GiB. It fits in the Strix… 30 r/LocalLLaMA community 3d ago Gave a try to Exllamav3 and it's great! Following this post I decided to try GLM 5.3 Flash on a 8x3090 setup and I can now run a Q4 with surprising speed; 700tk/s prefill & 42tk/s decoding! (lcp & vllm do not allow me to get that). Was afraid about quality but > 30m tokens with DSH and no issue (did not test vision… 36 r/LocalLLaMA community 3d ago How I got 280 tok/s on Qwen3.8 27B on 2xr9700's and 940k tokens kv cache 2 Months ago I had made a post how I was working on my dual R9700's. It's wild to look back at where we were then and where things now stand. Since then after many users commenting and complaining about developers doing the same thing. I threw out a discord link and expected… 25 Hugging Face Daily Papers research 3d ago RECAP-Forcing: Retaining Content Appearances for Long Video Generation Abstract RECAP-Forcing improves long video generation by indexing memory according to appearance novelty rather than recency, preserving key-value caches for newly visible content to maintain long-range consistency without extra training. Generated by… 6 r/LocalLLaMA community 3d ago Question: Why is prefill unbelievably faster in vLLM than other inference engines? I only started using some vLLM forks recently in a 4 x 48GB 4090 system. DS4F - ~5000pp/180tg (DSpark) Qwen3.8 Flash next - ~7500pp/135tg (MTP) This is amazing, like having the API in my house. But it's also really hard to go back. It's weird that we never come close to prefill… 27 Hacker News — AI on Front Page community 3d ago Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s I built slotstream, a way to run Qwen3.8-Flash-Next 4-bit on a low-memory mac starting from 16GB, a 125B parameter model that would need 100GB+ memory/RAM, thanks to expert-offloading/ssd-streaming. Easy to install/update, and mac-native using MLX and Swift. It ships with… 34 Hugging Face Daily Papers research 4d ago Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation Abstract KATok is an adaptive transformer-based video tokenizer that selectively drops uninformative tokens to achieve data-dependent compression while preserving spatial consistency for diffusion-based generation. Generated by thinkingmachines/Inkling-Small Latent diffusion… 10 arXiv — Machine Learning research 4d ago MEL: Coordinate-Preserving EEG Tokenization for fMRI Translation arXiv:2608.29304v1 Announce Type: new Abstract: Translating electroencephalography (EEG) into functional magnetic resonance imaging (fMRI) is important for medical neuroimaging, clinical brain-state monitoring, and multimodal neural decoding, because it aims to infer spatially… 34 arXiv — Machine Learning research 4d ago Creation begins with understanding: LLMs as strategy designers for privacy-preserving tabular data synthesis arXiv:2608.29674v1 Announce Type: new Abstract: Sharing tabular data in high-stakes domains is constrained by privacy regulations. Synthetic data offer a promising alternative, but deep generative models are costly to train and difficult to audit, while LLM-based methods often… 28 arXiv — Machine Learning research 4d ago INTERVenE: Temporal-Abstraction-Interval Based Transformers for Short-Horizon Medical Event Prediction arXiv:2608.29901v1 Announce Type: new Abstract: Electronic Health Record (EHR) prediction models in the intensive care unit must learn from sparse and irregular measurements while preserving the clinical meaning of time and supporting transparent decision-making. We present… 21 arXiv — NLP / Computation & Language research 4d ago When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization arXiv:2608.29327v1 Announce Type: new Abstract: Large language models deployed in specialized domains must improve in-domain performance without sacrificing general capabilities. Existing parameter-efficient fine-tuning methods are typically always on: their learned… 38 r/LocalLLaMA community 4d ago Qwen3.8-Flash-Next in llama.cpp from CPU-only to 96GB VRAM: 8.5 to 109 tok/s, max context and parameters test. My findings on RTX 6000 PRO. Hey guys, I tested Qwen3.8 Flash with llama.cpp from CPU-only to the full 96GB of my RTX PRO 6000. Short version: CPU-only reached 8.34 tok/s at a 2K prompt Full 96GB reached 109.07 tok/s At 245K context, 24GB to 96GB gave 14.89 to 21.61 tok/s The 96GB advantage over 24GB… 6 Hugging Face Daily Papers research 5d ago GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models Abstract GGSS reduces demographic bias in generative vision-language models by steering visual tokens along geodesic arcs with an adaptive gate, preserving visual-language accuracy. Generated by thinkingmachines/Inkling-Small Generative vision-language models (VLMs) are… 19 OpenAI official-blog 5d ago OpenAI supports California’s bill to advance youth AI safety OpenAI supports California SB 1119, advancing strong, age-appropriate AI safeguards for teens while preserving opportunities to learn, create, and explore. 28 arXiv — Machine Learning research 5d ago The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension arXiv:2608.28150v1 Announce Type: new Abstract: Which geometry controls the rank complexity of normalized softmax attention? We study maximum-row-$\ell_1$ approximation rank, exactly the least unrestricted rank preserving every bounded vector-valued output. Two sharp worst-case… 19 arXiv — Machine Learning research 5d ago Performative Privacy: When Differential Privacy Maximizes Utility arXiv:2608.28198v1 Announce Type: new Abstract: Privacy-preserving learning is often motivated by the idea that protecting users' data can preserve trust and thus participation, improving utility in the long term. However, this claim has not been formalized so far. In parallel,… 21 arXiv — Machine Learning research 5d ago Not to Break, but to Attest: Adversarial Probes for Privacy-Preserving LLM Verification arXiv:2608.27954v1 Announce Type: cross Abstract: Post-deployment changes to large language models can alter behavior while leaving routine outputs largely unchanged, creating a challenge for AI governance when model weights are proprietary. We present a privacy-preserving… 20 arXiv — Machine Learning research 5d ago Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms arXiv:2608.28044v1 Announce Type: cross Abstract: Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete, since average output-token energy can… 9 arXiv — NLP / Computation & Language research 5d ago Trajectory-Level Speculative Decoding for Diffusion Language Models arXiv:2608.27514v1 Announce Type: new Abstract: Diffusion-based language models (dLLMs) enable parallel token generation through iterative denoising, but existing decoding strategies collapse to single-token generation under low confidence, severely limiting throughput. Unlike… 21 arXiv — NLP / Computation & Language research 5d ago SimpCue: Cue-Based Prompting for Multilingual Text Simplification arXiv:2608.28042v1 Announce Type: new Abstract: Text simplification aims to make complex texts easier to understand while preserving their original meaning. Recent large language models can perform simplification through prompting, but it remains unclear whether adding explicit… 25 arXiv — NLP / Computation & Language research 5d ago A Probabilistic Interpretation of KV Cache Eviction arXiv:2608.28293v1 Announce Type: new Abstract: The premise and promise of KV (cache) eviction is simple: higher throughput can be achieved by evicting some entries from the KV cache, at a negligible cost to quality. This holds empirically for many existing methods, though most… 37 arXiv — NLP / Computation & Language research 5d ago ContextPilot: Teaching Agents for Proactive Context Management via Fine-grained RL arXiv:2608.28476v1 Announce Type: new Abstract: Long-horizon agentic tasks require large language models (LLMs) to iteratively retrieve, integrate, and maintain dispersed information across multi-turn interactions, but preserving all interaction histories leads to a continuously… 27 arXiv — NLP / Computation & Language research 5d ago Semantic Watermarking with Order-Robust Detection over Sub-sentence Units arXiv:2608.27666v1 Announce Type: cross Abstract: Semantic watermarks tie the mark to sentence meaning rather than token choices, promising robustness to content-preserving edits. However, the detector only observes attacker-supplied text, which can be reworded, reordered, or… 34 Hugging Face Daily Papers research 5d ago LMSM: LLM Security Framework Inspired by Linux Security Modules Abstract LMSM applies a modular security framework to LLM serving by separating evidence calibration, policy evaluation, and output gating, enabling flexible interpretability-based enforcement without rebuilding request handling. Generated by thinkingmachines/Inkling-Small Large… 19 Page 1 of 10 · 500 articles Older →
