News / #inference Tag Inference 500 articles archived under #inference · RSS Sign in to follow r/LocalLLaMA community 21d ago vllm + p2p driver hack + qwen 3.8 27B vs llamacpp + qwen flash next ? Hi everyone I'm running four rtx 4090, 64GB ram, on a threadripper pro motherboard so all PCIe x16 ports, as a homelab machine for coding. I was migrating from vllm + qwen 3.8 27B (fp8+256k kv cache) to llamacpp + qwen flash next iq4xs + 8 bit cache 200k kv cache... until… 34 r/LocalLLaMA community 21d ago 2x R9700, 64 GB DDR5 is an absolute beast machine with vLLM Radiance / R9V and Qwen 3.8 27b and Flash next I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, also because I wanted to upgrade my gaming graphics for 4k. As the 5090 was about… 14 r/LocalLLaMA community 22d ago Validate your local LLM advertised KV cache against real pressure; see exactly how old contexts get evicted from cache Hello, I'm a bit obsessed with cache management on local LLMs. For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was off so I investigated, found issues, fixed them, but needed a way to validate… 22 r/LocalLLaMA community 22d ago Which qwen for vllm? Hugging face has over 300 versions of qwen3.8-27b. I have no idea how to identify the beat model to download and use. Should I sort by “most likes” or “most downloads”? Is there any sort of other guidance on which model to select? I’m finally getting my local linux box up and… 4 r/LocalLLaMA community 22d ago Qwen3.8-Flash-Next-oQ4e-mtp: 45 tok/s on M4 Max, 25 tok/s on M2 Ultra for local inference — llm-bench.io Qwen 3.8 Flash Next gives similar speed than Qwen3.8 27B on Apple Silicon.   submitted by   /u/DerTomsn [link]   [comments] 26 r/LocalLLaMA community 22d ago NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090 I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share… 9 r/LocalLLaMA community 23d ago How to estimate tokens/sec for your hardware We all want more tokens per second but I keep seeing confusion on what to expect for given hardware. For the decoding phase (TG/s) to produce one token all the model weights and KV cache needs to be read from VRAM. The compute isn't the bottleneck, only memory bandwidth. This… 23 r/LocalLLaMA community 23d ago Updated my benchmark with a new vLLM based recipe for Qwen 3.8 Flash Next : now up to 98/100 (instead of 91 previously) I was using: weights https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 with the optimized SGLANG (patched) from https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_on_1x_rtx_pro_6000_171_ts_c1_428/ Now I'm using: weights (AWQ W4A16) from:… 11 r/LocalLLaMA community 23d ago You can now run a 90M conversational LLM on the Sony PSP (hardware from 2004). Doesn't get more local than this. Github link: https://github.com/thatblend/LLMPSP I wanted to see what the PSP can theoretically handle and I got my answer - a 90M model is about the max it can do without atrocious inference speeds. It's running around 0.5 - 0.6 tokens per second, which is very slow, but it's… 6 Hugging Face Daily Papers research 24d ago Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration Abstract CORD is a post-fit adapter that repairs calibrated probability vectors to exactly preserve original top-1 predictions while maintaining calibration quality. Generated by thinkingmachines/Inkling-Small Post-hoc calibration corrects reported confidence, yet a multiclass… 14 arXiv — NLP / Computation & Language research 24d ago Privacy-Preserving Heterogeneous Multi-LLM Federated Inference for Cognitive Diagnosis arXiv:2609.02947v1 Announce Type: cross Abstract: Significant challenges remain in AI-driven educational systems in balancing privacy preservation with accurate cognitive diagnosis. To overcome this, we propose a federated inference framework in which several commercial LLM APIs… 36 arXiv — Machine Learning research 24d ago Privacy-Preserving Topology-Guided Safety for LLM-Based Multi-Agent Systems via Federated Graph Learning arXiv:2609.02967v1 Announce Type: cross Abstract: Topology-guided safeguards for LLM-based multi-agent systems (MAS) train a GNN over the inter-agent communication graph to localize risky agents and intervene on the topology---but they assume one operator can pool all labeled… 11 arXiv — Machine Learning research 24d ago Privacy Leakage in Federated Learning: Gradient-Based Client Identity Inference and Defenses for Inertial Sensing in Vehicular Edge Networks arXiv:2609.02971v1 Announce Type: cross Abstract: As vehicular networks move toward 5G/6G edge intelligence, federated learning (FL) is widely promoted as a privacy-preserving way for vehicles and infrastructure to train shared models without exposing raw sensor data. Yet the… 22 arXiv — NLP / Computation & Language research 24d ago Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations arXiv:2609.03511v1 Announce Type: new Abstract: Large Language Models (LLMs) demonstrate strong multilingual reasoning performance, yet their robustness to semantics-preserving structural variation remains underexplored, particularly for relatively free word-order languages. We… 4 r/LocalLLaMA community 24d ago Qwen 3.8 27B Vs. Qwen 3.6 27B on oMLX Quality: 81.1 → 87.7 (+8%) Speed: 35 → 29 tok/s (−16%) Runtime: 8m51s → 44m39s (5x longer) Output tokens: 18K → 78K (🤯) Noticeably better quality, but you're paying for it with tokens and time. Full benchmark results (all hardware, all quants): llm-bench.io Qwen3.6-27B Vs.… 28 r/LocalLLaMA community 24d ago Qwen3.8-Flash-Next MTP merged in ik_llama.cpp (integrated head or separate -md file)... 45 → 90 tok/s on a 5090 + 128GB, works down to a 12GB 4070 ik_llama.cpp merged qwen4exp MTP support yesterday (PR #2369, mine, reviewed and tested by four other people on their own hardware). It's on main now, no fork or patch needed. Posting since the last couple threads had people saying MTP for this model only exists as an unsloth… 34 r/LocalLLaMA community 25d ago My RULE of Thumb of choosing a models This is mostly for setting up for expectation, since personally without LLM i could take 3 days (15 hours of active programming) to debug or implement a feature, but with Qwen 27B (even before Qwen 3.8) it take 4 hours. And yes 0.5 tok/s is human, not accounting of deletion and… 37 arXiv — Machine Learning research 25d ago OutageDiT: A Generative Foundation Model for Power Outage Forecasting and Scenario Simulation arXiv:2609.01896v1 Announce Type: new Abstract: Power-outage planning requires scenarios before an event occurs. These scenarios must represent uncertainty in magnitude, timing, and duration while preserving temporal dependence. However, severe events are rare, and data from any… 14 arXiv — Machine Learning research 25d ago Source-Free Class Relearning: Diagnosing Forgetting in Class Unlearning arXiv:2609.02018v1 Announce Type: new Abstract: Class unlearning aims to remove a model's ability to recognize designated forget classes while preserving performance on retain classes. However, low forget accuracy after unlearning does not necessarily mean the class structure… 16 arXiv — NLP / Computation & Language research 25d ago XMerge: Cross-Axis Selection and Reconstructive Layer Merging for LLM Depth Compression arXiv:2609.02083v1 Announce Type: cross Abstract: Removing complete transformer layers preserves a standard serving architecture, but existing depth-compression methods can lose substantial quality, and the loss varies unpredictably across models. We introduce XMerge, a… 10 arXiv — Machine Learning research 25d ago Exact Limits of Random Projections for Preserving Geometry: Distance Recovery, Nearest-Neighbor Rankings, and Covariance Shape in Gaussian Models arXiv:2609.02155v1 Announce Type: new Abstract: The Johnson-Lindenstrauss (JL) lemma guarantees that a random projection of $n$ points to $m=O(\varepsilon^{-2}\log n)$ dimensions preserves pairwise squared distances within relative error $\varepsilon$ with high probability, and… 16 arXiv — Machine Learning research 25d ago Similarity-Aware Personalized Federated Learning in Heterogeneous Environments arXiv:2609.02241v1 Announce Type: new Abstract: Federated Learning (FL) allows decentralized clients to train models collaboratively while preserving data privacy. However, distribution mismatch across clients often leads to poor global generalization and degraded local… 33 arXiv — Machine Learning research 25d ago Online Reinforcement Learning in the Met Office Unified Model through Distributed Model-Agent Coupling arXiv:2609.02566v1 Announce Type: new Abstract: Machine-learnt corrections can complement numerical weather prediction only if they adapt to the evolving model state while preserving dynamical consistency and numerical stability. To test this within a global forecasting model,… 24 arXiv — NLP / Computation & Language research 25d ago HyperStyler: Low-resource Authorship Style Transfer via Context-aware Style Navigation and Hypernetworks arXiv:2609.02772v1 Announce Type: new Abstract: Low-resource authorship style transfer (LAST) aims to rewrite text into the style of an arbitrary target author using only a few reference examples while preserving the original meaning. Existing methods often struggle to achieve… 33 Hugging Face Daily Papers research 25d ago Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation Abstract Influence-Directed Adaptive On-Policy Distillation improves diversity transfer in reasoning model distillation by selectively preserving entropy-expanding updates and replacing entropy-contracting ones with adaptive advantage shrinkage. Generated by… 23 r/LocalLLaMA community 25d ago Quad R9700 AI Pro with vLLM-Radiance easily reaching 17,6k PP https://preview.redd.it/74bmvel9b5nh1.png?width=1602&format=png&auto=webp&s=0d0c1adaa016a486ffd97c4c466e980dc611b139 I've only recently started looking deeper into vLLM after running llama.cpp for a good while. Initially vLLM (official repo) was terribly slow on my four R9700s… 10 Hugging Face Daily Papers research 26d ago From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix Abstract A smaller self-hosted LLM trained with separate GRPO experts merged via SLERP outperforms a much larger baseline on instruction following, function-calling, and internal tasks while serving half of platform traffic at lower cost. Generated by… 31 arXiv — Machine Learning research 26d ago QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization arXiv:2609.00224v1 Announce Type: new Abstract: Weight-only post-training quantization (PTQ) can alleviate the computational burden of serving large language models (LLMs) at scale. However, existing PTQ methods often fail to generalize across models and suffer severe accuracy… 36 arXiv — Machine Learning research 26d ago Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration arXiv:2609.01072v1 Announce Type: new Abstract: Post-hoc calibration corrects reported confidence, yet a multiclass calibrator can also change the associated top-1 prediction. Accuracy captures only the net effect of these changes on correctness, not how often predictions… 6 arXiv — Machine Learning research 26d ago Bandits in Prod: Hyperparameter Optimization at Inference Time arXiv:2609.01335v1 Announce Type: new Abstract: Many production systems can assess a configuration only by using it on live requests and observing noisy feedback. Modern agentic systems are a prominent example, with inference-time choices such as model selection, retrieval… 35 arXiv — NLP / Computation & Language research 26d ago OUTLETS: Output-Length Prediction from Speculative Decoding Backbones arXiv:2609.01068v1 Announce Type: new Abstract: The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing… 17 r/LocalLLaMA community 26d ago Running 104GB Qwen3.8-Flash-Next on 48GB Mac at ~12 tok/s   submitted by   /u/yogthos [link]   [comments] 34 r/LocalLLaMA community 26d ago 2/5 of my CMP 170HX have died after 2 weeks and the 3rd came with defective tensor cores. Current prices DO NOT justify the risk you are taking One GPU drops off immediately when vllm is started and the other throws CUDA errors on start   submitted by   /u/cantgetthistowork [link]   [comments] 22 r/LocalLLaMA community 26d ago Qwen3.8-Flash-Next (104 GB MoE) on a Strix Halo + RTX 3090 Ti eGPU: 22 -> 84 tok/s, and within one HumanEval+ problem of a dual-3090 vLLM box at 0.4x the wall time Follow-up to my Qwen3.8-27B post. This time the target is Qwen3.8-Flash-Next: 512 experts per layer, 36 layers of gated DeltaNet, 12 layers of top-k sparse attention, a 26.8 GiB n-gram table and a built-in MTP draft head. unsloth UD-Q4_K_XL, 103.69 GiB. It fits in the Strix… 30 r/LocalLLaMA community 26d ago Gave a try to Exllamav3 and it's great! Following this post I decided to try GLM 5.3 Flash on a 8x3090 setup and I can now run a Q4 with surprising speed; 700tk/s prefill & 42tk/s decoding! (lcp & vllm do not allow me to get that). Was afraid about quality but > 30m tokens with DSH and no issue (did not test vision… 36 r/LocalLLaMA community 26d ago How I got 280 tok/s on Qwen3.8 27B on 2xr9700's and 940k tokens kv cache 2 Months ago I had made a post how I was working on my dual R9700's. It's wild to look back at where we were then and where things now stand. Since then after many users commenting and complaining about developers doing the same thing. I threw out a discord link and expected… 25 Hugging Face Daily Papers research 26d ago RECAP-Forcing: Retaining Content Appearances for Long Video Generation Abstract RECAP-Forcing improves long video generation by indexing memory according to appearance novelty rather than recency, preserving key-value caches for newly visible content to maintain long-range consistency without extra training. Generated by… 6 r/LocalLLaMA community 26d ago Question: Why is prefill unbelievably faster in vLLM than other inference engines? I only started using some vLLM forks recently in a 4 x 48GB 4090 system. DS4F - ~5000pp/180tg (DSpark) Qwen3.8 Flash next - ~7500pp/135tg (MTP) This is amazing, like having the API in my house. But it's also really hard to go back. It's weird that we never come close to prefill… 27 Hacker News — AI on Front Page community 26d ago Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s I built slotstream, a way to run Qwen3.8-Flash-Next 4-bit on a low-memory mac starting from 16GB, a 125B parameter model that would need 100GB+ memory/RAM, thanks to expert-offloading/ssd-streaming. Easy to install/update, and mac-native using MLX and Swift. It ships with… 34 Hugging Face Daily Papers research 27d ago Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation Abstract KATok is an adaptive transformer-based video tokenizer that selectively drops uninformative tokens to achieve data-dependent compression while preserving spatial consistency for diffusion-based generation. Generated by thinkingmachines/Inkling-Small Latent diffusion… 10 arXiv — Machine Learning research 27d ago MEL: Coordinate-Preserving EEG Tokenization for fMRI Translation arXiv:2608.29304v1 Announce Type: new Abstract: Translating electroencephalography (EEG) into functional magnetic resonance imaging (fMRI) is important for medical neuroimaging, clinical brain-state monitoring, and multimodal neural decoding, because it aims to infer spatially… 34 arXiv — Machine Learning research 27d ago Creation begins with understanding: LLMs as strategy designers for privacy-preserving tabular data synthesis arXiv:2608.29674v1 Announce Type: new Abstract: Sharing tabular data in high-stakes domains is constrained by privacy regulations. Synthetic data offer a promising alternative, but deep generative models are costly to train and difficult to audit, while LLM-based methods often… 28 arXiv — Machine Learning research 27d ago INTERVenE: Temporal-Abstraction-Interval Based Transformers for Short-Horizon Medical Event Prediction arXiv:2608.29901v1 Announce Type: new Abstract: Electronic Health Record (EHR) prediction models in the intensive care unit must learn from sparse and irregular measurements while preserving the clinical meaning of time and supporting transparent decision-making. We present… 21 arXiv — NLP / Computation & Language research 27d ago When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization arXiv:2608.29327v1 Announce Type: new Abstract: Large language models deployed in specialized domains must improve in-domain performance without sacrificing general capabilities. Existing parameter-efficient fine-tuning methods are typically always on: their learned… 38 r/LocalLLaMA community 27d ago Qwen3.8-Flash-Next in llama.cpp from CPU-only to 96GB VRAM: 8.5 to 109 tok/s, max context and parameters test. My findings on RTX 6000 PRO. Hey guys, I tested Qwen3.8 Flash with llama.cpp from CPU-only to the full 96GB of my RTX PRO 6000. Short version: CPU-only reached 8.34 tok/s at a 2K prompt Full 96GB reached 109.07 tok/s At 245K context, 24GB to 96GB gave 14.89 to 21.61 tok/s The 96GB advantage over 24GB… 6 Hugging Face Daily Papers research 28d ago GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models Abstract GGSS reduces demographic bias in generative vision-language models by steering visual tokens along geodesic arcs with an adaptive gate, preserving visual-language accuracy. Generated by thinkingmachines/Inkling-Small Generative vision-language models (VLMs) are… 19 OpenAI official-blog 28d ago OpenAI supports California’s bill to advance youth AI safety OpenAI supports California SB 1119, advancing strong, age-appropriate AI safeguards for teens while preserving opportunities to learn, create, and explore. 28 arXiv — Machine Learning research 28d ago The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension arXiv:2608.28150v1 Announce Type: new Abstract: Which geometry controls the rank complexity of normalized softmax attention? We study maximum-row-$\ell_1$ approximation rank, exactly the least unrestricted rank preserving every bounded vector-valued output. Two sharp worst-case… 19 arXiv — Machine Learning research 28d ago Performative Privacy: When Differential Privacy Maximizes Utility arXiv:2608.28198v1 Announce Type: new Abstract: Privacy-preserving learning is often motivated by the idea that protecting users' data can preserve trust and thus participation, improving utility in the long term. However, this claim has not been formalized so far. In parallel,… 21 arXiv — Machine Learning research 28d ago Not to Break, but to Attest: Adversarial Probes for Privacy-Preserving LLM Verification arXiv:2608.27954v1 Announce Type: cross Abstract: Post-deployment changes to large language models can alter behavior while leaving routine outputs largely unchanged, creating a challenge for AI governance when model weights are proprietary. We present a privacy-preserving… 20 Page 4 of 10 · 500 articles ← Newer Older →
