News / #gpu Tag Gpu 500 articles archived under #gpu · RSS Sign in to follow Hugging Face Daily Papers research 10d ago GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning Abstract Optimization-based latent reasoning improves large language model outputs by optimizing instance-specific continuous states at test time while keeping model parameters frozen. Existing methods, however, typically connect these states to the reasoning trajectory through… 8 Smol AI News news-outlet 10d ago not much happened today **Alibaba** launched **Qwen3.8-Max**, enhancing multimodal capabilities and agent ecosystem integration. **NVIDIA** introduced **Alpamayo 2 Super** for autonomous vehicle reasoning, while **Mistral AI** released **Shieldstral**, a 3B parameter open-weights safety model for… 17 arXiv — Machine Learning research 10d ago Caliber: Cross-Architecture Extraction-Cost Control for Score-Returning APIs arXiv:2608.01023v1 Announce Type: new Abstract: We present Caliber, an output-perturbation defense against model extraction that formulates noise selection as a calibration problem: how much the defense degrades the supervision signal used to train a surrogate, and the provable… 6 arXiv — Machine Learning research 10d ago Meganeura: Portable GPU Training and Inference through Vulkan and Metal arXiv:2608.01563v1 Announce Type: new Abstract: Training and deployed inference often cross export, conversion, and platform-specific runtime boundaries. Meganeura asks whether one compact native compiler can span both phases on consumer GPUs. Its typed static graph, automatic… 29 arXiv — NLP / Computation & Language research 10d ago Writing-System-Level Tokenizer Adaptation for Byte-Level BPE arXiv:2608.00582v1 Announce Type: new Abstract: Pretrained byte-level BPE tokenizers can segment underrepresented languages inefficiently. Replacing a tokenizer changes the meaning of nearly every token ID, while vocabulary expansion enlarges the model's embedding and output… 6 arXiv — NLP / Computation & Language research 10d ago TreeProbe : A Tibetan Medicine Benchmark for Cultural Bias in LLMs arXiv:2608.00640v1 Announce Type: new Abstract: Large language models are increasingly viewed as a potential means of mitigating global health inequities, yet their outputs often reflect dominant high-resource medical traditions and provide limited coverage of traditional… 38 arXiv — NLP / Computation & Language research 10d ago Understanding Sparse Attention Selectivity in Long-Context Foundation Models via Counterfactual Evaluation arXiv:2608.01676v1 Announce Type: new Abstract: Sparse attention is widely deployed in long-context serving stacks, yet no framework audits how discarding blocks changes the influence of specific content on model output. We first establish that the phenomenon is real and causal:… 18 Simon Willison community 10d ago Don't be a meat proxy Don't be a meat proxy Niklas Gruhn coins an excellent new term - meat proxy - for people who blindly copy and paste the output of AI systems to their peers. By all means, prompt AI. But don't just relay the output. Read it, understand it, validate it, and then write a… 17 r/LocalLLaMA community 10d ago NousResearch keeps doing things on hermes Has anyone followed nousresearch work on Hermes? I mean we are Q3 2026. We have some crazy models trickling down from HGX territory to multi gpu workstation. And we have nousresearch deploying the 0.20 of its hermes agent while starting releasing the project with a 0.2 mid… 13 r/LocalLLaMA community 10d ago nvidia/NVIDIA-NemotronLabs-VoiceChat-11B · Hugging Face (full duplex)   submitted by   /u/adefa [link]   [comments] 9 r/LocalLLaMA community 10d ago Thermal paste PSA for old GPUs I know many of us are using older GPUs like the 3090 because they work great. I just replaced the thermal paste and am seeing consistently 10 C lower temperatures. The old paste was cracking and like dry dust when I removed it. This made the difference between super loud fans… 8 llama.cpp releases dev-tools 10d ago b10242 CUDA: Add backend sampler for penalties sampler ( #25262 ) sampling: enhance penalty handling in common_sampler_init Set default value for penalty_last_n based on model context if not specified. Ensure penalty_last_n and n_prev are non-negative. Update llama_sampler_penalties… 20 r/MachineLearning community 10d ago It's time to desk reject papers that don't include code that can reproduce the results [D] As review season for NeurIPS wraps up, I have now reviewed for 3 major conferences this year. And I'm noticing a worrying trend: Out of the 12 papers I reviewed this year, only 1 provided full code (that runs the whole training pipeline from input dataset to output AUROC). 4… 28 NVIDIA Developer Blog official-blog 10d ago How to Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure Running a dedicated Kubernetes cluster per team often results in more isolation than an organization requires. While one cluster can be successfully shared... 9 NVIDIA Developer Blog official-blog 10d ago NVIDIA Vera Storage Benchmarks: Faster Encryption, Compression, Integrity Checking, and Recovery for AI-Native Storage Storage is an active part of every agentic AI workflow. As agents retrieve enterprise knowledge, access persistent memory, reuse key-value (KV) cache data,... 17 r/LocalLLaMA community 10d ago All DeepSeek model oneshots: 242 outputs to look at and compare! Continuing my weekend of oneshotting the cheap OpenRouter models, here are all 10 DeepSeek models across the same 35 prompts. DeepSeek had a rougher time (more provider errors / empty completions), so only 242 made it out of the 10*35 matrix. Here they are… 31 llama.cpp releases dev-tools 10d ago b10241 CUDA: Fix data-races when reusing SMEM in block_reduce ( #26385 ) CUDA: Fix data-races when reusing block_reduce block_reduce currently doesn't resync after reading from SMEM, causing potential data-races when reusing SMEM for multiple reductions. One may consider simply always… 24 r/LocalLLaMA community 10d ago I compared MinerU, Granite-Docling, and PaddleOCR-VL on 12 PDF-parsing capabilities using 6 document types I tested them by sending the 6 documents, each meant to represent a different document type, through my own webapp and comparing every output against the source. All ran on the same L4 GPU. The documents: Financial statements with merged multi-level headers (A typical annual… 37 Hacker News — AI on Front Page community 11d ago AirLLM 70B inference with single 4GB GPU Article URL: https://github.com/lyogavin/airllm Comments URL: https://news.ycombinator.com/item?id=49154228 Points: 203 # Comments: 76 20 Hugging Face Daily Papers research 11d ago ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction Abstract Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a… 4 r/LocalLLaMA community 11d ago Seedance 2.5 Vs Minimax H3 (Open Weight). Excellent Output Comparison!   submitted by   /u/Hannibalj2ca [link]   [comments] 23 arXiv — Machine Learning research 11d ago Topology-Aware Data Movement for Disaggregated GPU Inference arXiv:2607.28633v1 Announce Type: new Abstract: Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly. When prefill and decode run on separate GPU pools, the KV cache must be transferred between them. For a 70B model this is… 6 arXiv — Machine Learning research 11d ago Autonomous Repair for Multi-Agent Systems via Monte-Carlo Tree Search arXiv:2607.29055v1 Announce Type: new Abstract: Multi-agent systems (MAS) are increasingly deployed to solve complex tasks. In case of incorrect or unsatisfactory outputs, users have to manually locate agent mistakes by inspecting agent trajectories (i.e., {\em failure… 9 arXiv — NLP / Computation & Language research 11d ago How Hard Does It Think? Analyzing Step-Aware Reasoning Energy in LLM Chain-of-Thought Trajectories arXiv:2607.28674v1 Announce Type: cross Abstract: Understanding how computational effort is allocated across individual chain-of-thought (CoT) reasoning steps remains an open challenge: existing interpretability methods rely on output-level signals or collapse processing depth… 28 arXiv — Machine Learning research 11d ago DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs arXiv:2607.28848v1 Announce Type: cross Abstract: LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak. We present DeltaServe, a host-agnostic co-serving design that converts this… 20 arXiv — NLP / Computation & Language research 11d ago CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation arXiv:2607.29252v1 Announce Type: new Abstract: Reliable evaluation of open-ended LLM outputs requires fine-grained rubrics, yet expert curation is costly and difficult to scale. Existing automated pipelines rely on strict judge unanimity and binary variance filters, which… 19 r/LocalLLaMA community 11d ago China’s DFSX Offers 2x The Memory Bandwidth Of NVIDIA’s GB200   submitted by   /u/MundanePercentage674 [link]   [comments] 8 r/LocalLLaMA community 11d ago DeepSeek V4 @ IQ3XXS on M1 Ultra 128GB- 16 tok/s in LM Studio after patch M1 Ultra 128GB, Unsloth UD-IQ3_XXS, wired limit at 120GB. I was at 5-6 tok/s before the patch. Getting 15-16 tok/s now with the patched engine, and the output seems to have improved. Big thanks to this guy.   submitted by   /u/mil_phickelson [link]   [comments] 9 r/LocalLLaMA community 11d ago DSpark Benchmark Result on Deepseek v4 Flash 0731 TensorSharp supports DSpark on Deepseek v4 Flash 0731 now. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8 with/without DSpark: Model: DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF DSpark draft model from:… 31 r/LocalLLaMA community 11d ago All Qwen model oneshots: 1109 outputs to look at and compare! I've been busy this weekend generating oneshots for all the cheapest models on the openrouter and ended up going through all 33 qwen models across 35 prompts (there were some failures and only 1109 made out of 33*35 matrix). Here they are https://oneshotlm.com/model/?q=qwen Qwen… 35 r/LocalLLaMA community 11d ago Are you ready for Le Chaton FAT or still wasting money on GPUs? According to rumors (spread by myself) Le Chaton FAT will be 26T-a3b and I AM READY for it. Let's be real, I can't afford that many 5060Ti, so I got 12x Gen 4 3.2 TB (two per card). This gives me about 60GBs bandwidth on 30TB. Added 256gb ddr4 just for kv cache, but I can also… 8 r/LocalLLaMA community 11d ago Deepseek v4 flash - 100-150 faster t/s in prefill/pp. You have two choices here (in order of pref): Downgrade CUDA from 13.3 to 13.1 (skip 13.2 due to bugs) <- prefer this (thanks to u/fairydreaming for pointing this out) Use this vibed fork that works with CUDA 13.3 https://github.com/vektorprime/working_ds4_speed I was… 34 r/LocalLLaMA community 11d ago Deepseek-V4-Flash-0731 Dwarfstar on Mac Here is the prefill performance in an M2 Ultra with 192GB of RAM. For decode, at the following depth: Start: 28 t/s 45k: 23.5 t/s 192k: 18 t/s That speed is maintained with 8k token output at those depths.   submitted by   /u/Badger-Purple [link]   [comments] 17 llama.cpp releases dev-tools 12d ago b10226 sycl: fix classification of iGPUs ( #26105 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64… 14 llama.cpp releases dev-tools 12d ago b10224 ggml-webgpu: add support for f16 repeat ( #26307 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64… 15 r/LocalLLaMA community 12d ago DeepSeek-V4-Flash-0731 UD-IQ3_XXS about 11t/s on 1x 7900 XTX 24GB + 3x MI60 32GB + 128GB DDR4 Hello, Also I want to join the hype of posting token specs. CPU: 2x Intel Xeon CPU E5-2650 v4 @ 2.20GHz RAM: 2x 4 Channel 2400MHz DDR4 GPU: 1x AMD Radeon 7900 XTX 24GB 3x AMD Instinct MI60 32GB Strange GPU combination, right? One of my AMD Instinct MI60 32GB failed, and I have… 35 r/LocalLLaMA community 12d ago I pushed Kimi K3 onto one CPU with 8 GB of RAM I deployed K3 on 32 H100s at work a couple of weeks ago and then got annoyed that there was no way to poke at it on my own machine . So I wrote an inference engine for it in C99. Nothing clever going on. 93% of that 1.56 TB checkpoint is routed experts, and only 16 of 896 fire… 14 r/LocalLLaMA community 12d ago DeepSeek-V4-Flash-0731 UD-Q8_K_XL 17.20~ t/s on A6000 + 256GB DDR4 Hello everyone I want to join the hype of posting specs. CPU: AMD EPYC 74F3 24-Core RAM: 8 Channel 3200 DDR4 GPU: RTX A6000 48GB Prompt processing is in the high 70t/s (got down to mid 30t/s at 300k context). Inference is a steady 17.20t/s~ and the 48GB VRAM is enough to have… 10 r/LocalLLaMA community 12d ago Expert-only IQ3 requant of DeepSeek-V4-Flash-0731: better KLD than UD-IQ3_S, 1.4x decode on a CPU-spill rig Hey all, tldr / who this helps: you run a mixed multi-GPU box where the experts spill to RAM, and you want to stay in the 3-bit tier instead of dropping to Q2 to make it fit. https://huggingface.co/TacoTakumi/DeepSeek-V4-Flash-0731-GGUF I requantized only the 129 routed expert… 19 r/LocalLLaMA community 12d ago DeepSeek-V4-Flash-0731 on Bosgame M5 with RTX PRO 6000 Max-Q eGPU Here are my numbers: Quant Size Layout Decode Prefill Draft acceptance UD-Q8_K_XL 150.8 GiB 20 layers CUDA0 / 23 ROCm0 + drafter 44.0 t/s 564 t/s 0.535 UD-Q4_K_XL 144.4 GiB 22 / 21 + drafter 48.4 t/s 585 t/s 0.532 UD-Q2_K_XL 90.2 GiB entirely on CUDA0, no drafter 59.5 t/s 1513… 17 r/LocalLLaMA community 12d ago DeepSeek V4 Flash 0731 IQ2_M benchmark for Dual 3060 and 96GB RAM ≈ 3.5 tok/s. Thanks to the community help I finally launched this llm. LM Studio refused to load weight onto second GPU but Unsloth Studio did so everything was done in there. Not a proper benchmark (used PC in parallel as well) but it gives an idea of the performance from dual 3060 with… 16 r/LocalLLaMA community 12d ago I'm kinda tired of obsession for one-shot tests in coding, there are good tests for multi-step debugging with analyzing output/images/videos? Personally, i think good coding model shouldn't be focused on one-shot "everything in one html-file" tests, but should be really good on debugging, fixing and modifying its own output. Anyone know such simple tests that i would able to run with local models? May be some kind of… 15 r/MachineLearning community 12d ago Github repo to learn the OPD/OPSD and how they perform compared to GRPO, on a consumer grade GPU [P] I am trying to learn concepts like On Policy Distillation (OPD), On Policy Self Distillation (OPSD) and how do they compare to RL algorithms like GRPO. There are a lot of papers on this, but because of limited compute I cannot try these papers out and learn them by implementing… 18 r/LocalLLaMA community 13d ago Qwen 3.6 27B Q5 on 3x2080ti: 55tps with llama.cpp. Can I squeeze out more? CPU: Threadripper 3970X RAM: 128GB DDR4 GPUs: 3x2080ti 11GB The current best parameters to run it: llama-server \ --model Qwen3.6-27B-Q5_K_S.gguf \ --n-gpu-layers 999 \ --split-mode tensor \ --flash-attn on \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --ctx-size 16384 \… 8 r/LocalLLaMA community 13d ago 60-82% accuracy swing on 4B model classification task: the only variable was harness design I ran a pre-registered ablation on a classification task (Kubernetes issue → SIG triage) using a 4B model on a 6GB laptop GPU. Same frozen weights, same 250-issue gold corpus, same scorer across every run. The variable under test was harness design: rule placement, evidence… 19 r/LocalLLaMA community 13d ago Minimum VRAM GPU to run DeepSeek-V4-Flash-0731 Q4_K_XL at around 30 t/s ? Hello guys, I'm curious about running DeepSeek-V4-Flash-0731 locally. Since it’s a Mixture of Experts (MoE) model with only 13B active parameters, I was hoping the VRAM requirements might be manageable. Did someone tried out in some reasonable GPU sizes up to 48GB VRAM? Thanks… 37 llama.cpp releases dev-tools 13d ago b10209 cuda: extract Q2_0 elements via __byte_perm ( #25603 ) Website: https://llama.app macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu… 8 NVIDIA Developer Blog official-blog 13d ago NVIDIA Video Codec SDK 13.1: Zero-Copy Transcode, AV1 B-Frames, and Frame-Accurate Seek The demand for high-quality video continues to accelerate across industries, powering everything from immersive streaming experiences to remote collaboration,... 29 r/LocalLLaMA community 13d ago Meituan just dropped LongCat-Flash-Lite-Sparse It’s an MoE with ~3B active params and a 30B n-gram lookup table offloaded to RAM for fast 256k context on a 24GB GPU. Reminds me of Gemma 4’s PLE trick. Initial analysis suggest it wont be replacing my Qwen 3.6 27b.   submitted by   /u/Gohab2001 [link]   [comments] 16 Hugging Face Daily Papers research 14d ago ReToken: One Token to Improve Vision-Language Models for Visual Retrieval Abstract Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints. We present ReToken, a single learnable embedding… 22 Page 4 of 10 · 500 articles ← Newer Older →
