News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow r/LocalLLaMA community 19d ago How much are you actually using your local models these days? Which ones do you reach for the most? I started tracking my local model usage about four weeks ago and was wondering if anyone else here keeps track of how much they use them. I’ve also been running some tests with the cheapest SOTA open-weight Chinese models via OpenRouter. Apart from that, I’m mainly using GPT-5.5… 4 r/LocalLLaMA community 20d ago Can LLMs solve mazes? https://reddit.com/link/1v5rvuq/video/bgmwc754i9fh1/player My goal was to create a benchmark to measure the spatial awareness and memory of models. Eventually, I came up with the simple idea of a maze where the model must find a key and use it to open the escape door. Here’s the… 22 r/LocalLLaMA community 20d ago Getting the most out of MTP If you want to get the most out of MTP. You have to run some tests / benchmarks to do so. Turning it on with defaults will get improvements, but for many models and card combinations, you are leaving a lot of performance on the table if you don't tune n_max. Can be easily… 37 Hugging Face Daily Papers research 20d ago Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction Abstract We introduce Tencent WorkBuddy Bench, a multi-domain evaluation suite for coding agents; this report documents its construction methodology, scoring protocol, and a cross-model leaderboard. At its core is a unified evaluation framework for constructing and running… 30 Hacker News — AI on Front Page community 20d ago Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard Article URL: https://artificialanalysis.ai/models Comments URL: https://news.ycombinator.com/item?id=49040741 Points: 220 # Comments: 130 26 r/LocalLLaMA community 20d ago How Laguna team even passed any benchmark? Im not telling this model good or bad. Im just wondering how they passed benchmarks if their templates and many other things was broken? And it took some time to fix that (so they haven’t had the right one laying around I suppose?) not just they uploaded “wrong” files Maybe… 30 r/LocalLLaMA community 20d ago Spent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet. I've been building a C99 inference engine from scratch (no Python, no BLAS, just gcc and make) that runs BitNet's ternary models on CPU. A few weeks ago I got obsessed with the matmul kernel - wrote a new one using AVX-512BW's vpermt2w to pack 5 ternary weights per byte instead… 22 r/MachineLearning community 21d ago I built an open-source multi-agent SDLC harness that beats a cold Claude Code run on large repos, by learning the repo once. Real benchmarks (incl. where it loses) inside. [P] Built an open-source AI coding agent that was 7%–75% cheaper than a cold "claude -p" run on 6/6 well-localized tasks across repositories up to ~82k LOC. The biggest difference: Cold agent: $6.83, 207 turns AutoDev Studio: ~$1.70 for the same bug The full benchmark (including… 14 Hugging Face Daily Papers research 21d ago FinanceComplexQA: Benchmarking Agentic Reasoning on Industrial-grade Financial Documents Abstract Agentic Reasoning has become a transformative force in financial analysis due to its ability to integrate large-scale information and generate reliable and accurate content. However, when handling complex real-world problems, different agents still show significant… 12 Hugging Face Daily Papers research 21d ago K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs Abstract Large language models are increasingly used in K-12 education, but existing benchmarks mainly test exam question answering rather than understanding how curriculum knowledge is structured and visually presented. We call this capability curriculum cognition. It covers… 30 Smol AI News news-outlet 21d ago Opus 5 **Anthropic** launched the **Claude Opus 5** model, which sparked mixed reactions including benchmark scrutiny and praise for its coding-agent capabilities. The model achieved an **Epoch Capabilities Index (ECI) of 159**, slightly below **Fable 5's 161**, but matched Fable 5 on… 36 arXiv — NLP / Computation & Language research 21d ago DataPrep-Bench: Benchmarking LLMs as Training Data Preparators arXiv:2607.20465v1 Announce Type: cross Abstract: The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data… 12 arXiv — Machine Learning research 21d ago StabilityBench: Benchmarking Instability in LLMs arXiv:2607.20558v1 Announce Type: new Abstract: AI Assistants are increasingly deployed in high-stakes settings, such as healthcare or government services. Yet their real-world behavior remains poorly understood due to strong context dependence. Current evaluation protocols… 6 arXiv — NLP / Computation & Language research 21d ago Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks arXiv:2607.20864v1 Announce Type: cross Abstract: Position bias in multiple-choice LLM evaluation is widely cited as a confound in capability comparisons, but published measurements rely on single answer-order shuffles whose results confound the bias signal with content-level… 35 arXiv — Machine Learning research 21d ago From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python arXiv:2607.21069v1 Announce Type: new Abstract: The original ALPHA benchmark introduced a taxonomy-aware penalty for evaluating CWE-level vulnerability prediction in Python and proposed that the penalty could theoretically also serve as a training signal. This paper provides… 31 arXiv — Machine Learning research 21d ago TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning arXiv:2607.21111v1 Announce Type: new Abstract: Offline Reinforcement Learning (RL) agents are trained on fixed behavioral trajectories, which makes trajectory-level deletion important when selected data must be removed after training. Evaluating such deletion is difficult… 30 arXiv — Machine Learning research 21d ago GlucoTune: A Unified Framework for Blood Glucose Preprocessing, Forecasting, and Benchmarking in Diabetes arXiv:2607.21117v1 Announce Type: new Abstract: Preprocessing blood glucose time-series data is a critical yet often overlooked step in developing data-driven methods for diabetes management, particularly for type 1 diabetes. The lack of standardized preprocessing workflows and… 30 arXiv — Machine Learning research 21d ago Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearning arXiv:2607.21353v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of specific training data while preserving model utility. Many state-of-the-art approaches pursue this goal by restricting the forgetting update to a subset of parameters selected… 38 arXiv — NLP / Computation & Language research 21d ago Making Open-Source Text LLM Watermarks Durable Against Merging arXiv:2607.20435v1 Announce Type: new Abstract: Open-source LLMs (OSMs)arereaching near state-of-the-art performance, prompting prior works to trace the text they generate by embedding text watermarking algorithms directly into their weights. Yet, OSMs are subject to… 5 arXiv — NLP / Computation & Language research 21d ago Frontier Financial Judgement: Can agents tell what might move a stock? arXiv:2607.20645v1 Announce Type: new Abstract: We introduce Frontier Financial Judgement, a challenging new benchmark developed in collaboration with professional equity analysts to assess agents' ability to replicate expert human judgements. Rapidly identifying new… 34 arXiv — NLP / Computation & Language research 21d ago Rushes: A Human Preference Dataset for Pluralistic Alignment arXiv:2607.20767v1 Announce Type: new Abstract: We introduce Rushes, a dataset and benchmark for studying revealed human engagement preferences in interactive narrative environments. Rushes is collected through a game interface where users interact with AI-generated branching… 15 arXiv — NLP / Computation & Language research 21d ago LegalCiteTrust: Benchmarking Citation Trustworthiness in Chinese Long-Form Legal Research Reports arXiv:2607.20872v1 Announce Type: new Abstract: Long-form legal research reports increasingly rely on LLMs and agentic research systems, but their reliability depends not only on answering the task, but also on whether cited legal authorities are trustworthy. A citation can be… 33 arXiv — NLP / Computation & Language research 21d ago Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction arXiv:2607.20911v1 Announce Type: new Abstract: We introduce Tencent WorkBuddy Bench, a multi-domain evaluation suite for coding agents; this report documents its construction methodology, scoring protocol, and a cross-model leaderboard. At its core is a unified evaluation… 5 arXiv — NLP / Computation & Language research 21d ago CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages arXiv:2607.21016v1 Announce Type: new Abstract: Culture is lived through conversation, yet existing Indonesian cultural commonsense benchmarks evaluate LLMs on short and isolated prompts, stripping away the dialogic context in which cultural nuances actually surface. We… 5 arXiv — NLP / Computation & Language research 21d ago QuantiBias: Benchmarking Quantization-Induced Bias in LLMs arXiv:2607.21063v1 Announce Type: new Abstract: Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is assumed harmless and its safety is rarely re-checked. We find its principal side… 20 arXiv — NLP / Computation & Language research 21d ago One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies arXiv:2607.21143v1 Announce Type: new Abstract: Ambiguous user requests make clarification a sequential decision problem for conversational LLM assistants: they must decide whether to ask, what to ask, when to stop, and when to answer. We introduce RegretBench, a multi-turn… 25 arXiv — NLP / Computation & Language research 21d ago A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset arXiv:2607.21274v1 Announce Type: new Abstract: We present CUP, a Greek book retrieval benchmark consisting of 868 catalog records and 104 expert-annotated queries with graded relevance judgments. We evaluate sparse (BM25), dense (sentence-transformers), hybrid, and LLM-assisted… 20 arXiv — NLP / Computation & Language research 21d ago RUMBA: Russian User Memory Benchmark arXiv:2607.21447v1 Announce Type: new Abstract: The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context,… 32 arXiv — NLP / Computation & Language research 21d ago Benchmarking Large Language Models on Multi-Sensor Physical Hazard Assessment arXiv:2607.20476v1 Announce Type: cross Abstract: We present an empirical benchmark evaluating how five large language models assess multisensor physical hazard data. Testing 60 scenarios across three categories - multi-sensor joint assessment, response proportionality, and… 17 arXiv — NLP / Computation & Language research 21d ago PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails arXiv:2607.20482v1 Announce Type: cross Abstract: Recent advances in large language models have enabled web agents to autonomously execute complex tasks. In practice, users frequently provide underspecified instructions, requiring agents to infer the missing context from their… 22 arXiv — NLP / Computation & Language research 21d ago Expectation Alignment of Language Models for Real-World User Expectations arXiv:2607.20485v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated remarkable performance on standard benchmarks, yet it remains largely unexplored whether they truly meet user expectations. Existing evaluation approaches, relying on model… 34 arXiv — NLP / Computation & Language research 21d ago DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making arXiv:2607.20491v1 Announce Type: cross Abstract: Standard evaluation benchmarks measure what a tool-using agent decides, not whether it arrives at that decision through the same process each time. We introduce DFAH-Bench, a replay benchmark that measures observable behavioral… 22 arXiv — NLP / Computation & Language research 21d ago Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domain arXiv:2607.20510v1 Announce Type: cross Abstract: We introduce Telco-GAIA, a bilingual, multi-modal benchmark for evaluating tool-using agents on the data of a real-world telecommunications operator. Telco-GAIA comprises 100 human-verified question-answering tasks, in English… 20 arXiv — NLP / Computation & Language research 21d ago AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use arXiv:2607.20536v1 Announce Type: cross Abstract: Tool-use agents that address day-to-day digital tasks such as ordering groceries must not only operate applications, but also interact with the user, e.g., to ask clarification questions, prompt for confirmation, and inform the… 36 arXiv — NLP / Computation & Language research 21d ago WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms arXiv:2607.20638v1 Announce Type: cross Abstract: Large Language Models (LLMs) have demonstrated strong capabilities in code generation and reasoning, yet their ability to perform temporal reasoning over digital waveform data remains largely unexplored. Although reasoning over… 18 r/MachineLearning community 21d ago GPT-5.5 Scores 10.6% on ActiveVision, Humans Hit 96.1% [R] The interesting finding from a new [arXiv paper]( https://arxiv.org/abs/2607.16165 ) isn't that a frontier vision model failed a new benchmark, that happens weekly, but the specific shape of the failure and the fact that the models cannot patch it by writing their own code. The… 25 r/LocalLLaMA community 21d ago Deepseek V4 Flash ~105 t/s on two Nvidia 4090d 48G (ada) in vLLM TLDR: I (with the help of AI) re-implemented every Blackwell-only kernel (DeepGEMM, FlashInfer sparse-MLA, block-scaled FP8) in Triton, because they simply don't exist for sm89. The performance is 2-3x more for parallel agentic workflows. Benchmark llama-server vs vLLM I was… 27 r/LocalLLaMA community 21d ago Benchmarks: AntLing-3.0-flash a hybrid-reasoning MoE model built for production-scale agents. Now live on OpenRouter, and free to use through August 3, 2026. Hoping they will going openweight soon~   submitted by   /u/niacolhealth [link]   [comments] 28 r/LocalLLaMA community 21d ago CPU-only inference on a Celeron N5095 SBC: 6 models from 0.6B to 8B, benchmarked I wanted to know how cheap you can go and still run local models, so I ran Ollama CPU-only on a Youyeetoo X1S. It's a single-board x86 machine with a Celeron N5095 (Jasper Lake, 4C/4T, 15W), 16GB of RAM, and a 128GB NVMe, running Kali 2025.4. Base configs of this board go for… 28 Hugging Face Daily Papers research 22d ago DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations Abstract As autonomous agents rapidly evolve, their ability to reliably manipulate ubiquitous digital documents has become critical for enabling general-purpose AI assistants and automating complex workspace workflows. In this paper, we introduce DocOps, a deterministically… 30 r/LocalLLaMA community 22d ago A caveman qwen3.6 27B Just saw this on huggingface: https://huggingface.co/ProCreations/grug-27b The benchmarks claim that it's quite a bit better than qwen3.6 27B original and that they reduced the amount of necessary tokens by more than 90%. It would make 27B running on my old laptop at 3tps feel… 25 arXiv — Machine Learning research 22d ago CruiseBench: A Real-Flight-Aligned N-CMAPSS Benchmark for Engine RUL Prediction arXiv:2607.19380v1 Announce Type: new Abstract: Remaining useful life (RUL) prediction estimates how long an engine can continue safe operation and is central to maintenance planning. N-CMAPSS extends C-MAPSS by simulating run-to-failure aero-engine trajectories using recorded… 29 arXiv — Machine Learning research 22d ago Air Quality Arena: A Large-Scale Multi-Region Ground Monitoring Dataset and Benchmark for Air Quality Forecasting with Time-Series Foundation Models arXiv:2607.19381v1 Announce Type: new Abstract: Air pollution causes an estimated 7.9 million premature deaths annually, making accurate forecasting a critical public health priority. Machine learning is increasingly being applied to forecast air pollution levels, yet existing… 16 arXiv — NLP / Computation & Language research 22d ago Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks arXiv:2607.19393v1 Announce Type: cross Abstract: While auditing a perturbation-based OOD detector on a document benchmark, we recorded an AUROC of 0.326 -- well below the 0.5 chance level. The cause is a benchmark leak: the designated "OOD" class is one the model was trained… 19 arXiv — NLP / Computation & Language research 22d ago Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models arXiv:2607.19847v1 Announce Type: cross Abstract: Predicting missing cell values in tabular data is a fundamental problem in data cleaning. While state-of-the-art reasoning models show great promise in predicting missing values in tables, by reasoning holistically across rows… 27 arXiv — Machine Learning research 22d ago Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX arXiv:2607.19353v1 Announce Type: cross Abstract: Confidential computing is becoming a practical deployment requirement for AI inference workloads that process sensitive inputs or protect proprietary model assets. However, the performance cost of enabling confidential execution… 26 arXiv — Machine Learning research 22d ago Trend strength predicts when generative foundation models win: a power-controlled benchmark, a mechanism, and an actionable selection rule arXiv:2607.19383v1 Announce Type: cross Abstract: Pretrained generative foundation models cast forecasting as conditional generation from a learned predictive distribution and forecast unseen series zero-shot. We establish three results that turn their reported success into an… 8 arXiv — NLP / Computation & Language research 22d ago On the Computational Complexity of Structural Generalization arXiv:2607.19573v1 Announce Type: new Abstract: Structural generalization has been measured repeatedly by several benchmarks, yet it has never been formally defined. We give a definition that translates the two premises (compositional structure and unbounded generalization) into… 7 arXiv — NLP / Computation & Language research 22d ago D2VBench: Benchmarking Large Language Models with Value Dilemmas in Daily Scenarios arXiv:2607.19834v1 Announce Type: new Abstract: With the wide application of large language models (LLMs) in real-world scenarios, the value implication of their outputs is crucial. However, existing evaluation benchmarks suffer from insufficient coverage of value dilemmas in… 34 arXiv — NLP / Computation & Language research 22d ago When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization arXiv:2607.19956v1 Announce Type: new Abstract: Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined. On the BanSum Bangla summarization benchmark, we find that standard KD improves ROUGE-L… 28 Page 9 of 10 · 500 articles ← Newer Older →
