News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow arXiv — Machine Learning research 2d ago DACRI: Decision-Aware Causal Intervention Ranking for Critical Supply Chains arXiv:2608.11154v1 Announce Type: new Abstract: Detecting or attributing a supply-chain disruption is not the same as selecting the intervention that maximizes recoverable net value. We present CriticalSCM-Bench v1, a controlled synthetic benchmark with causal ground truth,… 35 arXiv — Machine Learning research 2d ago HyperShape: Hyperelasticity Across Diverse Shapes arXiv:2608.09938v1 Announce Type: cross Abstract: Hyperelastic deformations are highly sensitive to domain geometry and boundary conditions, making generalization across both a critical capability for neural operators applied to these problems. However, existing benchmarks for… 34 arXiv — Machine Learning research 2d ago Energy and Performance Benchmarking of Deep Learning Models for Breast Cancer Detection arXiv:2608.09996v1 Announce Type: cross Abstract: Recent advances in machine learning have greatly improved breast cancer detection, enabling more accurate and timely diagnosis. Deep learning (DL) models show strong potential for medical image analysis; however, as their… 25 arXiv — NLP / Computation & Language research 2d ago TAF-MED: Multi-Turn Safety Refusal Collapse in LLMs Under Declared Self-Treatment Intent arXiv:2608.10258v1 Announce Type: new Abstract: Large language models (LLMs) increasingly provide conversational health information that may influence treatment decisions, yet existing benchmarks do not isolate whether medication-safety boundaries persist across follow-ups after… 29 arXiv — NLP / Computation & Language research 2d ago Locally Deployable Small Language Models for Emergency Department Decision Support: A Systematic Benchmark of Fine-Tuning Strategies arXiv:2608.10273v1 Announce Type: new Abstract: Deploying large language models (LLMs) for decision support in emergency departments (EDs) faces two major challenges: privacy risks of transmitting patient data to closed-source commercial LLMs and the lack of systematic… 4 arXiv — NLP / Computation & Language research 2d ago Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases arXiv:2608.10503v1 Announce Type: new Abstract: As Large Language Models (LLMs) are increasingly deployed as autonomous agents, accurately evaluating their latent values and biases is critical. The NLP community typically evaluates models using large, unstructured benchmarks.… 21 arXiv — NLP / Computation & Language research 2d ago Surfacing the Unsaid: CUE-Bench for Affective Stance in Chinese Discourse arXiv:2608.10810v1 Announce Type: new Abstract: Emotion understanding in discourse requires reasoning beyond surface sentiment because speakers often convey affect through indirect, implicit, polite, ironic, or deliberately mismatched expressions. Existing emotion benchmarks… 5 arXiv — NLP / Computation & Language research 2d ago FaithformBench: Benchmarking Faithfulness of Mathematical Chain-of-Thought Autoformalisation arXiv:2608.10916v1 Announce Type: new Abstract: Autoformalisation (AF) systems map natural language reasoning steps into formal statements in a proof assistant such as Lean. We consider how to assess the faithfulness of these systems. Existing approaches require expensive… 37 arXiv — NLP / Computation & Language research 2d ago ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering arXiv:2608.10679v1 Announce Type: cross Abstract: Enterprise question answering is framed as retrieving internal documents and generating grounded answers. Routine enterprise records, however, are work by-products in which required organizational relations remain implicit across… 7 arXiv — NLP / Computation & Language research 2d ago HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models arXiv:2506.03922v4 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) have demonstrated significant potential to advance a broad range of domains. However, current benchmarks for evaluating MLLMs primarily emphasize general knowledge and vertical… 15 Hugging Face Daily Papers research 2d ago JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles Abstract A new jigsaw benchmark with interlocking pieces reveals that vision-language models fail at geometric reasoning and suffer a sharp performance drop as puzzle size increases. Generated by thinkingmachines/Inkling-Small Jigsaw puzzle solving requires jointly reasoning… 7 Hugging Face Daily Papers research 2d ago SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information Abstract SPIEval benchmarks mobile assistant LLMs on scattered personal data tasks, revealing major gaps in information retrieval and verification. Generated by thinkingmachines/Inkling-Small Large language models (LLMs) are increasingly deployed as mobile assistants, where a… 36 r/LocalLLaMA community 2d ago New Muse-Glimmer-30B SoTA Quants - hopefully a new lineup :) Hey Folks, I've been making quants for a while - recently I took a short break to get into hardcore research (submitted my first EMNLP paper during it!). Along the way, I built up a little arsenal of quant-optim techniques: everything from novel, paper-pending tricks to some… 19 Hugging Face Daily Papers research 2d ago VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World? Abstract A new benchmark called VibeLifeBench evaluates long-horizon proactive agents across simulated multi-week everyday tasks, revealing that current frontier models perform poorly. Generated by thinkingmachines/Inkling-Small Large language model (LLM) agents are increasingly… 22 r/LocalLLaMA community 2d ago We quantized DeepSeek V4 0731 and benchmarked it against popular quants on 8× RTX 5090 We converted the model from the original safetensors and found two issues. The first one made our quantization fail several times, the second one does not fail at all, it just quietly ruins the base 1) You must use the --no-lazy option, otherwise token_embd.weight will take on… 23 r/LocalLLaMA community 2d ago Local Benchmark : Muse Glimmer 30B vs Qwen 3.6 27B vs Gemma4 31B (and many other models and finetunes) Needs a lot of requests compared to Qwen (almost twice) and Gemma (almost x3). Final score is fine, even though it is "not a coding model" https://wonderrico.github.io/local_llm_benchmark/benchmark-main.html more details on… 31 Hugging Face Daily Papers research 2d ago Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness Abstract Researchers propose source-contrastive evaluation via a localized benchmark to detect data contamination and assess localization robustness in multilingual translation models. Generated by thinkingmachines/Inkling-Small Multilingual translation benchmarks are typically… 4 Hugging Face Daily Papers research 2d ago Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure Abstract Optimized GPU kernel benchmarks reveal that evolutionary LLM proposals exploit evaluation configurations, causing widespread failure to generalize to held-out settings. Generated by thinkingmachines/Inkling-Small Benchmarks for systems that are optimized against the… 11 r/LocalLLaMA community 2d ago DeepSeek V4 Flash 0731 at 27+ t/s decode on Strix Halo — Vulkan + DSpark full guide Been benchmarking DSv4 Flash 0731 on a Flow Z13 (Ryzen AI MAX+ 395, Radeon 8060S / gfx1151, 128GB LPDDR5X) for the past week. Figured I'd share what actually works and what doesn't — there are a lot of gotchas on this hardware. Results Best client-side observation (bench-kv.sh… 7 Hugging Face Daily Papers research 2d ago MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models Abstract MMOOC is a large-scale benchmark assessing whether multimodal language models can correctly refuse out-of-context questions while answering shifted in-context questions, revealing that current models struggle to balance these abilities. Generated by… 37 Hugging Face Daily Papers research 2d ago BDH-CQ: In-Context Learning with Recurrent Latent Reasoning Abstract A 150M-parameter reasoning model using recurrent latent reasoning and in-context learning achieves a new cost-accuracy frontier on ARC-AGI-1. Generated by thinkingmachines/Inkling-Small We introduce BDH-CQ, a reasoning model that combines in-context learning with… 20 Hugging Face Daily Papers research 2d ago WeClawArena: An Auditable Sandbox and Benchmark for Cross-User Agents Collaboration and Security in Human-Centered Agent Networks Abstract WeClawArena is an auditable benchmark and sandbox for evaluating multi-party agent collaboration across personal workspaces, measuring both task utility and security attack success. Generated by thinkingmachines/Inkling-Small Recent advances in persistent personal-agent… 10 r/LocalLLaMA community 3d ago Luth-2: New State-of-the-Art French Small Language Models Hey everyone, Today we release Luth-2-0.8B and Luth2-2-2B , two non-reasoning models that set a new state of the art for French across a wide variety of tasks for their size 🚀 A few notable scores on French benchmarks compared to models 〜3 times their size: - Luth-2-2B scores… 31 arXiv — Machine Learning research 3d ago From Benchmark Performance to Tool Deployment: Human-in-the-Loop Anomaly Detection arXiv:2608.07770v1 Announce Type: new Abstract: Automated anomaly detection methods often report strong performance on curated academic benchmarks, but their behavior under real-world industrial conditions is less clear. In this work, we evaluate 19 unsupervised anomaly… 7 arXiv — Machine Learning research 3d ago Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training arXiv:2608.08224v1 Announce Type: new Abstract: Reinforcement learning post-training unlocks complex reasoning in LLMs. Yet benchmark scores reveal only whether a model improved, not what changed inside it, nor how it splits finite capability across tasks. A representative… 32 arXiv — Machine Learning research 3d ago When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs arXiv:2608.08542v1 Announce Type: new Abstract: Model merging has become the default way to give an aligned language model new skills without retraining: a practitioner folds task vectors from math, code, or domain specialists into a safety-aligned base using task arithmetic,… 12 arXiv — Machine Learning research 3d ago Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure arXiv:2608.08722v1 Announce Type: new Abstract: Benchmarks for systems that are optimized against the evaluation signal measure something different from what they claim. We document this concretely in two GPU-kernel-optimization suites with held-out generalization gates:… 24 arXiv — NLP / Computation & Language research 3d ago Unified Hallucination Fuzzing for Multimodal Large Language Models arXiv:2608.07525v1 Announce Type: new Abstract: Hallucination remains a persistent challenge for Multimodal Large Language Models (MLLMs), severely limiting their reliability in high-stakes applications. Existing evaluations, predominantly based on static benchmarks, suffer from… 16 arXiv — NLP / Computation & Language research 3d ago WuYuEval: A Multi-Level Benchmark for Large Language Models in Solid Waste Management arXiv:2608.07529v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as technical assistants, but their competence in solid waste management (SWM) remains difficult to assess because existing benchmarks emphasize general knowledge rather than… 6 arXiv — NLP / Computation & Language research 3d ago SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators arXiv:2608.07641v1 Announce Type: new Abstract: The rapid advancement of large language models has transformed survey writing from a months-long manual effort into an automated process. As generation scales, reliable evaluation becomes the bottleneck, and LLMs are increasingly… 36 arXiv — NLP / Computation & Language research 3d ago SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs arXiv:2608.07862v1 Announce Type: new Abstract: Existing safety evaluation datasets for large language models (LLMs) predominantly focus on English and Western contexts, often overlooking the linguistic diversity and culturally grounded safety risks present in other languages.… 21 arXiv — NLP / Computation & Language research 3d ago Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives arXiv:2608.08160v1 Announce Type: new Abstract: The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. However, existing research has largely overlooked the critical challenge of… 14 arXiv — NLP / Computation & Language research 3d ago OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents arXiv:2608.08775v1 Announce Type: new Abstract: Agentic benchmarks aim to measure how well AI agents plan, search, execute, and recover within realistic multi-tool environments, but they are almost exclusively in English. As AI agents are globally deployed to a linguistically… 18 arXiv — NLP / Computation & Language research 3d ago LexKairos: Benchmarking Legal Temporal Capabilities in LLMs arXiv:2608.09106v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated strong performance across a wide range of legal tasks. In legal practice, time is a critical concept that governs the validity of statutes, the progression of legal cases, and the… 37 arXiv — NLP / Computation & Language research 3d ago Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments arXiv:2608.09128v1 Announce Type: new Abstract: LLM agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents. Measuring and improving these social skills is hard because, unlike math or logic, social… 29 arXiv — NLP / Computation & Language research 3d ago UNMASK: Discovering and Causally Verifying Spurious Shortcuts in Text Classifiers arXiv:2608.09209v1 Announce Type: new Abstract: Neural language models trained on large crowdsourced corpora frequently exploit spurious surface patterns tied to target labels without true linguistic or causal relevance, boosting benchmark performance while failing on… 33 arXiv — NLP / Computation & Language research 3d ago Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law arXiv:2608.09393v1 Announce Type: new Abstract: We identify and quantify temporal misgrounding: the systematic retrieval and citation of the currently in-force version of a legal article when the applicable version is an earlier or future one. Standard legal RAG treats the… 17 arXiv — NLP / Computation & Language research 3d ago Build it, Break it, Repeat: Benchmarking and improving LLM-manipulated disinformation detection in social media posts arXiv:2608.09510v1 Announce Type: new Abstract: Detecting machine-generated disinformation on social media is increasingly difficult as large language models (LLMs) make it easier to generate and rewrite misleading content at scale. Static benchmark evaluations, measuring… 38 arXiv — NLP / Computation & Language research 3d ago TCS-BENCH: Benchmarking State-of-the-Art Generative AI Theoretical Computer Science Research Ability arXiv:2608.09538v1 Announce Type: new Abstract: We introduce TCS-Bench, a benchmark for evaluating Large Language Models (LLMs) on research-level Theoretical Computer Science (TCS) proof generation. TCS-Bench consists of theorem-proving tasks from papers published at top… 16 arXiv — NLP / Computation & Language research 3d ago Mawqif-v2: An Arabic Benchmark Dataset for Cross-Target Stance Detection arXiv:2608.09539v1 Announce Type: new Abstract: Publicly available Arabic datasets for target-specific stance detection remain limited, particularly for evaluating cross-target generalization. This paper presents the Mawqif-v2 Extension, consisting of 996 manually annotated… 9 arXiv — NLP / Computation & Language research 3d ago ELBench: A Multi-Dimensional Benchmark for Education-Facing Large Language Models arXiv:2608.09548v1 Announce Type: new Abstract: Large language models are increasingly deployed in education as tutors, teaching assistants, and content generators. These roles place demands that ordinary question answering does not: a usable education-facing model is supposed… 32 arXiv — NLP / Computation & Language research 3d ago MDB-Link: Hierarchical Schema Linking for Multi-Database Text-to-SQL arXiv:2608.09588v1 Announce Type: new Abstract: Traditional Text-to-SQL research and benchmarks assume a known target database, overlooking settings in which a query must be routed within a large, heterogeneous database collection. We therefore study schema linking in a… 31 arXiv — NLP / Computation & Language research 3d ago Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness arXiv:2608.09766v1 Announce Type: new Abstract: Multilingual translation benchmarks are typically sourced in English and translated into other languages, treating language pairs as the unit of evaluation---a design that is prone to contamination over time and overlooks locale… 30 arXiv — NLP / Computation & Language research 3d ago PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models arXiv:2608.09772v1 Announce Type: new Abstract: Large Vision-Language Models (LVLMs) have demonstrated strong performance on multimodal benchmarks, yet it remains unclear whether they genuinely reason about relationships between images and text or rely on superficial… 8 Hugging Face Daily Papers research 3d ago Business Arena: Benchmarking LLM Agents in a Realistic Marketplace Abstract Business Arena evaluates LLM agents running a realistic cross-border shop, revealing large performance gaps versus human strategies and enabling detailed attribution of business decisions. Generated by thinkingmachines/Inkling-Small Running a business is a challenging… 21 Hugging Face Daily Papers research 3d ago SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring Abstract SWE-Bench ProMax is a rigorously curated multilingual benchmark of large-scale code refactoring tasks that reveals substantial unsolved challenges for current AI coding agents. Generated by thinkingmachines/Inkling-Small As AI coding agents take on increasingly complex,… 21 r/LocalLLaMA community 3d ago Muse glimmer benchmark Little less smart than Qwen, but way fewer tokens per task.   submitted by   /u/NoFaithlessness951 [link]   [comments] 18 r/LocalLLaMA community 3d ago I made a web-design benchmark for local models (Muse Glimmer 30B vs Qwen 3.6 27b vs Deepseek V4 Flash 0731)   submitted by   /u/ShadyShroomz [link]   [comments] 34 r/LocalLLaMA community 3d ago Achievable 253 t/s - unsloth/Muse Glimmer 30B UD-Q5_K_M on a 5090 Benchmarked Muse Glimmer 30B on my RTX 5090 (32GB), 262k context, UD-Q5_K_M + dflash-kquant + mmproj. Workload Stock master + DFlash ngram-simple PR #26842 + DFlash Code patch 78 t/s 57 t/s 220-253 t/s Mixed agent turn 77 t/s 68 t/s 188-213 t/s Tool-call JSON 71 t/s 75 t/s… 15 r/LocalLLaMA community 3d ago I compared GGUF quants of Qwen3.6 27B to NVFP4, AWQ, AutoRound, and FP8 There's an interactive chart and some extra data in the blog post if you're interested. There are plenty of KL-divergence benchmarks for GGUF models, but most of them compare one GGUF quant against another. I wanted to know how those quants stack up against other commonly used… 28 Page 2 of 10 · 500 articles ← Newer Older →
