News / #benchmark Tag Benchmark 500 articles archived under #benchmark · RSS Sign in to follow arXiv — Machine Learning research 8d ago Design Choices That Matter: A Functional ANOVA Analysis for Remote Sensing Multi-Label Classification arXiv:2608.04702v1 Announce Type: new Abstract: Benchmarking deep learning (DL) models for multi-label classification (MLC) of remote sensing images (RSI) typically yields rankings that do not generalize beyond the evaluated datasets. In this work, we move beyond rankings by… 29 arXiv — Machine Learning research 8d ago Benchmarking Deep Learning Models for Dense Event Classification of Offshore Wind Infrastructure in Sentinel-1 Time Series arXiv:2608.04706v1 Announce Type: new Abstract: Monitoring of offshore wind energy infrastructure life cycles, especially during the deployment phase, is an important contribution for stakeholders to make informed decisions in a phase of increasing deployment activities. ESA's… 31 arXiv — Machine Learning research 8d ago BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning arXiv:2608.05104v1 Announce Type: new Abstract: Deep neural networks have shown impressive success in NLP tasks owing to their complex structure and huge number of edges. Achieving state-of-the-art performance in natural language processing with a large pre-trained model such as… 26 arXiv — NLP / Computation & Language research 8d ago FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation arXiv:2608.04374v1 Announce Type: new Abstract: Large language models can produce fluent financial analysis, but fluency alone does not establish whether a report is suitable for institutional delivery. We introduce FinReportBench, an expert-grounded benchmark for measuring and… 5 arXiv — NLP / Computation & Language research 8d ago NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap arXiv:2608.04397v1 Announce Type: new Abstract: We introduce NOLLI, a procedurally generated English-Korean puzzle benchmark designed to diagnose where Korean performance gaps arise. It comprises 15 puzzle types (25 tasks; 7,500 items), with every instance seed-regenerable,… 5 arXiv — NLP / Computation & Language research 8d ago RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care arXiv:2608.04514v1 Announce Type: new Abstract: Background: Respiratory specialty care requires multimodal interpretation, longitudinal risk assessment, guideline-concordant intervention, and whole-course management, which are poorly represented by examination-oriented medical… 11 arXiv — NLP / Computation & Language research 8d ago Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark arXiv:2608.04670v1 Announce Type: new Abstract: Large Language Models (LLMs) have transformed computational linguistics and achieved remarkable performance across numerous natural language processing tasks, yet significant gaps persist in understanding how these systems process… 38 arXiv — NLP / Computation & Language research 8d ago IslamicTurathBench: A Multi-Task, Multi-Discipline Benchmark for Evaluating Large Language Models on the Islamic Scholarly Tradition (turath) arXiv:2608.04703v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for question answering, education, and research, including in religious and cultural domains where answers depend on specialised source traditions. Yet in Islamic Studies, key… 29 arXiv — NLP / Computation & Language research 8d ago Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning arXiv:2608.05139v1 Announce Type: new Abstract: Long-horizon reasoning in recent LLMs demands that the model switch between distinct skills inside a reasoning chain, such as first doing a math derivation, then using the result to plan a schedule. We call such problems… 17 arXiv — NLP / Computation & Language research 8d ago FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents arXiv:2608.04095v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used as personalized assistants in high-stakes domains such as financial advising, yet it remains unclear whether they can maintain and update an individualized user model over… 27 arXiv — NLP / Computation & Language research 8d ago GEB-Bench: Abstract Structures Told in Many Voices arXiv:2608.04111v1 Announce Type: cross Abstract: Can a model look at a river delta and a lightning bolt and see that they share a structure? We introduce GEB-Bench, a benchmark whose unit is an abstract structural motif--self-reference, a strange loop, a Mobius twist--in the… 21 arXiv — NLP / Computation & Language research 8d ago SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models arXiv:2608.04244v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) make grounded predictions in real-world scenes by combining visual and textual cues, yet existing benchmarks rarely reveal how they arbitrate between these evidence sources when they… 21 arXiv — NLP / Computation & Language research 8d ago Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness arXiv:2608.04519v1 Announce Type: cross Abstract: Benchmarking machine unlearning methods is critical to understand whether sensitive knowledge is removed from large language models (LLMs) or not. Current unlearning benchmarks include mainly single-hop questions and a narrow set… 31 Hugging Face Daily Papers research 8d ago Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning Abstract Long-horizon reasoning in recent LLMs demands that the model switch between distinct skills inside a reasoning chain, such as first doing a math derivation, then using the result to plan a schedule. We call such problems cross-skill long-horizon tasks: multi-step tasks… 30 r/LocalLLaMA community 8d ago Introducing BetterBench - more accurate PP and TPS measurement I built this because the existing benchmarks were using random data and with MTP content types can vary a lot on what performance you see. 5% or more with content types. BetterBench is designed to have content consistency within 1% and also measures across different content… 8 Hugging Face Daily Papers research 8d ago NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap Abstract We introduce NOLLI, a procedurally generated English-Korean puzzle benchmark designed to diagnose where Korean performance gaps arise. It comprises 15 puzzle types (25 tasks; 7,500 items), with every instance seed-regenerable, verified to have a unique solution, and… 18 r/LocalLLaMA community 8d ago 40% speedup of MoE training with faster megakernel, by cursor, of all people (for B200s) daily reminder not to trust benchmarks and run it yourself. claimed e2e speedup is ~40%, forwards are ~140% faster I would wager that compared to a naive kernel anyone can write it's more in the range of 10-20% faster e2e in reality, if at all, but hey, it's free and open!… 8 r/LocalLLaMA community 8d ago MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature: Mixture-of-Experts CPU offload: --n-cpu-moe <N> | -ncmoe <N> Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on the CPU;… 29 r/LocalLLaMA community 9d ago I updated my localy run benchmark with DeepSeek V4 Flash 0731 It's the purple cluster on the top left (the good corner...) I'm running the MXFP4 version from Bartoswski with Dspark at 1K t/s prefill and 90 t/s gen (average). I tried different sampling params, you can check the detail. It's very efficient while scoring the best yet. Too bad… 12 r/LocalLLaMA community 9d ago Anyone interested in building a harness-only benchmark? There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one. End goal: a leaderboard of harness performance (multiple axis) on a set of diverse real world tasks [1] , grouped by underlying models… 12 Hugging Face Daily Papers research 9d ago Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements Abstract Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and multi-step logic over long contexts, is an ideal testbed. Existing benchmarks fail to capture… 10 Hugging Face Daily Papers research 9d ago PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents Abstract Recursive self-improvement requires agents to turn accumulated experience into better future behavior. Personal AI agents offer a concrete setting for studying this capability because they retain preferences, task histories, tool routines, and learned skills across… 6 Hugging Face Daily Papers research 9d ago When Agents Learn to Be You: Benchmarking Privacy Leakage, Impersonation Risk, and Defenses in Persona Skills Abstract Persona skills distill personal interaction histories into portable and executable artifacts for downstream agents. While enabling flexible personalization, this process concentrates fragmented personal signals, amplifies their impact through reuse, and challenges… 29 arXiv — Machine Learning research 9d ago PLAN: Parallel Liquid-Inspired Approximation Network for Efficient Representation Learning in Flexible Job Shop Scheduling arXiv:2608.03041v1 Announce Type: new Abstract: Deep reinforcement learning (DRL) approaches for flexible job shop scheduling (FJSP) heavily rely on attention-centric architectures to achieve state-of-the-art performance. However, these models suffer from excessive parameter… 8 arXiv — Machine Learning research 9d ago FinVerse: Financial Time-Series Benchmark arXiv:2608.03259v1 Announce Type: new Abstract: As time-series foundation models have emerged, the need for benchmarks that can evaluate their forecasting ability in meaningful ways has become increasingly important. Existing time-series forecasting benchmarks provide useful… 14 arXiv — NLP / Computation & Language research 9d ago MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale arXiv:2608.02613v1 Announce Type: new Abstract: Edge-deployed personal memory assistants must handle private interpersonal conversations on-device with open-weight models. Yet, existing memory benchmarks often under-test the combination of activity-dense interaction, ego-centric… 15 arXiv — NLP / Computation & Language research 9d ago OncoTriad-QA: A Patient-Level Radiology-Pathology-Genomics Benchmark for Pan-Cancer Reasoning arXiv:2608.02615v1 Announce Type: new Abstract: Cancer diagnosis and characterization require integrating complementary evidence from radiology, pathology, genomics, and clinical metadata. However, most medical large language model (LLM) and vision-language model (VLM)… 10 arXiv — NLP / Computation & Language research 9d ago Evaluating OpenAI's Privacy Filter: Cross-Lingual, Cross-Domain PII Detection Across 42 Benchmarks arXiv:2608.02616v1 Announce Type: new Abstract: We present the first independent, systematic evaluation of OpenAI's Privacy Filter (OPF), a 1.5B-parameter bidirectional PII detector, across 42 synthetic benchmarks spanning 22 languages and 5 domains. Zero-shot, OPF achieves… 34 arXiv — NLP / Computation & Language research 9d ago JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation arXiv:2608.02620v1 Announce Type: new Abstract: LLM-as-a-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks ship their own code base, hardcode a specific closed-model judge, and support a single… 36 arXiv — NLP / Computation & Language research 9d ago Knowing the Form, Not the Function: Automatically Auditing Answer--Authority Decoupling in Legal Benchmarks arXiv:2608.02621v1 Announce Type: new Abstract: Legal benchmarks typically score final answers even when models also state legal authority. We test whether answer correctness can serve as a proxy for authority grounding. Under ordinary reasoning prompts that did not request… 33 arXiv — NLP / Computation & Language research 9d ago FLARE: Few-shot Learning-based Adaptive Reflective Engine arXiv:2608.02919v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in complex, compound AI systems where performance hinges on the quality of prompts. Recent state-of-the-art optimizers like GEPA (Genetic-Pareto) have argued that reflective… 32 arXiv — NLP / Computation & Language research 9d ago Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks arXiv:2608.02966v1 Announce Type: new Abstract: Most multiple-choice question (MCQ) benchmarks evaluate Large Language Models (LLMs) only by whether they select the correct answers. This binary scoring treats all incorrect responses alike, even though an LLM's preferences among… 22 arXiv — NLP / Computation & Language research 9d ago VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP arXiv:2608.03095v1 Announce Type: new Abstract: We present VIVID (Vietnamese Idioms for Validation and Interpretation Depth), the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese. VIVID comprises 1,636 idioms and… 24 arXiv — NLP / Computation & Language research 9d ago Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks arXiv:2608.03340v1 Announce Type: new Abstract: Predicting LLM's capabilities on real-world tasks is essential, yet the extent to which performance on commonsense benchmarks predicts downstream performance remains underspecified. To establish the practical usability of widely… 7 arXiv — NLP / Computation & Language research 9d ago ArtECulture: Benchmarking Culture-Conditioned Visual Emotion Understanding in Multimodal Large Language Models arXiv:2608.03358v1 Announce Type: new Abstract: Existing visual emotion understanding methods typically ignore cultural variations in emotional perception. We introduce culture-conditioned visual emotion understanding, a task that predicts the culture-specific emotional… 10 arXiv — NLP / Computation & Language research 9d ago M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models arXiv:2608.03803v1 Announce Type: new Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with… 14 arXiv — NLP / Computation & Language research 9d ago VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs arXiv:2608.03810v1 Announce Type: new Abstract: Large language models routinely describe socially salient targets, including political figures, countries, religions, organizations, historical events, and social groups, encoding affective framing alongside factual content: a… 11 arXiv — NLP / Computation & Language research 9d ago MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning arXiv:2608.03882v1 Announce Type: new Abstract: Geospatial reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to navigation and logistics, yet large language models (LLMs) struggle with the required geometric and… 17 arXiv — NLP / Computation & Language research 9d ago PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents arXiv:2608.04003v1 Announce Type: new Abstract: Recursive self-improvement requires agents to turn accumulated experience into better future behavior. Personal AI agents offer a concrete setting for studying this capability because they retain preferences, task histories, tool… 36 arXiv — NLP / Computation & Language research 9d ago WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament arXiv:2608.04008v1 Announce Type: new Abstract: Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewhere on the Web, and the evaluation must defend itself against memorisation. We… 23 arXiv — NLP / Computation & Language research 9d ago SocietyBench: Forecasting Counterfactual Social-World Evolution arXiv:2608.04009v1 Announce Type: new Abstract: Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task -- fix a bug, drive a browser, operate a GUI. A complementary social ability, namely how well a model… 11 arXiv — NLP / Computation & Language research 9d ago Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity arXiv:2608.02665v1 Announce Type: cross Abstract: A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form. We ask whether that reading is faithful: when an item's intent is held fixed and only its meaning-preserving… 38 r/LocalLLaMA community 9d ago GPT-X2.5-135M scores 3rd place on Open SLM Leaderboard on Huggingface, Beating Facebook's MobileLLM-R1-140M   submitted by   /u/Megneous [link]   [comments] 28 r/LocalLLaMA community 9d ago Local LLM 35B MoE — Real-world coding benchmarks (Qwen vs Ornith vs KAT) I’ve been running a fairly opinionated evaluation loop on ~35B A3B/MoE-class models for coding over the past few months. Not synthetic benchmarks: actual dev workflows, iterative debugging, refactoring passes, and failure recovery. Here’s where things stand for me: Qwen 3.6 (35B… 9 r/LocalLLaMA community 9d ago Design systems from code alone - Without external images, Ling-3.0-flash generated webpages across Bauhaus, Bohemian, acid design, and more—using CSS gradients, SVG paths, typography, and layout to preserve each visual language. Weights went up today so this is downloadable now, MIT, ~128GB for the official FP8. I ran these on the API before that landed, so treat it as a preview of what you'd be pulling rather than a local benchmark   submitted by   /u/AcanthisittaOk1699 [link]   [comments] 32 r/LocalLLaMA community 9d ago inclusionAI/Ling-3.0-flash · Hugging Face The Ling-3.0-flash MoE is now open-weighted at 124B A5B params. I know the original announcements were before the Kimi K3, DeepSeek-V4-Flash and Qwen3.8 hype, but this model might still have a good niche for itself due to its sizing. Discussion on the benchmarks are here:… 12 r/LocalLLaMA community 9d ago Deepseek V4 Flash 2-bit quant is the first model I can run locally that achieves 100% in this SQL benchmark I really like to use this one SQL benchmark when testing new models. I had another post some time ago with my benchmarks, but I decided to post a new one because of how well Deepseek did. I like the benchmark because it's quick to run, is pretty "real-world" and requires good… 37 Hugging Face Daily Papers research 9d ago MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations Abstract Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across… 13 Hugging Face Daily Papers research 9d ago ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures Abstract Scientific figure comprehension and reasoning using multimodal AI requires integrating visual perception with domain-specific reasoning to extract meaningful knowledge, often not presented in the text of a research publication. The Sci-ImageMiner benchmark dataset,… 6 Hugging Face Daily Papers research 9d ago GEOID-Flood: A Large-Scale Multi-Modal Benchmark Dataset for Flood Segmentation Abstract Geospatial foundation models aim to learn representations that transfer across regions and sensors, yet evaluating them on specific tasks requires large, high-quality, multi-modal benchmarks that measure how well such models extract value from data. Concerning flood… 19 Page 4 of 10 · 500 articles ← Newer Older →
