News / #multimodal Tag Multimodal 500 articles archived under #multimodal · RSS Sign in to follow Hugging Face Daily Papers research 8d ago OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents Abstract LLM agents are increasingly applied to open-ended everyday requests that span work, study, and life. These tasks are long-horizon, cross-environment, and multimodal, forcing the agent to preserve goals and constraints across many steps while navigating heterogeneous… 6 Hugging Face Daily Papers research 8d ago Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes Abstract Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain… 12 r/LocalLLaMA community 8d ago Xiaomi-Robotics-1: New robotics model released Xiaomi-Robotics-1 is a robot foundation model trained on over 100K hours of real-world manipulation trajectories. It is a Vision-Language-Action (VLA) model engineered for out-of-the-box mobile manipulation in unseen environments and efficient adaptation to new tasks. XR-1… 14 Hugging Face Daily Papers research 8d ago CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning Abstract Chart question answering (CQA) requires multimodal large language models (MLLMs) to integrate visual comprehension with logical reasoning, yet current models struggle with accurate visual grounding and coherent reasoning chains. While extrinsic chain-of-thought… 11 Hugging Face Daily Papers research 9d ago PosterMELD: Multi-Agent Paper-to-Poster Generation for Controllable Design Diversity with Editable Print-Ready Outputs Abstract Scientific poster construction compresses a long multimodal paper into a readable, editable canvas. Existing systems hide request-level failures by scoring only completed outputs; direct image generation is not element-editable, while coding-agent workflows are costly.… 35 Hugging Face Daily Papers research 9d ago Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent Abstract We introduce Video-DeepResearch (Video-DR), extending multimodal agents from static images to continuous video streams, a setting that demands dense spatiotemporal grounding coupled with open-web exploration. Preliminary evaluations reveal two critical bottlenecks in… 8 Hugging Face Daily Papers research 9d ago ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts Abstract World Action Models (WAMs) have emerged as a promising paradigm by jointly modeling robot actions and future visual dynamics. However, their reliance on pixel-generative future supervision can entangle action-relevant state transitions with task-irrelevant visual… 5 Hugging Face Daily Papers research 9d ago Quo Vadis, World Modeling? Abstract Continually improving agents require dynamic interaction feedback beyond static supervision, yet direct real-environment interaction is costly, slow, unsafe, and hard to parallelize. World modeling offers a natural intermediate proxy that allows agents to query… 19 arXiv — Machine Learning research 9d ago Multimodal Auto-regressive Transformer Surrogate for Modeling Variable Operations and Quantifying Uncertainty in Geological Carbon Storage arXiv:2608.02629v1 Announce Type: new Abstract: The use of variable well perforation and injection strategies can improve the efficiency of geological carbon storage operations. We develop a new multimodal auto-regressive transformer surrogate to model these operations under… 24 arXiv — Machine Learning research 9d ago Learning Molecular Representations from Cellular Phenotypes with Structure Preservation arXiv:2608.02688v1 Announce Type: new Abstract: Phenotypic drug discovery enables the discovery of functional relationships between molecular structures and cellular responses. However, existing multimodal representation learning methods often optimize cross-modal alignment… 11 arXiv — Machine Learning research 9d ago PatTree: a novel approach for automated creation of multimodal, graph-based patient representations for medical classification tasks arXiv:2608.02692v1 Announce Type: new Abstract: Access to holistic, multimodal data improves the performance of Artificial Intelligence (AI) in medical classification tasks compared to utilizing single modalities or data sources. However, the inherent heterogeneity and… 24 arXiv — Machine Learning research 9d ago Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering arXiv:2608.02907v1 Announce Type: new Abstract: Multimodal retrievers are essential for knowledge-based visual question answering, where they retrieve external evidence for image-question pairs. However, existing contrastive training methods typically treat all unmatched… 4 arXiv — Machine Learning research 9d ago CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning arXiv:2608.03673v1 Announce Type: new Abstract: Many critical reasoning tasks, including clinical diagnosis, legal judgment, and industrial fault diagnosis, require step-dependent causal chains in which early errors propagate and correct conclusions can mask invalid reasoning.… 28 arXiv — Machine Learning research 9d ago Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning arXiv:2608.03875v1 Announce Type: new Abstract: Designing effective reward functions remains a major bottleneck in Reinforcement Learning (RL). Recent work uses large foundation Vision-Language Models (VLMs) as reward models, computing text-observation similarity to bypass… 8 arXiv — NLP / Computation & Language research 9d ago OncoTriad-QA: A Patient-Level Radiology-Pathology-Genomics Benchmark for Pan-Cancer Reasoning arXiv:2608.02615v1 Announce Type: new Abstract: Cancer diagnosis and characterization require integrating complementary evidence from radiology, pathology, genomics, and clinical metadata. However, most medical large language model (LLM) and vision-language model (VLM)… 10 arXiv — NLP / Computation & Language research 9d ago Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach arXiv:2608.03204v1 Announce Type: new Abstract: Post-training reinforcement learning (RL) algorithms are commonly used to align large vision-language models (LVLMs) with human intent and the requirements of visual reasoning tasks. However, existing RL-based alignment methods are… 18 arXiv — NLP / Computation & Language research 9d ago ArtECulture: Benchmarking Culture-Conditioned Visual Emotion Understanding in Multimodal Large Language Models arXiv:2608.03358v1 Announce Type: new Abstract: Existing visual emotion understanding methods typically ignore cultural variations in emotional perception. We introduce culture-conditioned visual emotion understanding, a task that predicts the culture-specific emotional… 10 arXiv — NLP / Computation & Language research 9d ago TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning arXiv:2608.04007v1 Announce Type: new Abstract: Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit… 24 arXiv — NLP / Computation & Language research 9d ago CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning arXiv:2608.02833v1 Announce Type: cross Abstract: Chart question answering (CQA) requires multimodal large language models (MLLMs) to integrate visual comprehension with logical reasoning, yet current models struggle with accurate visual grounding and coherent reasoning chains.… 14 Hugging Face Daily Papers research 9d ago Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editing Abstract Recent advances in image generation have demonstrated the potential of unified multimodal models that integrate understanding, generation, and editing. However, unified 3D modeling remains constrained by scarce multimodal data, particularly the lack of large-scale and… 8 Hugging Face Daily Papers research 9d ago CAPEval: A Decoupled Caption Evaluation across Understanding and Generation Abstract Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information… 22 Hugging Face Daily Papers research 9d ago TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning Abstract Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios.… 26 Hugging Face Daily Papers research 9d ago Wnuan: Staged Post-Training for Question Answering over Proprietary Enterprise Knowledge Abstract Enterprise question answering requires models to acquire proprietary knowledge without discarding general capabilities. We present Wnuan, a three-stage pipeline that constructs task-oriented supervision from documents, performs supervised fine-tuning with general-data… 24 Hacker News — AI on Front Page community 9d ago Mistral's Shieldstral: 3B open-weights model for multimodal moderation Article URL: https://mistral.ai/news/shieldstral/ Comments URL: https://news.ycombinator.com/item?id=49171268 Points: 307 # Comments: 74 7 Hugging Face Daily Papers research 9d ago Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI Abstract Multimodal clinical models are usually judged on accuracy with every modality present, but deployment removes modalities; an echocardiogram is often unavailable where an ECG is routine. Two questions then matter beyond the size of the accuracy loss: which modality was… 14 Hugging Face Daily Papers research 9d ago Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models Abstract Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We… 26 Hugging Face Daily Papers research 9d ago ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures Abstract Scientific figure comprehension and reasoning using multimodal AI requires integrating visual perception with domain-specific reasoning to extract meaningful knowledge, often not presented in the text of a research publication. The Sci-ImageMiner benchmark dataset,… 6 r/LocalLLaMA community 10d ago Why are Chinese models better* at Frontend than the western top labs? I use A LOT both openAI and Anthropic products. When I need some frontend work (pure web dev) (or answer that feel less verbose and more to the point) I use Anthropic. For multimodality openAI feels better (understanding audio, screenshots, generating images, etc). But openAI… 17 Hugging Face Daily Papers research 10d ago Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts Abstract Vision-language MoE batches contain different numbers of image and text tokens. Image resolution, image count, tiling, and prompt length all change this token mix. We call the standard token-level Switch auxiliary loss Std-Aux. Std-Aux balances only the mixed load, so… 33 Hugging Face Daily Papers research 10d ago DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents Abstract Multimodal large language models (MLLMs) have advanced visual understanding and reasoning, yet their static parametric knowledge limits their ability to address knowledge-intensive and dynamically evolving open-world problems. To move beyond this limitation, multimodal… 24 Hugging Face Daily Papers research 10d ago 3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering Abstract Recent 3D vision-language models (3D VLMs) construct geometry aware tokens by projecting 2D visual features into world coordinates, enabling spatial reasoning for tasks such as 3D question answering. However, this design generates thousands of tokens per scene,… 13 Hugging Face Daily Papers research 10d ago LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks Abstract Large language model (LLM) agents increasingly undertake long-horizon tasks that require sustained reasoning, tool use, and revision across many interdependent steps. However, existing agent harnesses maintain task execution, task state, and completion assessment within… 37 Smol AI News news-outlet 10d ago not much happened today **Alibaba** launched **Qwen3.8-Max**, enhancing multimodal capabilities and agent ecosystem integration. **NVIDIA** introduced **Alpamayo 2 Super** for autonomous vehicle reasoning, while **Mistral AI** released **Shieldstral**, a 3B parameter open-weights safety model for… 17 arXiv — Machine Learning research 10d ago Rethinking Pretraining for Specialized Design Data: Evidence from the JONES-19 Cultural Design Dataset arXiv:2608.00135v1 Announce Type: new Abstract: Design and architectural archives encode expert human knowledge in graphical formats, providing a critical testbed for design-inspired Machine Learning (ML) challenges absent with typical computer vision benchmarks. Building on… 33 arXiv — Machine Learning research 10d ago RHEA: Reliability-Harmonized Reconstruction and Assignment for Robust Multimodal-Attributed Graph Clustering arXiv:2608.00621v1 Announce Type: new Abstract: Multimodal-attributed graphs (MAGs), whose nodes carry heterogeneous attributes such as text and images over a relational structure, have become a fundamental substrate for label-free entity grouping tasks, including community… 23 arXiv — Machine Learning research 10d ago Towards Effective Federated Multimodal Graph Learning via Navigating Multifaceted Heterogeneity arXiv:2608.00623v1 Announce Type: new Abstract: Multimodal-attributed graphs (MAGs), where nodes carry heterogeneous semantic content across multiple modalities while edges encode relational dependencies, have been widely adopted across diverse domains. Federated multimodal… 4 arXiv — Machine Learning research 10d ago Generic Vision and Cross-Attention for Reaction Yield Prediction arXiv:2608.00776v1 Announce Type: new Abstract: Traditional reaction yield prediction is constrained by 1D quantum descriptors that lack explicit spatial information. To address this gap, a dual-modal Vision Cross-Attention architecture is proposed, fusing tabular… 36 arXiv — Machine Learning research 10d ago GeoArbiter: Verifiability-Guided Grounding for Remote-Sensing Multimodal LLMs arXiv:2608.00877v1 Announce Type: new Abstract: Remote-sensing multimodal large language models (MLLMs) often assert facts that imagery cannot establish, such as a facility's identity or function. Coordinate-keyed geographic retrieval can supply this missing knowledge, improving… 28 arXiv — Machine Learning research 10d ago Caliber: Cross-Architecture Extraction-Cost Control for Score-Returning APIs arXiv:2608.01023v1 Announce Type: new Abstract: We present Caliber, an output-perturbation defense against model extraction that formulates noise selection as a calibration problem: how much the defense degrades the supervision signal used to train a surrogate, and the provable… 6 arXiv — Machine Learning research 10d ago Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models arXiv:2608.01263v1 Announce Type: new Abstract: On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories. This aligns the… 6 arXiv — NLP / Computation & Language research 10d ago Obshazard-bench: Benchmarking Multimodal Foundation Models for Real-Time Disaster Intelligence from Raw Earth Observation Streams arXiv:2608.00012v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) are increasingly used to interpret Earth observation data, yet their capability to support real-world disaster emergency response remains insufficiently evaluated. Existing remote sensing… 26 arXiv — NLP / Computation & Language research 10d ago What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs arXiv:2608.00013v1 Announce Type: new Abstract: Choosing the right large language model (LLM) backbone is the most consequential decision when building a vision-language model (VLM), yet it remains fundamentally unprincipled: compute-based scaling laws fail to generalize across… 11 arXiv — NLP / Computation & Language research 10d ago Cross-Task Dissociation in Frontier Vision-Language Model Theory of Mind arXiv:2608.00261v1 Announce Type: new Abstract: Do frontier vision-language models present a coherent Theory-of-Mind (ToM) profile across tasks, matching the same human reference group, or does that profile fragment from one paradigm to the next? We evaluate a shared panel of… 22 arXiv — NLP / Computation & Language research 10d ago AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction arXiv:2608.00434v1 Announce Type: new Abstract: Multi-Token Prediction (MTP) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads, training the model to predict several future tokens in parallel to enrich its supervision… 28 arXiv — NLP / Computation & Language research 10d ago Evaluating VLMs on Multimodal Aristotelian Persuasion Tasks arXiv:2608.01238v1 Announce Type: new Abstract: Vision Language Models (VLMs) have demonstrated exceptional performance across various tasks. However, they have not yet been thoroughly evaluated on more complex tasks. The Persuasion Model, conceived by Aristotle, resembles a… 35 arXiv — NLP / Computation & Language research 10d ago LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning arXiv:2608.01328v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are rapidly evolving with expanded context windows and stronger reasoning capabilities, enabling multi-chart understanding and multi-step inference. These abilities are increasingly… 10 arXiv — NLP / Computation & Language research 10d ago Discriminative Axis, Not Data Volume: What a Contrastive Corpus Teaches an Audio Embedding arXiv:2608.01560v1 Announce Type: new Abstract: Scaling the corpus is the default remedy when a contrastive representation lacks an attribute. We report a case where it does nothing, and identify what does: adding a lexical-speech round to a frozen-base multimodal embedding… 29 Hugging Face Daily Papers research 10d ago UEmbed: Unified Sparse and Dense Multimodal Embeddings Abstract Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semantics. Yet LSR has so far remained tied to… 5 Hugging Face Daily Papers research 10d ago WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning Abstract Reinforcement learning (RL) post-training of Vision-Language-Action (VLA) models has shown strong promise for robotic manipulation. Among RL methods, critic-based approaches rely on a value estimator that predominantly operates on single-frame observations or… 24 Hugging Face Daily Papers research 10d ago Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs Abstract Recent Vision-Language-Action (VLA) models for autonomous driving (AD) increasingly utilize chain-of-thought (CoT) supervision to enhance the reasoning capabilities of their Vision-Language Model (VLM) components, yet existing annotation pipelines commonly expose the… 28 Page 4 of 10 · 500 articles ← Newer Older →
