News / #multimodal Tag Multimodal 500 articles archived under #multimodal · RSS Sign in to follow Simon Willison community 4d ago SQLite compressed text-history prototypes Research: SQLite compressed text-history prototypes I'm perennially interested in options for storing revision histories in relational databases. While out on a dog walk I had a new idea: how about taking the full text of every prior version in a big JSON array of strings and… 8 Simon Willison community 4d ago SQLite compressed text-history prototypes Research: SQLite compressed text-history prototypes I'm perennially interested in options for storing revision histories in relational databases. While out on a dog walk I had a new idea: how about taking the full text of every prior version in a big JSON array of strings and… 32 Hugging Face Daily Papers research 6d ago Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills Abstract Robot learning is splitting into two bets: policies that bake competence into frozen weights (vision-language-action, or VLA, models), and agents that write and refine their own executable skills as code. This survey organises the field around that axis of weights… 38 r/LocalLLaMA community 6d ago Qwen 3.6 27B flags/settings in llama.cpp I run the following on a 5090 and have been okay with its performance, it does most things somewhere 80-100 t/s, though that can slow down at full 262k context - more like 40 t/s at times. I use it primarily in appdev tasks. This just barely fits in the 5090, no vision, with… 31 Hugging Face Daily Papers research 6d ago KVAE: Family of Tokenizers for Multimodal Generative Models Abstract Latent diffusion modeling (LDM), a prominent paradigm, utilizes tokenizers to map input signal to compressed representation. This dependency positions tokenizer as an integral part of generation process itself, since it affects learning speed, quality of synthesized… 17 Hugging Face Daily Papers research 6d ago DyPES-VLA: Learning Shared Dynamics Priors and Embodiment-Specific Control for Cross-Embodiment Manipulation Abstract Vision-Language-Action (VLA) models have become a powerful paradigm for robot manipulation, but training a single generalist policy for heterogeneous robot embodiments remains an open problem. Existing methods have two main limitations. First, they underuse dynamics… 38 Hugging Face Daily Papers research 7d ago Invisible Shortcuts: Why Vision Encoders Know Your Camera Abstract Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces… 37 arXiv — Machine Learning research 7d ago Decoupling Perception from Description: Computation-Grounded Representation Alignment between Multivariate Time Series and Language arXiv:2608.05238v1 Announce Type: new Abstract: Training multimodal models to align time series with language runs into a self-supervision trap. The usual recipe asks an LLM to read a series and write a description, so label quality is capped by the perceptual skill the model is… 31 arXiv — Machine Learning research 7d ago PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis arXiv:2608.05249v1 Announce Type: new Abstract: Real-world multimodal instructions often bundle multiple requirements with unequal importance, yet most multimodal training data still reduce instruction following to answering one self-contained question. We study this gap through… 29 arXiv — Machine Learning research 7d ago GAUGE: Granularity-Adaptive Counterfactual Gating of Evidence for Incomplete Multimodal Classification arXiv:2608.05608v1 Announce Type: new Abstract: Multimodal classification typically assumes all modalities are available, yet real-world inputs are often incomplete. Imputation and dynamic fusion can mitigate such incompleteness, but existing methods operate at a coarse modality… 28 arXiv — Machine Learning research 7d ago Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading arXiv:2608.05675v1 Announce Type: new Abstract: Label-free reliability for vision-language models rests on invariance: perturb the input and a faithful reader's answer should not change. This has a known blind spot, a systematic misreading survives the perturbation and gets… 24 arXiv — Machine Learning research 7d ago Equipment-centric workpiece localization in near real-time using deep learning-based vision and event-driven finite state machines arXiv:2608.05744v1 Announce Type: new Abstract: Continuous workpiece localization is essential for traceability and process coordination in hot forging, but direct tracking is unreliable because of extreme temperatures, surface degradation, and irregular routing. This study… 13 arXiv — Machine Learning research 7d ago Is Self-Pretraining really useful to improve diagnosis in medical Time Series? arXiv:2608.06122v1 Announce Type: new Abstract: Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether similar gains extend to multimodal, multivariate, and even simple univariate medical… 23 arXiv — Machine Learning research 7d ago On-Policy Self-Distillation without Any Supervision arXiv:2608.06296v1 Announce Type: new Abstract: On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals,… 37 arXiv — Machine Learning research 7d ago Positive-Unlabeled Preference Optimization For Chest X-ray Report Generation arXiv:2608.05341v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) for radiology report generation are typically trained on retrospective clinical reports, which suffer from omission noise: clinically present findings are left unreported due to the omission of… 25 arXiv — NLP / Computation & Language research 7d ago RIG-RoPE: Relation- and Instance-Gated Rotary Positional Encoding with Duration-Aware Temporal Coordinates arXiv:2608.05154v1 Announce Type: new Abstract: Rotary positional encoding (RoPE) is a core component of modern language models and has been extended to multimodal LLMs through multidimensional variants such as multimodal RoPE (M-RoPE), which split positional channels into… 22 arXiv — NLP / Computation & Language research 7d ago M$^3$R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding arXiv:2608.05817v1 Announce Type: new Abstract: Metaphor enables the understanding of abstract concepts through cross-domain mappings while conveying affective attitudes. In multimodal scenarios, visual and textual information jointly construct Target--Source mappings, requiring… 25 arXiv — NLP / Computation & Language research 7d ago GenGA: Editable and Data-Grounded Graphical Abstract Generation for Academic Papers arXiv:2608.05478v1 Announce Type: cross Abstract: Graphical Abstracts (GAs) visually summarize the key findings of academic papers, playing a crucial role in facilitating the understanding of research content. Recently, advancements in vision-language models and image generation… 38 Hugging Face Daily Papers research 7d ago World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation Abstract Vision-language-action (VLA) models often treat main-view and wrist-view observations as parallel visual inputs, overlooking their distinct roles in robot manipulation. Fine-grained manipulation, however, benefits from anticipating how wrist-local interactions may… 38 Hugging Face Daily Papers research 7d ago Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval Abstract Unified multimodal retrieval aims to identify candidates that satisfy complex user intent expressed through heterogeneous inputs. Although Large Vision-Language Model (LVLM)-based retrievers are efficient and scalable, directly encoding raw multimodal inputs often… 14 Hugging Face Daily Papers research 7d ago ChronoVision: Temporal Reasoning via Latent State Reconstruction Abstract Multimodal large language models excel at passive perception but struggle with complex visual cognitive tasks requiring multi-step temporal reasoning. This degradation largely stems from the inherent ambiguity of language-based reasoning, which often fails to accurately… 15 r/LocalLLaMA community 7d ago nvidias nemotron omni only loads its text half on a mac, so i wrote the vision and audio towers in mlx nvidias nemotron omni is open weights and it sees, hears and reasons. theres already a 4bit mlx quant on hugging face but only the text backbone loads with standard mlx tooling. the model card says it plainly, the vision and audio towers need a runtime that implements the… 20 Hugging Face Daily Papers research 7d ago SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models Abstract Multimodal large language models (MLLMs) make grounded predictions in real-world scenes by combining visual and textual cues, yet existing benchmarks rarely reveal how they arbitrate between these evidence sources when they conflict. We introduce SIGNPOST-Bench, a… 14 Hugging Face Daily Papers research 7d ago DRIFT: Derailing Denoising Trajectories of Flow-Matching VLAs with Adversarial Patch Attack Abstract Flow-matching vision-language-action (VLA) models such as pi0 generate robot actions by integrating a learned denoising velocity field, and have been reported to resist adversarial perturbations that readily fool autoregressive VLAs. We show that this robustness is… 25 Hugging Face Daily Papers research 7d ago OPD-V: Visual On-Policy Self-Distillation with Modality Balance Abstract On-Policy Self-Distillation (OPSD) has become a standard post-training approach for improving visual reasoning in multimodal large language models (MLLMs). Existing methods draw privileged information from diverse input sources to guide self-distillation. Yet these… 22 Hugging Face Daily Papers research 7d ago ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation Abstract Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent… 36 Hugging Face Daily Papers research 8d ago FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory Abstract GUI agents must remember both useful experience from earlier tasks and unfinished progress in the current interaction. Latent memory offers a compact solution by compressing multimodal trajectories into a few continuous tokens. Existing methods, however, usually map… 15 r/MachineLearning community 8d ago What are the biggest challenges in collecting high-quality speech and egocentric video datasets? [D] We're currently involved in collecting two types of datasets that seem to be increasingly important for multimodal AI Studio quality speech/audio datasets (high fidelity recordings) Egocentric household activity video datasets (first person daily task recordings) One thing that… 37 arXiv — Machine Learning research 8d ago C$^2$MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning arXiv:2608.04013v1 Announce Type: new Abstract: Recent advances in Multimodal Emotion Recognition in Conversations (MERC) highlight its reliance on complete multimodal inputs. However, real-world data often suffer from missing modalities due to transmission errors or user… 35 arXiv — Machine Learning research 8d ago EvtGraph: Event-Adaptive Compression for Sparse Temporal Graph Learning in Multimodal Time Series arXiv:2608.04368v1 Announce Type: new Abstract: Multimodal temporal data are inherently irregular and uneven in information density, yet most models rely on uniform discretization, leading to inefficient representations. We propose \textbf{EvtGraph}, a unified framework that… 37 arXiv — Machine Learning research 8d ago When Proxy Prediction Becomes Equation Reconstruction: Diagnostics and Residual Learning for Factor-Derived Proxy Supervision arXiv:2608.04393v1 Announce Type: new Abstract: Scientific machine learning often relies on proxy targets computed from known domain factors when direct observations are limited. When those same factors are used as model inputs, however, high predictive accuracy may reflect… 6 arXiv — Machine Learning research 8d ago SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation arXiv:2608.04419v1 Announce Type: new Abstract: On-policy distillation (OPD) provides dense teacher supervision on student-generated trajectories, but standard reverse-KL training can assign insufficient probability to other plausible continuations. Teacher entropy alone does… 25 arXiv — Machine Learning research 8d ago A Model Merging Approach for Continual MLLM Unlearning arXiv:2608.04548v1 Announce Type: new Abstract: Multimodal large language model (MLLM) unlearning methods have been proposed to remove private, sensitive, or proprietary information from well-trained models. However, most existing MLLM unlearning methods are designed for… 5 arXiv — Machine Learning research 8d ago Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervision arXiv:2608.04879v1 Announce Type: new Abstract: Vision Transformers (ViTs) achieve strong image-recognition performance, but their parameter count grows linearly with depth when each block is independently parameterized. Single-block recurrent ViTs (bViT) remove this growth by… 22 arXiv — Machine Learning research 8d ago Multimodal Spatiotemporal Atmospheric Data Assimilation with Latent Flow-matching arXiv:2608.05103v1 Announce Type: new Abstract: Data assimilation (DA) uses Bayesian inference to update the state of a numerical forecast model with observed data. In this study, we propose a fundamentally different, unified approach to atmospheric data assimilation. We use… 35 arXiv — Machine Learning research 8d ago AI-driven Multimodal Representation Learning for Latent Mediation Structure Discovery of Socioeconomic Disadvantage, Psychosocial Factors, and Cardiometabolic Multimorbidity: Insights from the All of Us Research Program arXiv:2608.04016v1 Announce Type: cross Abstract: Social disadvantage is associated with multimorbidity, but the pathways linking social conditions to disease burden remain poorly understood. We developed an AI-driven multimodal mediation framework that integrates socioeconomic,… 19 arXiv — NLP / Computation & Language research 8d ago The Calibration Floor: Format Repair Can Masquerade as Self-Correction at Small-to-Mid Scale arXiv:2608.04355v1 Announce Type: new Abstract: Accuracy changes after language-model self-revision are usually interpreted as changes in reasoning. We show this can fail at the answer-extraction boundary, and test the failure causally rather than only observationally. Across… 11 arXiv — NLP / Computation & Language research 8d ago The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity arXiv:2608.04463v1 Announce Type: new Abstract: Prior work on LLM conformity largely measures discrete answer flips under verifiable labels. Open-ended revisions require a different measurement strategy because answer quality is graded, latent, and judged imperfectly. We… 26 arXiv — NLP / Computation & Language research 8d ago RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care arXiv:2608.04514v1 Announce Type: new Abstract: Background: Respiratory specialty care requires multimodal interpretation, longitudinal risk assessment, guideline-concordant intervention, and whole-course management, which are poorly represented by examination-oriented medical… 11 arXiv — NLP / Computation & Language research 8d ago Breaking the Curse ofMultilinguality inMany-to-Many Speech-to-Text Translation via a Resource-AwareMixture of Speech Encoders arXiv:2608.04586v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from… 15 arXiv — NLP / Computation & Language research 8d ago Guideline-as-Oracle: Zero-Annotation Training of an Ophthalmic Telephone Triage Agent arXiv:2608.04772v1 Announce Type: new Abstract: Scaling supervision for multi-turn medical agents is difficult because expert dialogue annotation is costly and clinical conversations are privacy-restricted. We introduce Guideline-as-Oracle (GAO), which compiles American Academy… 19 arXiv — NLP / Computation & Language research 8d ago Reading Between the Frames: Interpreting Implicit and Non-literal Meaning in Social Media Videos arXiv:2608.04939v1 Announce Type: new Abstract: Social media videos often communicate meanings that go beyond their visible actions, captions, or speech. A mundane clip may become humorous, ironic, or satire only through the interaction of multimodal cues and cultural context,… 20 arXiv — NLP / Computation & Language research 8d ago OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents arXiv:2608.05013v1 Announce Type: new Abstract: LLM agents are increasingly applied to open-ended everyday requests that span work, study, and life. These tasks are long-horizon, cross-environment, and multimodal, forcing the agent to preserve goals and constraints across many… 27 arXiv — NLP / Computation & Language research 8d ago Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understanding arXiv:2608.04054v1 Announce Type: cross Abstract: Multimodal intent recognition requires understanding not only what textual, acoustic, and visual signals share, but also how they disagree. Such disagreement is frequently class-informative; for example, lexical positivity… 26 arXiv — NLP / Computation & Language research 8d ago SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models arXiv:2608.04244v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) make grounded predictions in real-world scenes by combining visual and textual cues, yet existing benchmarks rarely reveal how they arbitrate between these evidence sources when they… 21 arXiv — NLP / Computation & Language research 8d ago Q-CueGraph: Query-Conditioned Visual Evidence Graphs for Multimodal Reasoning arXiv:2608.04452v1 Announce Type: cross Abstract: High-resolution pixels and crop or zoom tools give multimodal large language models the ability to inspect an image, but they do not provide a reliable task-conditioned policy for deciding where to inspect. Q-CueGraph makes this… 23 arXiv — NLP / Computation & Language research 8d ago EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment arXiv:2608.04472v1 Announce Type: cross Abstract: The development of foundation models (FMs) is crucial for advancing endoscopic image analysis. However, existing endoscopy FMs mainly rely on self-supervised learning from uni-modal images or videos, overlooking the rich semantic… 18 arXiv — NLP / Computation & Language research 8d ago Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs arXiv:2608.04759v1 Announce Type: cross Abstract: Although Multimodal Large Language Models (MLLMs) have made substantial progress, their spatial reasoning may still produce intermediate judgments inconsistent with the input image, allowing errors to propagate through the… 9 arXiv — NLP / Computation & Language research 8d ago Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations arXiv:2608.04885v1 Announce Type: cross Abstract: Standard accuracy metrics for VLMs often mask significant reliability failures in sensitive domains. In this work, we utilize a histopathology-validated brain MRI dataset to systematically assess the diagnostic robustness of four… 19 Hugging Face Daily Papers research 8d ago BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation Abstract Leveraging pre-trained vision-language models (VLMs) to construct vision-language-action (VLA) models has emerged as a promising paradigm for 3D robot manipulation. However, existing 3D VLA methods remain data-hungry, exhibit limited generalization under distribution… 4 Page 3 of 10 · 500 articles ← Newer Older →
