<span class=\"SVELTE_PARTIAL_HYDRATER contents\" data-target=\"UserMention\" data-props=\"{"user":"jaagli"}\"><span class=\"inline-block\"><span class=\"contents\"><a href=\"/jaagli\">@<span class=\"underline\">jaagli</span></a></span> </span></span>, when do you think the benchmark going to be released? Nice work</p>\n","updatedAt":"2026-08-04T18:38:46.023Z","author":{"_id":"6177322d37f32ecb1e2d4cdf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg","fullname":"Hugo Laurençon","name":"HugoLaurencon","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":163,"isUserFollowing":false}},"numEdits":2,"identifiedLanguage":{"language":"en","probability":0.9500197172164917},"editors":["HugoLaurencon"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.26326","authors":[{"_id":"6a6b26bfb2106777884ab069","user":{"_id":"6686658221f096c7fb4e0b24","avatarUrl":"/avatars/581f5d56b393b95c74f38a06f6bb4f69.svg","isPro":false,"fullname":"Jiaang Li","user":"jaagli","type":"user","name":"jaagli"},"name":"Jiaang Li","status":"claimed_verified","statusLastChangedAt":"2026-08-04T16:45:04.551Z","hidden":false},{"_id":"6a6b26bfb2106777884ab06a","name":"Chengzu Li","hidden":false},{"_id":"6a6b26bfb2106777884ab06b","name":"Zhaochong An","hidden":false},{"_id":"6a6b26bfb2106777884ab06c","name":"Yifei Yuan","hidden":false},{"_id":"6a6b26bfb2106777884ab06d","name":"Xi Liu","hidden":false},{"_id":"6a6b26bfb2106777884ab06e","name":"Serge Belongie","hidden":false},{"_id":"6a6b26bfb2106777884ab06f","name":"Vésteinn Snæbjarnarson","hidden":false}],"publishedAt":"2026-07-28T00:00:00.000Z","submittedOnDailyAt":"2026-08-04T00:00:00.000Z","title":"Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models","submittedOnDailyBy":{"_id":"6686658221f096c7fb4e0b24","avatarUrl":"/avatars/581f5d56b393b95c74f38a06f6bb4f69.svg","isPro":false,"fullname":"Jiaang Li","user":"jaagli","type":"user","name":"jaagli"},"summary":"Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.","upvotes":2,"discussionId":"6a6b26c0b2106777884ab070"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6177322d37f32ecb1e2d4cdf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg","isPro":false,"fullname":"Hugo Laurençon","user":"HugoLaurencon","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.26326.md","query":{}}">
Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
Abstract
Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.
Community
This comment has been hidden @jaagli , when do you think the benchmark going to be released? Nice work
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.26326 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.26326 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2607.26326 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.