We propose MedFocus and MedGround-Bench.</p>\n","updatedAt":"2026-05-21T02:48:27.756Z","author":{"_id":"657e56d11e3e9c41a4a57d2c","avatarUrl":"/avatars/1a6e7cff2693e1523f87ad24f4529872.svg","fullname":"Guangzhi Xiong","name":"gzxiong","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8571930527687073},"editors":["gzxiong"],"editorAvatarUrls":["/avatars/1a6e7cff2693e1523f87ad24f4529872.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2605.20158","authors":[{"_id":"6a0e7224164dbbc68a26c4b7","name":"Guangzhi Xiong","hidden":false},{"_id":"6a0e7224164dbbc68a26c4b8","name":"Qiao Jin","hidden":false},{"_id":"6a0e7224164dbbc68a26c4b9","name":"Sanchit Sinha","hidden":false},{"_id":"6a0e7224164dbbc68a26c4ba","name":"Zhiyong Lu","hidden":false},{"_id":"6a0e7224164dbbc68a26c4bb","name":"Aidong Zhang","hidden":false}],"publishedAt":"2026-05-19T00:00:00.000Z","submittedOnDailyAt":"2026-05-21T00:00:00.000Z","title":"Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models","submittedOnDailyBy":{"_id":"657e56d11e3e9c41a4a57d2c","avatarUrl":"/avatars/1a6e7cff2693e1523f87ad24f4529872.svg","isPro":false,"fullname":"Guangzhi Xiong","user":"gzxiong","type":"user","name":"gzxiong"},"summary":"Large Vision Language Models (LVLMs) show promise in medical applications, but their inability to faithfully ground responses in visual evidence raises serious concerns about clinical trustworthiness. While visual attribution methods are widely used to explain LVLM predictions, whether these explanations actually reflect the visual evidence underlying the model's decision is largely unverified, since ground-truth annotations for internal model reasoning are typically unavailable. We address this question for chest X-ray (CXR) reasoning by developing a causal evaluation framework that retains only CXR-VQA samples for which the expert-annotated region is verified, via counterfactual editing, to be causally responsible for the model's prediction. Using this framework across 11 attribution methods, six open-source LVLMs, and two output modes (direct answer and step-by-step reasoning), we find that existing attribution methods often fail to identify the evidence used by LVLMs. To address this failure, we propose MedFocus, a concept-based attribution method that localizes clinically meaningful anatomical regions via unbalanced optimal transport and measures their causal effect on model outputs through targeted interventions. MedFocus produces spatial, concept-level, and token-level attributions and substantially outperforms prior methods, taking a step toward more trustworthy attribution for medical LVLMs. Our data and code are available at https://github.com/gzxiong/medfocus/.","upvotes":1,"discussionId":"6a0e7225164dbbc68a26c4bc","githubRepo":"https://github.com/gzxiong/medfocus","githubRepoAddedBy":"user","ai_summary":"A causal evaluation framework is developed to verify visual evidence grounding in chest X-ray vision-language models, leading to the proposal of MedFocus, a concept-based attribution method that improves clinical trustworthiness through anatomical region localization and causal effect measurement.","ai_keywords":["visual attribution methods","causal evaluation framework","counterfactual editing","chest X-ray","vision-language models","concept-based attribution","unbalanced optimal transport","targeted interventions","clinical trustworthiness"],"githubStars":3},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"657e56d11e3e9c41a4a57d2c","avatarUrl":"/avatars/1a6e7cff2693e1523f87ad24f4529872.svg","isPro":false,"fullname":"Guangzhi Xiong","user":"gzxiong","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2605/2605.20158.md"}">
Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models
Abstract
A causal evaluation framework is developed to verify visual evidence grounding in chest X-ray vision-language models, leading to the proposal of MedFocus, a concept-based attribution method that improves clinical trustworthiness through anatomical region localization and causal effect measurement.
AI-generated summary
Large Vision Language Models (LVLMs) show promise in medical applications, but their inability to faithfully ground responses in visual evidence raises serious concerns about clinical trustworthiness. While visual attribution methods are widely used to explain LVLM predictions, whether these explanations actually reflect the visual evidence underlying the model's decision is largely unverified, since ground-truth annotations for internal model reasoning are typically unavailable. We address this question for chest X-ray (CXR) reasoning by developing a causal evaluation framework that retains only CXR-VQA samples for which the expert-annotated region is verified, via counterfactual editing, to be causally responsible for the model's prediction. Using this framework across 11 attribution methods, six open-source LVLMs, and two output modes (direct answer and step-by-step reasoning), we find that existing attribution methods often fail to identify the evidence used by LVLMs. To address this failure, we propose MedFocus, a concept-based attribution method that localizes clinically meaningful anatomical regions via unbalanced optimal transport and measures their causal effect on model outputs through targeted interventions. MedFocus produces spatial, concept-level, and token-level attributions and substantially outperforms prior methods, taking a step toward more trustworthy attribution for medical LVLMs. Our data and code are available at https://github.com/gzxiong/medfocus/.
Community
We propose MedFocus and MedGround-Bench.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2605.20158 in a model README.md to link it from this page.
Cite arxiv.org/abs/2605.20158 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2605.20158 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.