We identify and analyze <strong>Perceptual Judgment Bias</strong>, a failure mode where multimodal LLM judges over-reward fluent or plausible responses even when they contain visual errors.</p>\n<p>Importantly, this is not just a perception problem. We show that even when a judge can correctly perceive the image, it may still anchor on the response text and fail to use its own visual evidence during evaluation.</p>\n<p>To mitigate this, we build <strong>PPJD</strong>, a dataset of controlled perceptual perturbations, and train <strong>Perception-Judge</strong> using GRPO with a verifiable batch-ranking reward. This helps the judge distinguish correct responses from visually wrong but fluent ones, leading to more perception-grounded and human-aligned multimodal evaluation.</p>\n","updatedAt":"2026-06-03T14:36:12.772Z","author":{"_id":"67b56f4d489d68b981dec6b1","avatarUrl":"/avatars/ad364e1bd512706939755e8f35cb1560.svg","fullname":"Seojeong Park","name":"sjpark5800","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8864057660102844},"editors":["sjpark5800"],"editorAvatarUrls":["/avatars/ad364e1bd512706939755e8f35cb1560.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.02578","authors":[{"_id":"6a1e4b5a808ddbc3c7d43cd7","user":{"_id":"67b56f4d489d68b981dec6b1","avatarUrl":"/avatars/ad364e1bd512706939755e8f35cb1560.svg","isPro":false,"fullname":"Seojeong Park","user":"sjpark5800","type":"user","name":"sjpark5800"},"name":"Seojeong Park","status":"claimed_verified","statusLastChangedAt":"2026-06-02T12:08:26.020Z","hidden":false},{"_id":"6a1e4b5a808ddbc3c7d43cd8","name":"Jiho Choi","hidden":false},{"_id":"6a1e4b5a808ddbc3c7d43cd9","name":"Junyong Kang","hidden":false},{"_id":"6a1e4b5a808ddbc3c7d43cda","name":"Seonho Lee","hidden":false},{"_id":"6a1e4b5a808ddbc3c7d43cdb","name":"Jaeyo Shin","hidden":false},{"_id":"6a1e4b5a808ddbc3c7d43cdc","name":"Hyunjung Shim","hidden":false}],"publishedAt":"2026-06-01T00:00:00.000Z","submittedOnDailyAt":"2026-06-03T00:00:00.000Z","title":"Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling","submittedOnDailyBy":{"_id":"67b56f4d489d68b981dec6b1","avatarUrl":"/avatars/ad364e1bd512706939755e8f35cb1560.svg","isPro":false,"fullname":"Seojeong Park","user":"sjpark5800","type":"user","name":"sjpark5800"},"summary":"Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers. We identify and systematically analyze this phenomenon, which we term Perceptual Judgment Bias. Through controlled visual perturbations, existing multimodal judges frequently anchor on the response text instead of their own visual perception, leading to inconsistent and non-verifiable evaluations. To address this issue, we introduce the Perceptually Perturbed Judgment Dataset, which constructs minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision. Building on this dataset, we develop a unified training framework that combines a structured GRPO-based reward with a batch-ranking objective, achieving coherent global ordering without explicit pairwise labels. Experiments across diverse MLLM-as-a-Judge benchmarks show that our approach substantially improves perceptual fidelity, ranking coherence, and alignment with human evaluation. Our results establish a scalable and generalizable pathway for training multimodal judges that are perceptually grounded, interpretable, and robust to visual-reasoning conflicts.","upvotes":1,"discussionId":"6a1e4b5b808ddbc3c7d43cdd","projectPage":"https://perception-judge.github.io/","githubRepo":"https://github.com/kaist-cvml/perception-judge","githubRepoAddedBy":"user","ai_summary":"Researchers identify a perceptual judgment bias in multimodal large language models where visual evidence is overlooked for textual plausibility, and propose a training framework using a perturbed dataset and reward modeling to improve perceptual fidelity and evaluation consistency.","ai_keywords":["multimodal large language models","perceptual judgment bias","visual perturbations","counterfactual responses","structured GRPO-based reward","batch-ranking objective","perceptual fidelity","ranking coherence","human evaluation alignment"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":2,"organization":{"_id":"6475760c33192631bad2bb38","name":"kaist-ai","fullname":"KAIST AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/aaZFiyXe1qR-Dmy_xq67m.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6983057214c5880cb86c7768","avatarUrl":"/avatars/740b6705ba9bc7bd33424bb43e153ecf.svg","isPro":false,"fullname":"Oliver Kowalski","user":"browser-kid","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6475760c33192631bad2bb38","name":"kaist-ai","fullname":"KAIST AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/aaZFiyXe1qR-Dmy_xq67m.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.02578.md"}">
Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
Abstract
Researchers identify a perceptual judgment bias in multimodal large language models where visual evidence is overlooked for textual plausibility, and propose a training framework using a perturbed dataset and reward modeling to improve perceptual fidelity and evaluation consistency.
Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers. We identify and systematically analyze this phenomenon, which we term Perceptual Judgment Bias. Through controlled visual perturbations, existing multimodal judges frequently anchor on the response text instead of their own visual perception, leading to inconsistent and non-verifiable evaluations. To address this issue, we introduce the Perceptually Perturbed Judgment Dataset, which constructs minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision. Building on this dataset, we develop a unified training framework that combines a structured GRPO-based reward with a batch-ranking objective, achieving coherent global ordering without explicit pairwise labels. Experiments across diverse MLLM-as-a-Judge benchmarks show that our approach substantially improves perceptual fidelity, ranking coherence, and alignment with human evaluation. Our results establish a scalable and generalizable pathway for training multimodal judges that are perceptually grounded, interpretable, and robust to visual-reasoning conflicts.
Community
We identify and analyze Perceptual Judgment Bias, a failure mode where multimodal LLM judges over-reward fluent or plausible responses even when they contain visual errors.
Importantly, this is not just a perception problem. We show that even when a judge can correctly perceive the image, it may still anchor on the response text and fail to use its own visual evidence during evaluation.
To mitigate this, we build PPJD, a dataset of controlled perceptual perturbations, and train Perception-Judge using GRPO with a verifiable batch-ranking reward. This helps the judge distinguish correct responses from visually wrong but fluent ones, leading to more perception-grounded and human-aligned multimodal evaluation.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.02578 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.02578 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.02578 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.