Hugging Face Daily Papers · September 3, 2026 · 4 min read

Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

We introduce KBMR, an MLLM-based retriever designed for Knowledge-Based Visual Question Answering (KB-VQA). Instead of relying on surface-level visual similarity, KBMR learns entity-aligned semantic representations and leverages an MLLM-based semantic discriminator with continuous semantic distillation for more reliable retrieval. Experiments on E-VQA, InfoSeek, and OK-VQA demonstrate substantial improvements in both retrieval recall and end-to-end VQA performance. Code is available now — welcome to check it out! 🚀</p>\n","updatedAt":"2026-09-03T08:43:53.211Z","author":{"_id":"6954da80bc38ca0baf130a68","avatarUrl":"/avatars/e43cd2bd28c591dc0deabd808dc27e79.svg","fullname":"Hangrui Xu","name":"Hangrui-Xu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8198221325874329},"editors":["Hangrui-Xu"],"editorAvatarUrls":["/avatars/e43cd2bd28c591dc0deabd808dc27e79.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.21450","authors":[{"_id":"6a993158fea81827432200c7","user":{"_id":"6954da80bc38ca0baf130a68","avatarUrl":"/avatars/e43cd2bd28c591dc0deabd808dc27e79.svg","isPro":false,"fullname":"Hangrui Xu","user":"Hangrui-Xu","type":"user","name":"Hangrui-Xu"},"name":"Hangrui Xu","status":"claimed_verified","statusLastChangedAt":"2026-09-03T08:45:04.435Z","hidden":false},{"_id":"6a993158fea81827432200c8","name":"Zhengxian Wu","hidden":false},{"_id":"6a993158fea81827432200c9","name":"Yunyao Yu","hidden":false},{"_id":"6a993158fea81827432200ca","name":"Zhuohong Chen","hidden":false},{"_id":"6a993158fea81827432200cb","name":"Rui Cong","hidden":false},{"_id":"6a993158fea81827432200cc","name":"Xiangwen Deng","hidden":false},{"_id":"6a993158fea81827432200cd","name":"Zhifang Liu","hidden":false},{"_id":"6a993158fea81827432200ce","name":"Peng Jiao","hidden":false},{"_id":"6a993158fea81827432200cf","name":"Haoqian Wang","hidden":false}],"publishedAt":"2026-08-19T00:00:00.000Z","submittedOnDailyAt":"2026-09-03T00:00:00.000Z","title":"Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering","submittedOnDailyBy":{"_id":"6954da80bc38ca0baf130a68","avatarUrl":"/avatars/e43cd2bd28c591dc0deabd808dc27e79.svg","isPro":false,"fullname":"Hangrui Xu","user":"Hangrui-Xu","type":"user","name":"Hangrui-Xu"},"summary":"Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.","upvotes":14,"discussionId":"6a993159fea81827432200d0","ai_summary":"KBMR uses a multimodal language model to embed images by semantic identity rather than surface appearance, improving retrieval and visual question answering via continuous distillation and hard negative sampling.","ai_keywords":["KB-VQA","CLIP-style dual encoders","MLLM-based embedding retriever","autoregressive capabilities","semantic discriminator","continuous entity-consistency weights","continuous semantic distillation","hard negative sampling"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"628735cbc83a2d6ab8d14a66","name":"Tsinghua","fullname":"Tsinghua University","avatar":"https://www.gravatar.com/avatar/6c5c1441e3283e7543342e59277ea219?d=retro&size=100"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6954da80bc38ca0baf130a68","avatarUrl":"/avatars/e43cd2bd28c591dc0deabd808dc27e79.svg","isPro":false,"fullname":"Hangrui Xu","user":"Hangrui-Xu","type":"user"},{"_id":"68418d5eb64ba498927203b0","avatarUrl":"/avatars/6cc11ad4fba75860b2293df092400028.svg","isPro":false,"fullname":"YUYUNYAO","user":"Yyy195","type":"user"},{"_id":"6a831adee158190a83fe7a00","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a831adee158190a83fe7a00/18AtdnwoNPG1gf1BQ8ClB.jpeg","isPro":false,"fullname":"Timofey Mikhailov","user":"TimofeyMikhailov","type":"user"},{"_id":"6a84ab4a0a617a7f69df064e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a84ab4a0a617a7f69df064e/R4N3PGgVST7pps2LG2EfL.jpeg","isPro":false,"fullname":"Vincent L. Nguyen","user":"VincentNguyensen","type":"user"},{"_id":"6a849e96b6f17993fab28561","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a849e96b6f17993fab28561/AGcNVwRKxAg4YPje0g9kf.jpeg","isPro":false,"fullname":"오수빈","user":"hyunwooyoon","type":"user"},{"_id":"6a8cce682a864d8531b837d1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a8cce682a864d8531b837d1/OtOQZuhf5rdZie_Gzzr22.jpeg","isPro":false,"fullname":"Kenny Siregar","user":"itskennysiregar","type":"user"},{"_id":"6a8376bb42ceb5979a2709b6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a8376bb42ceb5979a2709b6/IM9wQEg3eJwkE_f5LaKFc.jpeg","isPro":false,"fullname":"Kaito Isikawa","user":"Kuisikawa","type":"user"},{"_id":"6a8d648b86120332b784d374","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a8d648b86120332b784d374/VWwp3cDi6rxLWFpSYz7l4.jpeg","isPro":false,"fullname":"J. Perez","user":"joshuaperezson","type":"user"},{"_id":"6a8c6ec2c32617d725bd635c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a8c6ec2c32617d725bd635c/EdP-szyUFcORd1juwa-q9.jpeg","isPro":false,"fullname":"Daniel Johnson","user":"Dqjohnson","type":"user"},{"_id":"6a81bad9997888cc8321a4a6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a81bad9997888cc8321a4a6/4TEuczIV5OD1dgGWrjYRs.jpeg","isPro":false,"fullname":"윤지민","user":"Yijungkook","type":"user"},{"_id":"6a8d7e29f79586668cf46aa4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a8d7e29f79586668cf46aa4/HljQw6amTuP80o5TNJAuw.jpeg","isPro":false,"fullname":"Samuel Adeyemi","user":"SAMUELADEYEMI","type":"user"},{"_id":"6a8b4f1861e393d86330bb6d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a8b4f1861e393d86330bb6d/NDrNTXTf-IJIOImSSi68r.jpeg","isPro":false,"fullname":"Tyler Smith","user":"tylersmit","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"628735cbc83a2d6ab8d14a66","name":"Tsinghua","fullname":"Tsinghua University","avatar":"https://www.gravatar.com/avatar/6c5c1441e3283e7543342e59277ea219?d=retro&size=100"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.21450.md","query":{}}">

Papers

arxiv:2608.21450

Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering

Published on Aug 19

· Submitted by

Hangrui Xu on Sep 3

Tsinghua University

Upvote

Authors:

Hangrui Xu ,

Abstract

KBMR uses a multimodal language model to embed images by semantic identity rather than surface appearance, improving retrieval and visual question answering via continuous distillation and hard negative sampling.

Generated by thinkingmachines/Inkling-Small

Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.