Previous work usually treats entity rarity as popularity. We propose a broader view of rarity with <strong>15 Wikipedia and Wikidata notions of rarity</strong> spanning attention, documentation, knowledge-graph structure, and cross-lingual coverage. These notions identify very different entities and they reveal that existing evaluations substantially underestimate the long tail. State-of-the-art accuracy drops by 15.4–39.9% across the rare-entity slices they uncover.</p>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/62886c851f542794efe6548d/sf5uNzFO-1FgulMv5E7ty.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/62886c851f542794efe6548d/sf5uNzFO-1FgulMv5E7ty.png\" alt=\"image\"></a></p>\n<p>A simple training-free framework in which a vision-language model iteratively searches and reasons over Wikipedia, recovers a lot of this performance. Controlled experiments show that <strong>reasoning and retrieval are complementary</strong>: reasoning alone does not significantly improve rare-entity accuracy, while retrieval without reasoning helps rare entities but can hurt overall accuracy. Combining them performs best, improving multilingual multimodal entity linking by 6.9% overall and by up to 23.3% on rare-entity slices.</p>\n<p>We release <a href=\"https://huggingface.co/datasets/neulab/merlin-rare\">MERLIN-Rare</a>: 1,105 mentions across five languages, augmented with 15 rarity metrics, complete model predictions, and reasoning and search traces.</p>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/62886c851f542794efe6548d/lXLg4-J5W4DwD4IqR8AQx.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/62886c851f542794efe6548d/lXLg4-J5W4DwD4IqR8AQx.png\" alt=\"image\"></a></p>\n","updatedAt":"2026-09-11T18:45:16.374Z","author":{"_id":"62886c851f542794efe6548d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62886c851f542794efe6548d/GzGAjr_vncJN4FdJ0TaXi.jpeg","fullname":"Parinthapat Pengpun","name":"parinzee","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":22,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8489982485771179},"editors":["parinzee"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/62886c851f542794efe6548d/GzGAjr_vncJN4FdJ0TaXi.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.10745","authors":[{"_id":"6aa351cd47a406da7901e688","user":{"_id":"62886c851f542794efe6548d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62886c851f542794efe6548d/GzGAjr_vncJN4FdJ0TaXi.jpeg","isPro":false,"fullname":"Parinthapat Pengpun","user":"parinzee","type":"user","name":"parinzee"},"name":"Parinthapat Pengpun","status":"claimed_verified","statusLastChangedAt":"2026-09-11T08:45:04.696Z","hidden":false},{"_id":"6aa351cd47a406da7901e689","name":"Simran Khanuja","hidden":false},{"_id":"6aa351cd47a406da7901e68a","name":"Graham Neubig","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/62886c851f542794efe6548d/mMcB42BgYIfqvFvTwSJ4g.png"],"publishedAt":"2026-09-09T00:00:00.000Z","submittedOnDailyAt":"2026-09-11T00:00:00.000Z","title":"Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking","submittedOnDailyBy":{"_id":"62886c851f542794efe6548d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62886c851f542794efe6548d/GzGAjr_vncJN4FdJ0TaXi.jpeg","isPro":false,"fullname":"Parinthapat Pengpun","user":"parinzee","type":"user","name":"parinzee"},"summary":"Multimodal entity linking grounds entity mentions in text and images to knowledge-base entries. These systems degrade on rare entities, but prior work measures rarity primarily through popularity-based metrics such as pageviews. We broaden this view using knowledge-graph structural metrics that capture how well an entity is documented and connected. These metrics identify many rare entities that popularity metrics miss. Across the resulting rare-entity slices, state-of-the-art accuracy drops by 15.4-39.9%, showing that different rarity definitions expose different failure modes. To address these failures, we introduce a simple, training-free framework in which a reasoning-capable vision-language model iteratively searches and reasons over Wikipedia, gathering evidence dynamically. Controlled experiments show that reasoning and retrieval are complementary. Reasoning alone does not significantly improve accuracy on rare entities. Retrieval without reasoning improves rare-entity accuracy but can hurt overall accuracy. Their combination performs best. On MERLIN, a multilingual multimodal entity linking benchmark over five languages (Hindi, Indonesian, Japanese, Tamil, Vietnamese), our best system improves over the state of the art by 6.9% overall and by up to 23.3% on rare-entity slices. We release MERLIN-Rare, rare-entity test slices for targeted evaluation, with our framework.","upvotes":1,"discussionId":"6aa351ce47a406da7901e68b","projectPage":"https://neulab.github.io/think-before-you-link/","githubRepo":"https://github.com/neulab/think-before-you-link","githubRepoAddedBy":"user","ai_summary":"A reasoning-capable vision-language model that iteratively retrieves and reasons over Wikipedia improves multimodal entity linking for rare entities defined by knowledge-graph structure.","ai_keywords":["multimodal entity linking","knowledge-graph structural metrics","vision-language model","iterative retrieval","reasoning","MERLIN"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":2,"organization":{"_id":"62cd831b7a036fc9941c147a","name":"neulab","fullname":"NeuLab @ LTI/CMU","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1657821011553-62cd8153248f9e6bc20ab250.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"62886c851f542794efe6548d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62886c851f542794efe6548d/GzGAjr_vncJN4FdJ0TaXi.jpeg","isPro":false,"fullname":"Parinthapat Pengpun","user":"parinzee","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"62cd831b7a036fc9941c147a","name":"neulab","fullname":"NeuLab @ LTI/CMU","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1657821011553-62cd8153248f9e6bc20ab250.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.10745.md","query":{}}">
Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking
Abstract
A reasoning-capable vision-language model that iteratively retrieves and reasons over Wikipedia improves multimodal entity linking for rare entities defined by knowledge-graph structure.
Multimodal entity linking grounds entity mentions in text and images to knowledge-base entries. These systems degrade on rare entities, but prior work measures rarity primarily through popularity-based metrics such as pageviews. We broaden this view using knowledge-graph structural metrics that capture how well an entity is documented and connected. These metrics identify many rare entities that popularity metrics miss. Across the resulting rare-entity slices, state-of-the-art accuracy drops by 15.4-39.9%, showing that different rarity definitions expose different failure modes. To address these failures, we introduce a simple, training-free framework in which a reasoning-capable vision-language model iteratively searches and reasons over Wikipedia, gathering evidence dynamically. Controlled experiments show that reasoning and retrieval are complementary. Reasoning alone does not significantly improve accuracy on rare entities. Retrieval without reasoning improves rare-entity accuracy but can hurt overall accuracy. Their combination performs best. On MERLIN, a multilingual multimodal entity linking benchmark over five languages (Hindi, Indonesian, Japanese, Tamil, Vietnamese), our best system improves over the state of the art by 6.9% overall and by up to 23.3% on rare-entity slices. We release MERLIN-Rare, rare-entity test slices for targeted evaluation, with our framework.
Community
Previous work usually treats entity rarity as popularity. We propose a broader view of rarity with 15 Wikipedia and Wikidata notions of rarity spanning attention, documentation, knowledge-graph structure, and cross-lingual coverage. These notions identify very different entities and they reveal that existing evaluations substantially underestimate the long tail. State-of-the-art accuracy drops by 15.4–39.9% across the rare-entity slices they uncover.
A simple training-free framework in which a vision-language model iteratively searches and reasons over Wikipedia, recovers a lot of this performance. Controlled experiments show that reasoning and retrieval are complementary: reasoning alone does not significantly improve rare-entity accuracy, while retrieval without reasoning helps rare entities but can hurt overall accuracy. Combining them performs best, improving multilingual multimodal entity linking by 6.9% overall and by up to 23.3% on rare-entity slices.
We release MERLIN-Rare: 1,105 mentions across five languages, augmented with 15 rarity metrics, complete model predictions, and reasoning and search traces.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2609.10745 in a model README.md to link it from this page.
Cite arxiv.org/abs/2609.10745 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.