Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking

AI summary

Overview

Research area: Multilingual and multimodal entity linking; retrieval-augmented reasoning with vision-language models; cultural representation and the "long tail" of knowledge bases.

Technical level: Intermediate. The paper is readable without deep background in entity linking, but the rarity analysis and the controlled factorial experiments assume familiarity with retrieval-augmented generation and model evaluation.

Scope: The paper redefines entity "rarity" using knowledge-graph structure rather than popularity, and shows that a training-free pipeline combining a reasoning-capable vision-language model with iterative retrieval over Wikipedia beats the state of the art on the multilingual multimodal MERLIN benchmark, with the largest gains on rare entities.

What This Paper Is About

Entity linking maps a mention in text to an entry in a knowledge base, and multimodal entity linking additionally uses an image to disambiguate mentions that text alone cannot resolve. Prior work measures how "rare" an entity is mostly through popularity proxies such as Wikipedia pageviews or incoming links, and this paper argues that popularity misses a different and important kind of difficulty: entities that are culturally specific, sparsely documented, or sparsely connected in the knowledge graph. The goal is to characterize that broader notion of rarity, show that state-of-the-art systems fail on the entities it reveals, and then fix those failures with a simple framework where a reasoning model searches and reasons over Wikipedia before answering.

Key Contributions

1. A multidimensional characterization of entity rarity. The authors collect two families of metrics — Wikipedia-based measures of editorial attention and documentation depth (pageviews, backlinks, article size, revisions, unique editors, categories, external links, references, images), and Wikidata-based measures of structural connectivity and cross-lingual coverage (incoming links, outgoing links, language editions, statements, qualifiers, entity age). They show these definitions flag largely different entities, with mean pairwise overlap of only 37% between bottom-5% sets.

2. MERLIN-Rare. A set of rare-entity test slices for targeted evaluation, released alongside the framework, covering bottom-5% slices across the 15 rarity metrics.

3. A training-free reasoning-plus-retrieval framework. A two-module pipeline in which a reasoning-capable vision-language model iteratively issues searches over English Wikipedia (up to 20 iterations), incorporates retrieved snippets, and then a second pass re-prompts the model to extract the exact Wikipedia title. No fine-tuning is used.

4. A controlled analysis of reasoning versus retrieval. A factorial design across model size (2B, 4B, 8B), variant (Thinking vs Instruct), and retrieval condition (No RAG, BM25, Embedding) isolates what actually drives gains on rare entities, plus an error taxonomy of the best system's 841 errors.

Main Findings

• Rarity definitions capture different entities. The bottom-5% entity sets for different metrics share only 37% of entities on average; within-family overlap is moderate (Wikipedia 48%, Wikidata 38%) and between-family overlap is lower still (30%), with some cross-family pairs sharing as few as 10%.

• The state of the art degrades sharply on rare entities. Cultural Pangea, the current SOTA on MERLIN at 81.1% average accuracy, drops by 15.4% to 39.9% across the bottom-5% slices. Its drop is 37.7% on the pageview slice and 37.0% on the Wikidata statement-count slice, so popularity-only evaluation would miss many structurally sparse entities where the baseline suffers comparably.

• Reasoning and retrieval are complementary; neither alone is enough. Reasoning alone (Thinking vs Instruct, no retrieval) does not significantly improve rare-entity accuracy (p > 0.5 across all slices). Retrieval alone improves rare-entity accuracy but can hurt overall accuracy — BM25 hurts the 8B Instruct model by −4.9% on the full dataset while helping by +8.1% to +12.6% on structural rare-entity slices. The combination performs best.

• The retrieval benefit scales with rarity. For 8B-Think+Embed, the RAG delta grows from +3.8% on the full dataset to +18.8% on the language-editions slice, a 5.0× increase; similar jumps appear for statements (+16.6%), Wikidata outgoing links (+15.8%), and qualifiers (+15.7%).

• The best system sets a new state of the art. 8B-Think+Embed reaches 87.9% average accuracy on the full MERLIN test set, +6.9% over Cultural Pangea, with per-language gains of +10.0% on Hindi and Indonesian, +9.3% on Tamil, +2.9% on Vietnamese, and +2.1% on Japanese.

• Gains are largest on the hardest rare slices. Across all 15 rare-entity slices, gains range from +5.5% to +23.3%, and 14 of 15 exceed the full-dataset gain. The largest are qualifiers (+23.3%), statements (+22.1%), and Wikidata outgoing links (+21.7%). On language editions in the bottom 5%, Pangea reaches 47.6% while the proposed system reaches 63.9%.

• Reasoning models use retrieval differently. Thinking models make fewer but more deliberate searches (1.0–2.2 per example versus 3.2–5.2 for Instruct), generate 1,454–1,534 tokens between consecutive searches, and 56–58% of their search transitions are refinements adding disambiguation context, compared with 35–40% for the Instruct variant. Instruct models' queries degrade over iterations, with verbatim repetition climbing to 34% by search number 15+.

• Effective tool use is an emergent capability. At 2B scale, no-RAG baselines outperform all retrieval-augmented configurations, because 2B-Think issues only one (forced) search on 97% of examples. Retrieval-augmented reasoning starts paying off at 4B: 4B-Think+Embed (83.7%) matches 8B-Instr (83.5%) with half the parameters, though it uses roughly 2.8× more inference tokens than 8B-Instr without RAG. The two are virtually identical on the full dataset (+0.3%), but the smaller reasoning model wins by +5 to +7% on rare entities.

• Retrieval is the dominant bottleneck. Of the 841 errors made by 8B-Think+Embed, 436 (51.8%) are completely wrong, 180 (21.4%) are name format, 112 (13.3%) are disambiguation/specificity, 55 (6.5%) are Wikipedia variant, 45 (5.4%) are concept granularity, and 13 (1.5%) are pipeline errors. In 72% of errors the correct entity was never surfaced by search, and a further 23.5% involve the model engaging with the correct entity but rejecting it. Think+Embed has the fewest total errors (841 versus 1,493 for 8B-Inst+BM25) but the highest proportion of completely wrong cases.

• Rare-entity errors have a different shape than head-entity errors. Error rate is 31.1% on rare entities versus 8.4% on head entities (χ² = 40.84, p = 1.0 × 10⁻⁷). Rare-entity errors shift toward completely wrong (56.1% vs 48.9%), concept granularity (8.7% vs 3.0%), and name format (23.0% vs 20.3%), and away from disambiguation (9.0% vs 16.3%).

• Script matters for retrieval. For the embedding system, first-search hit rates are similar across scripts (37.0% Latin vs 39.2% non-Latin), but retrieval failure is a larger share of errors on non-Latin inputs (76.8% vs 62.5%), and embedding's advantage over BM25 is largest on non-Latin inputs and rare subsets (+6.9% vs +6.3% on rare subsets).

• Results are robust. The advantage over Pangea remains positive at 1%, 5%, and 10% rarity thresholds, ranging from +8.0% to +41.1%. Under redirect-aware scoring, the system rises from 87.9% to 89.5% and Pangea from 80.1% to 84.6%, leaving an advantage of 4.9% (paired bootstrap 95% CI [+4.01, +5.79]). The proposed system degrades less than Pangea on 14 of the 15 rare slices, with robustness gaps up to 16.4%; entity age is the single exception.

Methodology in Plain English

The authors first build a vocabulary for rarity. Using the Wikipedia and Wikidata APIs, they compute 15 metrics for every entity in the MERLIN test set, split into Wikipedia metrics (how much an entity has been written about) and Wikidata metrics (how well it is connected and how many languages cover it). An entity is "rare on metric m" if it falls in the bottom 5% of the test-set distribution for that metric. This lets them slice the test set 15 different ways and measure how a fixed model's accuracy changes.

They then evaluate the current state-of-the-art Cultural Pangea model on each slice, and separately build their own system. The system is deliberately simple and requires no training. A reasoning-capable vision-language model (from the Qwen3-VL family, in both Thinking and Instruct variants at 2B, 4B, and 8B sizes) receives the image, the passage, and the marked mention. It can call a search tool over English Wikipedia, either a BM25 lexical index or a semantic index built from multilingual embeddings in FAISS. It reasons iteratively, incorporating retrieved snippets, up to 20 iterations per example. Because small models often answered without searching, the first search call is forced in every retrieval configuration. A second pass then re-prompts the model with its own reasoning trace and asks it to output only the final Wikipedia title.

To isolate what causes the gains, they cross model size, reasoning versus instruct, and retrieval condition, giving 18 configurations, and add a retrieval-aware baseline by prepending the top-5 retrieved title-description pairs to Cultural Pangea's input, since Pangea does not support tool calling. Finally, they hand-analyze the best system's errors into a taxonomy and decompose them by pipeline stage.

Why This Matters

Impact on research. The paper argues that prior work has underestimated the difficulty of the cultural long tail because it equated rarity with low popularity. By showing that structural sparsity in Wikidata and limited cross-lingual coverage expose failure modes that pageview-based evaluation misses, and by releasing MERLIN-Rare, it gives the field a different lens for measuring progress on underrepresented entities. The controlled reasoning-versus-retrieval result also clarifies a common assumption: retrieval, not reasoning, supplies the missing information on rare entities, while reasoning is what lets the model use that information well.

Real-world applications:

• Multilingual search and question answering. Grounding mentions to knowledge-base entries is a foundational step for knowledge-intensive NLP, and the multilingual, multimodal setting targets exactly the cases where a user's query in Hindi, Tamil, Japanese, Indonesian, or Vietnamese may not have a well-covered English entry to match against.

• Cultural heritage and local information access. Entities that are well-documented within one language community but sparsely represented cross-lingually are the ones the framework targets, which is directly relevant to digital archives, reference platforms, and services serving underrepresented languages.

• Media and news pipelines. Many of the qualitatively analyzed examples are news events that are editorially rich but structurally sparse in Wikidata, a pattern typical of current-events content that annotation pipelines need to link reliably.

• Content moderation or recommendation systems that depend on accurate entity resolution, where failing on niche entities means failing consistently on the communities those entities belong to.

Industry relevance. The strongest practical signal is the 4B-Think+Embed result: a small reasoning model with retrieval matches an 8B instruct model overall and outperforms it by +5 to +7% on rare entities, trading fewer parameters for roughly 2.8× more inference tokens. For teams deploying at scale, that is a concrete alternative to simply scaling up model size. The finding that 2B models cannot use the search tool effectively also sets a practical floor on model scale for tool-augmented pipelines. The 72% retrieval-failure share indicates that investment in cross-lingual retrieval quality, rather than reasoning capability, is where the remaining headroom lies.

Future Directions

• Improving cross-lingual retrieval. Retrieval failure accounts for 72% of residual errors, and neither BM25 nor embedding retrieval guarantees recall when transliterations diverge between the source language and English Wikipedia titles. The authors call this the most impactful direction for future work.

• Testing beyond English Wikipedia and beyond these five languages. All MERLIN entities have English articles by construction. Generalization to languages with even sparser Wikipedia representation, such as African languages, is unknown.

• Establishing whether the reasoning-by-retrieval interaction generalizes across model families. A GLM check in the appendix shows retrieval in thinking mode improves all 15 rare slices, but GLM's non-thinking mode could not sustain the retrieval loop, so the full interaction is not established family-independently.

• Closing the disambiguation gap on the long tail. In 23.5% of errors the model surfaced and engaged with the correct entity in its reasoning but rejected it, which the authors present as an open problem even for reasoning-native models.

Target Audience

Researchers and practitioners working on entity linking, retrieval-augmented generation, multilingual and multimodal NLP, and cultural representation in knowledge bases. It is also relevant to evaluation-focused readers interested in how the choice of a rarity metric shapes which model failures are visible, and to engineers considering reasoning models with tool use as a smaller-parameter alternative to scaling up.

Authors’ abstract

Multimodal entity linking grounds entity mentions in text and images to knowledge-base entries. These systems degrade on rare entities, but prior work measures rarity primarily through popularity-based metrics such as pageviews. We broaden this view using knowledge-graph structural metrics that capture how well an entity is documented and connected. These metrics identify many rare entities that popularity metrics miss. Across the resulting rare-entity slices, state-of-the-art accuracy drops by 15.4-39.9%, showing that different rarity definitions expose different failure modes. To address these failures, we introduce a simple, training-free framework in which a reasoning-capable vision-language model iteratively searches and reasons over Wikipedia, gathering evidence dynamically. Controlled experiments show that reasoning and retrieval are complementary. Reasoning alone does not significantly improve accuracy on rare entities. Retrieval without reasoning improves rare-entity accuracy but can hurt overall accuracy. Their combination performs best. On MERLIN, a multilingual multimodal entity linking benchmark over five languages (Hindi, Indonesian, Japanese, Tamil, Vietnamese), our best system improves over the state of the art by 6.9% overall and by up to 23.3% on rare-entity slices. We release MERLIN-Rare, rare-entity test slices for targeted evaluation, with our framework.