Leveraging the Power of Large Language Models in Entity Linking via Adaptive Routing and Targeted Reasoning

AI summary

Overview

Research area: Natural Language Processing, specifically Entity Linking (also called Named Entity Disambiguation) and the cost-efficient integration of large language models into structured NLP pipelines.

Technical level: Intermediate. The paper assumes familiarity with entity linking terminology (mentions, candidates, knowledge bases), embeddings, and LLM prompting, but the pipeline itself is described at a level an NLP practitioner or graduate student can follow without specialized background.

Scope: The paper introduces ARTER, a routing-based entity linking pipeline that sends only contextually difficult mentions to an LLM for reasoning while resolving the rest with a frozen traditional linker, and evaluates it on six standard benchmarks.

What This Paper Is About

Entity linking is the task of mapping an ambiguous text mention (for example, "Target" or "Washington") to the correct entry in a knowledge base, such as Wikipedia. Traditional systems need large annotated datasets and heavy fine-tuning, while newer LLM-based prompting approaches handle hard cases better but are expensive because they call a reasoning model for every single mention. ARTER's goal is to get most of the accuracy benefit of LLM reasoning while paying the LLM cost on only the mentions that actually need it.

Key Contributions

1. ARTER pipeline design: A structured, fine-tuning-free entity linking pipeline that combines ReFinED-based candidate generation, multi-signal candidate scoring, a lightweight router, and selective LLM-based reasoning, with ReFinED kept frozen throughout.

2. A learned difficulty router: A Random Forest classifier trained on four complementary signals (three cosine-similarity scores from a Sentence Transformer encoder plus a confidence score from a small non-reasoning LLM) that labels mentions as "easy" or "hard" using ReFinED's predictions on the AIDA training set as supervision.

3. Empirical validation on six benchmarks: Evaluation on AIDA, MSNBC, ACE2004, AQUAINT, CWEB, and WIKI showing that selective LLM reasoning improves over the ReFinED baseline by up to +4.47% while matching full-prompting pipelines within roughly 1% on most datasets.

4. Deployment efficiency analysis: Token-level accounting with and without the router, plus a cost-performance comparison across Claude 3 Haiku, Claude 3.5 Sonnet, Claude Opus 4, GPT-4.1, and DeepSeek, reporting an average 58.25% reduction in LLM tokens and a 59.25% reduction in output tokens.

Main Findings

• Accuracy gains over the traditional baseline: ARTER outperforms ReFinED by up to +4.47% (MSNBC), with an average gain of +2.53% on 5 out of 6 datasets. Per-dataset gaps versus ReFinED are +3.52% on ACE2004, +0.13% on AQUAINT, +2.46% on AIDA, +4.47% on MSNBC, +2.09% on CWEB, and −2.37% on WIKI.

• The WIKI exception is explained, not hidden: ReFinED is the only method that beats ARTER on WIKI (85.98% versus the best ARTER configuration), and the authors attribute this to ReFinED having been trained specifically on the full English Wikipedia, giving it a natural advantage on that dataset.

• Comparable to full prompting at much lower cost: Compared with pipelines that reason over every mention, ARTER is within roughly 1% on most datasets. The gap between ARTER's best configuration and the full-prompting best is −0.78% on ACE2004, −0.98% on AQUAINT, −0.28% on AIDA, −0.41% on MSNBC, −1.16% on CWEB, and +0.16% on WIKI.

• Most mentions never reach the LLM: The router classifies between 50.4% (ACE2004) and 67.3% (WIKI) of mentions as easy. Counts include 129 easy mentions out of 259 for ACE2004, 411 of 743 for AQUAINT, 2,967 of 4,464 for AIDA, 390 of 656 for MSNBC, 6,592 of 6,821 for CWEB, and 4,555 of 11,154 for WIKI. Hard mentions make up 32.7% to 49.6% of each dataset.

• Router behavior is deliberately asymmetric: With a decision threshold of τ = 0.735, selected on the AIDA validation set (4,784 instances) by maximizing Youden's J statistic, the router achieves an average AUC of 73.8, accuracy of 65.1%, F1 of 75.3, and an average easy-subset accuracy of 89.3% across the six test datasets. Per-dataset easy accuracy ranges from 0.791 (CWEB) to 0.938 (ACE2004), while hard accuracy ranges from 0.194 (AQUAINT) to 0.444 (CWEB). The design intentionally prioritizes easy-subset accuracy, since misrouting a hard mention into the fast path likely produces a wrong link, whereas misrouting an easy mention into the reasoning path costs a little compute but still links correctly.

• Token savings are substantial and skewed toward output: Input token reduction ranges from 48.28% (ACE2004) to 65.14% (AIDA), and output token reduction ranges from 49.01% (ACE2004) to 73.22% (AIDA), for averages of 58.25% and 59.25% respectively.

• Few-shot plus Chain-of-Thought is the strongest prompt: On ACE2004 hard cases with Claude 3 Haiku, few-shot + CoT reached 81.30%, versus 79.46% for contrastive prompting, 76.79% for few-shot, 76.79% for ReAct-style prompting, and 75.89% for zero-shot.

• Cheaper models are the better buy: Using observed usage of 20.06 million input tokens and 0.80 million output tokens, Claude 3 Haiku delivered 87.5% accuracy at $6.01, while DeepSeek was cheapest overall at $4.55 with 85.58% accuracy. Premium models cost far more for marginal gains: Claude 3.5 Sonnet at $72.14, GPT-4.1 at $46.50, and Claude Opus 4 at $360.72.

• Router feature ablation: Removing the LLM-driven confidence score φ (score_4) changes accuracy by −1.0% on the easy subset and +1.0% on the hard subset. Because the routing objective prioritizes easy accuracy and φ is cheap to compute with a lightweight model, it was retained.

• Router feature costs are negligible: Generating the LLM-based router feature across all datasets consumed 55.29 million input tokens and 2.80 million output tokens, totaling an estimated $2.99 at Groq pricing of $0.05 per 1M input tokens and $0.08 per 1M output tokens.

Methodology in Plain English

The pipeline works in four stages.

Candidate generation. For each mention, the frozen ReFinED candidate-generation module returns 30 ranked candidate entities from English Wikipedia. Structured metadata such as entity titles and short descriptions is pulled from the linked Wikidata entries to enrich each candidate.

Signal computation. The router looks only at the top 10 of those 30 candidates and computes four scores for each. Three are cosine similarities from a Sentence Transformer encoder: (1) similarity between the mention's surrounding context and the candidate's title+description, (2) similarity between the mention surface form and the candidate, and (3) the maximum similarity between the candidate and any other candidate, which captures how confusable the candidates are with each other. The fourth is a confidence score from a small, non-reasoning model (Llama-3.1-8B-Instruct), which sees the mention, its context, and the top-10 candidates in a single-turn prompt and returns a number between 0.0 and 1.0 per candidate. Peak scores are computed as (θ₁ + θ₂ − θ₃ + φ)/3, deliberately subtracting θ₃ to penalize candidates that resemble each other.

Routing. The per-candidate scores are collapsed into ten aggregate features: top1, top2, margin (top1 − top2), Shannon entropy (base-2) of the flattened score distribution, number of candidates, sentence length in tokens, and the averages of the four scores. A Random Forest classifier, trained on labels derived from whether ReFinED's prediction matched the gold entity on the 18,395 AIDA training instances, decides easy or hard. Easy mentions go straight to ReFinED; hard mentions go to the reasoning module.

Targeted reasoning. Hard mentions are turned into a multiple-choice question over the top 30 candidates. The winning prompt format pairs few-shot examples with Chain-of-Thought reasoning, and the model is instructed to return −1 if no candidate fits. The authors tested DeepSeek R1 0528, GPT-4.1, Claude 3 Haiku, and Claude 3.5 Sonnet as reasoning models.

Evaluation used Disambiguation Accuracy, defined as TP_link / (TP_link + FP_link + FN_link), across the six benchmark datasets. Token accounting used the tiktoken library with cl100k_base encoding, which is the tokenizer for ChatGPT-3.5 and GPT-4-turbo and serves as a proxy for Claude models, which have no public tokenizer. Tokens from the 8B routing model were excluded from the deployment-efficiency calculation because of their negligible cost.

Why This Matters

Impact on research. The paper reframes LLM integration for entity linking as a resource allocation problem rather than a prompting problem. It shows that selectively applying reasoning based on measurable mention difficulty can recover most of the accuracy of full prompting, and it provides a concrete feature set and ablation showing that inter-candidate similarity (avg θ₃) is the single most influential routing signal, followed by mention–entity similarity (avg θ₂). This gives future work a baseline for studying adaptive computation in structured NLP pipelines and a reason to doubt the assumption that every mention deserves a reasoning call.

Real-world applications.

• Search engines, where entity linking is used to map query mentions to knowledge base entries and where per-query latency and cost budgets are tight.
• Automated knowledge extraction platforms, which process large document streams and cannot afford an LLM call per mention.
• Question-answering systems, where accurate linking of ambiguous entities directly determines answer correctness.
• Large-scale document processing, where the routing design keeps most mentions on a fast path and prevents latency spikes and cost surges.

Industry relevance. The paper comes from a collaboration between the University of Massachusetts Amherst and Apple, and the framing is explicitly operational: avoid domain-specific fine-tuning, keep a frozen model for high-recall candidate generation, and reserve LLM spend for the hard fraction. The cost tables make this concrete, showing that Claude 3 Haiku reaches 87.5% accuracy for $6.01 while Claude Opus 4 costs $360.72 on the same token volume. The router itself adds only about $2.99 in feature-generation cost across all datasets, so the overhead of deciding is far below the cost of reasoning indiscriminately.

Future Directions

• Broaden the LLM comparison in full-prompting settings. The authors state that computational and budget constraints prevented benchmarking a wider range of large language models under full prompting.

• Test LLMs for candidate generation. The paper leaves unexplored the use of LLMs to generate entity candidates directly, which would enable an ablation separating the contribution of generation from that of reranking.

• Study router parameter configurations systematically. The effect of different router settings on accuracy has not been studied, and the authors note that the exact trade-offs depend on router configuration.

• Understand when the fast path should not be trusted. The WIKI result, where ReFinED beats all ARTER configurations because it was trained on the full English Wikipedia, raises the open question of how routing should behave when the underlying fast model has a domain advantage the LLM does not.

Target Audience

NLP researchers working on entity linking, entity disambiguation, or adaptive computation who want a baseline for selectively invoking LLMs. Also machine learning engineers and applied scientists deploying entity linking at scale who need to balance accuracy against inference cost and latency, and graduate students studying the trade-offs between fine-tuning, prompting, and hybrid pipelines. Readers primarily interested in prompt engineering alone will find less here, since the paper's central claim is about routing and efficiency rather than prompt design, and the strongest prompt strategy it reports (few-shot plus CoT) is already established practice.

Authors’ abstract

Entity Linking (EL) has traditionally relied on large annotated datasets and extensive model fine-tuning. While recent few-shot methods leverage large language models (LLMs) through prompting to reduce training requirements, they often suffer from inefficiencies due to expensive LLM-based reasoning. ARTER (Adaptive Routing and Targeted Entity Reasoning) presents a structured pipeline that achieves high performance without deep fine-tuning by strategically combining candidate generation, context-based scoring, adaptive routing, and selective reasoning. ARTER computes a small set of complementary signals(both embedding and LLM-based) over the retrieved candidates to categorize contextual mentions into easy and hard cases. The cases are then handled by a low-computational entity linker (e.g. ReFinED) and more expensive targeted LLM-based reasoning respectively. On standard benchmarks, ARTER outperforms ReFinED by up to +4.47%, with an average gain of +2.53% on 5 out of 6 datasets, and performs comparably to pipelines using LLM-based reasoning for all mentions, while being as twice as efficient in terms of the number of LLM tokens.