ActiShade: Activating Overshadowed Knowledge to Guide Multi-Hop Reasoning in Large Language Models

AI summary

Overview

• Research area: Natural Language Processing; specifically retrieval-augmented generation (RAG), multi-hop question answering, and knowledge extraction from large language models.
• Technical level: Advanced. The paper assumes familiarity with dense retrieval, contrastive learning objectives, embedding perturbation, and iterative RAG pipelines.
• Scope: The paper proposes a multi-round retrieval framework called ActiShade that detects "overshadowed" keyphrases in a query, retrieves documents relevant to both the query and that keyphrase, and reformulates the next-round query, evaluated on three multi-hop QA benchmarks across three instruction-tuned LLMs.

The paper is affiliated with Beijing Institute of Technology, QiYuan Lab, Tsinghua University, and Huazhong University of Science and Technology.

What This Paper Is About

Multi-round retrieval-augmented generation systems for multi-hop questions typically feed text generated by the LLM back into the retriever as the next search query. The paper identifies a failure mode called knowledge overshadowing: when a query contains several conditions, a dominant one can crowd out the others, so the LLM silently ignores essential information, produces an incomplete or wrong intermediate query, retrieves irrelevant documents, and accumulates error over successive rounds. ActiShade's goal is to detect which piece of the query has been overlooked, retrieve evidence for it, and use that evidence to write a better next-round query.

Key Contributions

1. The ActiShade framework. A multi-round retrieval framework that iteratively detects shadowed content in a query, retrieves documents relevant to both the query and the shadowed keyphrase, and formulates a new query from the retrieved documents, with the stated aim of reducing error accumulation caused by knowledge overshadowing.

2. GaP (Gaussian perturbation-based method). A new detection method that adds Gaussian noise to the token embeddings of a candidate keyphrase rather than removing tokens, then measures how much the LLM's output distribution changes. It preserves the reasoning chain of the query, in contrast to the token-removal approach used by CoDA (Zhang et al. 2025).

3. A novel contrastive learning loss and query formulation strategy. The loss trains a dense retriever to rank three document types in a desired order — positive (relevant to the query and keyphrase), semi-positive (relevant to the query but not the keyphrase), and negative (irrelevant to both). The query formulation strategy then converts the retrieved evidence into the next-round query.

4. Comprehensive experiments. Evaluations on three multi-hop datasets (HotpotQA, 2WikiMQA, MuSiQue) across multiple LLMs showing improvements over state-of-the-art baselines, plus ablations of the detection method, the retriever training, and the document-selection step.

Main Findings

• ActiShade outperforms DRAGIN, the prior state-of-the-art, on every dataset and every backbone LLM tested. With Llama-3-8B-Instruct it reaches ACC 25.25 / F1 26.94 on MuSiQue, ACC 54.60 / F1 56.33 on HotpotQA, and ACC 45.80 / F1 46.02 on 2WikiMQA, versus DRAGIN's 21.11 / 22.61, 50.87 / 52.52, and 40.78 / 42.31 respectively.

• Gains hold for Qwen2.5 models. Qwen2.5-7B-Instruct with ActiShade: 22.80 / 26.11 (MuSiQue), 48.20 / 55.45 (HotpotQA), 52.80 / 50.47 (2WikiMQA). Qwen2.5-14B-Instruct with ActiShade: 25.59 / 27.47, 53.97 / 57.45, and 51.13 / 53.29. Performance generally improves with model size, which the authors cite as evidence of scalability; they report they could not test larger models due to hardware resource constraints.

• GaP beats CoDA in both single-round and multi-round settings. In the single-round F1 comparison, Direct-R scores 16.06 (MuSiQue), 44.89 (HotpotQA), 35.56 (2WikiMQA); adding CoDA gives 15.86 / 45.98 / 35.49, while adding GaP gives 17.83 / 46.81 / 38.67. In the multi-round setting, ActiShade-NoKOD scores 22.83 / 51.23 / 45.18; with CoDA it becomes 21.23 / 52.45 / 41.29, and with GaP the full pipeline reaches 26.94 / 56.33 / 46.02. On MuSiQue and 2WikiMQA, CoDA performs worse than no detection at all, which the authors attribute to its token-removal approach disrupting the reasoning chain.

• Moderate noise is best for detection. In a sensitivity analysis with Llama-3-8B-Instruct measured by F1, σ was varied over [0.05, 0.5]; performance peaks at σ = 0.1 and then declines. The authors state that excessive noise shifts output distributions for all candidate keyphrases, weakening detection, while overall performance remains relatively stable across a wide range of σ values.

• Fine-grained contrastive learning (FCL) improves retrieval. Recall@1 / Recall@3 for positive documents: Base 29.20 / 50.40, SCL 57.84 / 69.21, FCL 75.33 / 84.80. For semi-positive documents: Base 12.57 / 25.42, SCL 40.12 / 59.99, FCL 43.21 / 61.42. For positive & semi-positive combined: Base 18.29 / 36.78, SCL 38.21 / 50.29, FCL 38.14 / 52.72.

• Retriever training improves downstream QA. ActiShade scores F1 26.94 / 56.33 / 46.02; replacing fine-grained contrastive learning with standard contrastive learning gives 24.10 / 54.25 / 44.97, and removing it gives 25.68 / 53.89 / 44.61. The authors note that even without retriever training, ActiShade still outperforms previous baselines.

• Document selection matters. Removing the relevant document selection step drops F1 to 25.10 (MuSiQue), 55.58 (HotpotQA), and 42.48 (2WikiMQA) from 26.94 / 56.33 / 46.02.

• ActiShade also outperforms decomposition-free multi-hop retrievers. In an appendix comparison using F1: MDR 19.19 / 45.23 / 38.74, Beam Retrieval 19.79 / 47.37 / 38.98, GritHopper 22.13 / 50.76 / 41.30, versus ActiShade 26.94 / 56.33 / 46.02. The authors stress that these methods target one-shot retrieval accuracy while ActiShade targets end-to-end QA through retrieval–generation interaction, and that the two lines are not typically compared directly.

• Illustrative case. For "What is the name of the famous bridge in the birthplace of Gloria in D Major's composer?", the first round scores were Gloria 0.68, bridge 0.39, birthplace 0.59, composer 0.50; the second round after reformulation gave Antonio Vivaldi 0.63, bridge 0.15, birthplace 0.25; the third round gave Venice 0.79, bridge 0.34. The final answer produced was Rialto Bridge.

Methodology in Plain English

Detecting the overlooked keyphrase (GaP). The system extracts candidate keyphrases from the query using SpaCy, keeping named entities and tokens with POS tags in {NOUN, ADJ, VERB, PROPN, NUM, ADV} and removing stopwords. For each candidate it injects Gaussian noise into only that keyphrase's token embeddings while leaving every other token untouched, then runs the model to get a new output distribution. The original and perturbed distributions are average-pooled along the temporal dimension, and cosine similarity is computed between them. The candidate whose perturbation changes the output the least is declared overshadowed — a high similarity means the model was barely using that keyphrase anyway. Injecting noise rather than deleting tokens keeps the query structure intact.

Retrieving for the overlooked keyphrase. The authors train a dense retriever by fine-tuning contriever-msmarco. Training data is built from MuSiQue: the subject entity of the first sub-question is treated as the keyphrase; the supporting document for that first sub-question is the positive example, supporting documents for the other sub-questions are semi-positive (needed for the answer but not tied to the keyphrase), and everything else is negative. The loss combines two terms — one pushing positives above both semi-positives and negatives, and another separating semi-positives from negatives — with weighting coefficient α = 0.7, so the retriever learns the ranking D+ > D* > D−. At inference, the query is concatenated with the detected overshadowed keyphrase and used for retrieval.

Formulating the next query. The retrieved set is passed to the LLM, which judges each document for relevance and outputs "Yes" or "No"; the document with the highest probability of "Yes" is selected. The LLM then writes a new query from that document, making an implicit reasoning step explicit. Finally, the LLM decides whether the new query is single-hop — if so, one more retrieval round is performed and the process stops; otherwise it iterates, terminating also at a maximum iteration count. The initial question plus all documents retrieved during iteration are then fed to the LLM for the final answer.

Experimental setup. HotpotQA and 2WikiMQA contain two-hop questions; MuSiQue contains questions with 2 to 4 hops. All three use the same test set provided by IRCoT, containing 500 randomly sampled instances from the original development set. Backbone LLMs are Llama-3-8B-Instruct and Qwen2.5-Instruct (7B and 14B). For retriever training, 5,000 high-quality examples were manually selected from the MuSiQue training set, split into 3,500 for training, 750 for validation, and 750 for testing. Training used the AdamW optimizer with learning rate 5e-5 and batch size 32 for up to 20 epochs with early stopping on validation loss, on two NVIDIA A6000 GPUs. Evaluation uses Accuracy (Cover Exact Match) and F1. Baselines are Standard Prompting, CoT Prompting, one-time retrieval, IRCoT, Iter-RetGen, Self-Ask, FLARE, and DRAGIN. BeamAggR was excluded because it relies on multi-source knowledge, a different setting.

Why This Matters

• Research impact. The paper reframes a specific failure mode of iterative RAG — error accumulation from partial intermediate generations — as a diagnosable and correctable phenomenon, and offers a detection signal (perturbation-induced output-distribution similarity) that other pipelines could adopt. It also provides evidence that a token-removal detector (CoDA) can degrade multi-hop performance, and that training a retriever to respect three grades of relevance instead of two improves both retrieval and final answer quality.

• Real-world applications:

  • Multi-hop question answering over enterprise or legal document collections, where a question chains several conditions and dropping one silently yields a confident wrong answer.
  • Agentic search and tool-use assistants that iteratively rewrite their own queries; a shadowed-entity detector could catch a degenerating search loop early.
  • Clinical or scientific literature lookup, where an overlooked co-condition (a dosage, a population, a gene) can make a retrieved citation misleading.
  • Customer-support and technical troubleshooting assistants that must track several simultaneous constraints across turns.

• Industry relevance. ActiShade is a drop-in pattern on top of existing dense retriever plus LLM stacks. The retriever is fine-tuned from a widely used checkpoint (contriever-msmarco) on 3,500 examples, and the perturbation detector needs only forward passes over the input with altered embeddings, not retraining of the LLM. The reported improvements are consistent across three open-weight model families and three datasets, and the paper notes the retriever trained only on MuSiQue still generalizes to HotpotQA and 2WikiMQA, which matters for teams that cannot annotate retrieval data per domain.

Future Directions

• Scaling to larger backbones. The paper states that hardware resource constraints prevented implementing ActiShade on models larger than Qwen2.5-14B-Instruct; whether the gains persist, shrink, or grow at larger scales is left open.

• Reducing or removing retriever training cost. The retriever requires curated three-way relevance labels derived from sub-question annotations. Since ActiShade still beats baselines without retriever training, an open question is how far the detection and query formulation modules can carry a fully training-free pipeline, or whether the labels can be generated automatically.

• Replacing keyphrase granularity and the single-detection assumption. The method flags one overshadowed keyphrase per round, chosen from SpaCy-extracted noun/adjective/verb/proper-noun/numeral/adverb spans. More ambiguous queries may hide several simultaneously shadowed conditions or spans that this extraction step misses.

• Tuning and generalizing the detection signal. Detection quality is sensitive to the noise scale (peaking at σ = 0.1), and the mechanism is evaluated on three English multi-hop QA benchmarks. Whether the same perturbation signal transfers to other languages, to longer documents, or to non-QA reasoning tasks is not reported.

Target Audience

Researchers and practitioners working on retrieval-augmented generation, iterative retrieval, and multi-hop question answering will get the most from this paper, particularly those building agentic or multi-round search loops where intermediate model output becomes the next query. It also suits engineers who need a concrete recipe — a perturbation-based detector plus a three-tier contrastive retrieval objective — rather than a purely conceptual discussion of hallucination or context loss. Readers should be comfortable with dense retrieval, contrastive losses, and the mechanics of instruction-tuned LLM pipelines, since the paper assumes that background and does not restate it.

Authors’ abstract

In multi-hop reasoning, multi-round retrieval-augmented generation (RAG) methods typically rely on LLM-generated content as the retrieval query. However, these approaches are inherently vulnerable to knowledge overshadowing - a phenomenon where critical information is overshadowed during generation. As a result, the LLM-generated content may be incomplete or inaccurate, leading to irrelevant retrieval and causing error accumulation during the iteration process. To address this challenge, we propose ActiShade, which detects and activates overshadowed knowledge to guide large language models (LLMs) in multi-hop reasoning. Specifically, ActiShade iteratively detects the overshadowed keyphrase in the given query, retrieves documents relevant to both the query and the overshadowed keyphrase, and generates a new query based on the retrieved documents to guide the next-round iteration. By supplementing the overshadowed knowledge during the formulation of next-round queries while minimizing the introduction of irrelevant noise, ActiShade reduces the error accumulation caused by knowledge overshadowing. Extensive experiments show that ActiShade outperforms existing methods across multiple datasets and LLMs.