PRISM: Agentic Retrieval with LLMs for Multi-Hop Question Answering

AI summary

Overview

Research area: Natural Language Processing / information retrieval for multi-hop question answering (retrieval-augmented generation and LLM agent systems).

Technical level: Intermediate.

Scope: This paper introduces PRISM, an inference-time, training-free multi-agent retrieval framework that separates precision and recall into two collaborating LLM agents to build compact yet complete evidence sets for multi-hop QA, evaluated on four benchmarks.

What This Paper Is About

Multi-hop questions require combining several pieces of evidence scattered across different documents, so a retriever must avoid two opposite failures: including distracting passages (hurting precision) and missing a required passage (hurting recall). The authors argue that existing pipelines either propagate early errors, accumulate noisy context, or provide no mechanism to recover evidence that was left out of an initial selection. PRISM addresses this by splitting retrieval into three specialized LLM agents — a Question Analyzer, a Selector, and an Adder — that loop until the evidence set is both compact and comprehensive.

Key Contributions

1. PRISM framework: An agentic retrieval framework that gives explicit control over the precision–recall balance in evidence selection for multi-hop QA through an iterative Selector–Adder loop.
2. Better evidence sets: The authors show PRISM builds evidence sets that are both compact and comprehensive, achieving higher precision and recall than strong recent baselines on HotpotQA, 2WikiMultiHopQA, MuSiQue, and MultiHopRAG.
3. Stronger downstream QA: These evidence sets let a zero-shot LLM reader surpass full-context baselines and recent competitive methods on challenging multi-hop benchmarks, while relying on significantly less irrelevant information.
4. Model-agnostic, training-free design: The framework operates purely at inference time with off-the-shelf LLMs, and results are reported across GPT-4o, Gemini-2.5-Flash-Lite, and DeepSeek-Chat without task-specific fine-tuning.

Main Findings

• Passage recall gains: PRISM reaches 90.9% recall on HotpotQA versus 61.5% for OneR and 72.8% for IRCoT; 91.1% on 2WikiMultiHopQA versus OneR's 68.1% and IRCoT's 90.7%; and 83.2% on MuSiQue versus 44.6% (OneR) and 57.1% (IRCoT).
• MultiHopRAG retrieval: PRISM achieves 28.18 precision and 42.22 recall, compared with BM25 (11.09 precision / 24.13 recall), bge-large-en-v1.5 (16.12 / 32.32), RankGPT (GPT-4o) (17.99 / 36.01), and SETR-CoT & IRI (22.68 / 36.69) — roughly a 6-point gain in both recall and precision over SETR-CoT & IRI.
• End-to-end QA: With a zero-shot QA agent, PRISM scores 54.20 EM / 66.96 F1 on HotpotQA, 48.60 / 56.97 on 2Wiki, 31.17 / 41.78 on MuSiQue, and 49.16 accuracy on MultiHopRAG. Full-context (no retrieval) baselines score 44.18 / 58.28, 43.20 / 52.11, 19.77 / 29.42, and 44.37 respectively, while oracle context (gold evidence only) scores 64.80 / 77.83, 61.40 / 71.10, 38.78 / 50.89, and 57.04.
• Comparison with IRCoT: IRCoT QA reports 49.3 / 60.7 on HotpotQA, 57.7 / 68.0 on 2Wiki, and 26.5 / 36.5 on MuSiQue — so 2WikiMultiHopQA favors the recall-heavy IRCoT, while PRISM leads on the other benchmarks.
• Alternative LLM backends: With Gemini-2.5-Flash-Lite, PRISM records precision/recall of 87.24 / 93.46 (HotpotQA), 93.73 / 95.19 (2Wiki), 63.06 / 83.74 (MuSiQue) and EM/F1 of 56.93 / 70.37, 47.65 / 55.58, 36.64 / 44.52. With DeepSeek-chat, precision/recall is 79.46 / 95.88, 93.01 / 97.38, 67.11 / 89.38 and EM/F1 is 57.60 / 70.62, 43.39 / 54.75, 31.93 / 42.45. GPT-4o values are 83.02 / 90.90, 90.97 / 91.07, 47.46 / 83.17 with EM/F1 54.20 / 66.96, 48.60 / 56.97, 31.17 / 41.78.
• Against fine-tuned and compression methods: On HotpotQA, PRISM's 66.96 F1 beats RECOMP (37.91 F1, 28.20 EM) and RankRAG (46.70 F1, 35.30 EM), and is competitive with the fine-tuned CoRAG (69.80 F1, 56.30 EM); on 2Wiki, CoRAG leads at 72.50 EM / 77.30 F1 versus PRISM's 48.60 / 56.97, and ARM reports 71.70 F1 on 2Wiki; on MuSiQue, PRISM's 41.78 F1 is close to CoRAG's 42.40 and exceeds R1-Researcher (17.20 F1) and O2-Searcher (16.00 F1).
• Partial Match Accuracy: Compared with the full-context baseline, retrieved evidence improves PMA by 10 points on HotpotQA and more than 10 points on MuSiQue, with consistent gains on 2WikiMultiHopQA and MultiHopRAG.
• Ablation: The full model gets precision/recall 83.0 / 90.9 (HotpotQA, 2.67 passages), 83.2 / 91.1 (2Wiki, 2.74 passages), 47.7 / 83.2 (MuSiQue, 6.15 passages). Removing the Question Analyzer drops MuSiQue recall from 83.2 to 68.8. Selector-only drops recall to 79.7 (HotpotQA), 80.5 (2Wiki), and 69.3 (MuSiQue); a no-op Adder variant gives nearly identical numbers (80.0, 80.5, 67.7), indicating the gain comes from the Adder's recovery mechanism rather than extra computation.
• Iteration budget: Performance generally improves with more Selector–Adder iterations, but gains become marginal after N=4 and the agents' outputs converged after the third iteration, so N=3 was chosen for final experiments. Figure 4's sensitivity analysis used 100 HotpotQA samples.
• Fact-level retrieval: On HotpotQA, sentence-level precision/recall/F1 is 56.70 / 75.51 / 64.77; on 2WikiMultiHopQA it is 60.81 / 74.42 / 66.93.
• Passage-level QA: Answering from retrieved passages (rather than only supporting facts) gives 47.8 EM / 61.15 F1 on HotpotQA and 44.4 / 56.81 on 2WikiMultiHopQA, both above the full-context baseline.
• Error analysis (HotpotQA): Most retrieval failures are bridge questions (90.4%), as are most answer mismatches (83.8%); about 37% of answer mismatches occur despite perfect retrieval recall.
• Error analysis (2WikiMultiHopQA): Retrieval failures are 57.6% compositional, 17.9% bridge_comparison, 19.2% inference, and 5.2% comparison; answer mismatches are 62.7% compositional, 11.3% bridge_comparison, 21.0% inference, and 5.1% comparison, with roughly 33% occurring despite perfect recall.
• Error analysis (MuSiQue): Retrieval failures split 33.3% 2-hop, 41.4% 3-hop, 25.3% 4-hop; among the 181 answer-mismatch questions, 41.4% are 2-hop, 38.1% 3-hop, 20.4% 4-hop, and about 53% occurred despite perfect retrieval recall.
• Error analysis (MultiHopRAG): Retrieval failures are dominated by inference queries (42.6%), with comparison at 30.4% and temporal at 27.0%.
• Baseline limitation noted: Several baselines underperform even the full-context setting because their pipelines fail to preserve necessary supporting evidence.

Methodology in Plain English

The system works over a candidate pool of passages that is already supplied by the benchmark (for example, 10 candidates per question in HotpotQA and 20 in MuSiQue); PRISM does not do the first-stage corpus search itself — it targets the selection and refinement stage.

1. Question Analyzer. A prompted LLM decomposes the complex question into a list of sub-questions, each targeting one factual unit, making explicit what evidence is required (e.g., splitting "Which painter who shared a house with Vincent van Gogh was married to a Danish ceramist?" into "Who shared a house with Van Gogh?" and "Who was that person married to?").
2. Selector (precision). Given the sub-questions and the candidate passages, the Selector keeps only passages that directly support the query, producing a high-precision but possibly incomplete set.
3. Adder (recall). Seeing the same candidates plus what the Selector already picked, the Adder looks for anything important that was left out, such as a bridging passage needed to combine two facts. If the Selector was already perfect, the Adder outputs nothing new.
4. Iteration. The Selector–Adder cycle repeats up to N=3 times; evidence from all iterations is merged and de-duplicated.
5. Answer Generator. A zero-shot prompted LLM receives the original question plus the final evidence set and produces the answer.

All agents are prompted LLMs with structured outputs (lists of indices/titles) for reliable parsing; no fine-tuning or few-shot demonstrations are used. Experiments use GPT-4o by default, plus Gemini-2.5-Flash-Lite and DeepSeek-Chat.

Why This Matters

Impact on research: The paper reframes retrieval as an active, agent-driven process rather than a static preprocessing step, and shows that explicitly decoupling precision (Selector) and recall (Adder) into separate agents is a workable design pattern. It also documents that many retrieval baselines score below a plain full-context baseline, which is a useful diagnostic for the field.

Real-world applications:

• Enterprise or legal research assistants that must assemble multiple supporting documents before answering a compliance or case-law question.
• Scientific and biomedical literature QA, where an answer may require linking findings across several papers.
• News and fact-checking systems, since MultiHopRAG evaluation is over a news-article corpus with inference, comparison, and temporal queries.
• Any retrieval-augmented generation deployment where context windows are expensive and hallucination from noisy context is a risk.

Industry relevance: Because PRISM is training-free and operates purely at inference time over an existing candidate pool, it can be layered on top of an existing retriever without a training pipeline. The paper reports a favorable performance-to-cost trade-off relative to fine-tuned systems such as CoRAG on HotpotQA and MuSiQue, and the modular agent structure offers interpretability and control. The acknowledged cost is higher computation than single-pass retrieval, which matters for latency-sensitive production systems.

Future Directions

• Efficiency: Reducing the computational overhead of the multi-agent loop to make it viable for larger corpora and latency-sensitive settings.
• Adaptive iteration and uncertainty: Replacing the fixed iteration cap with adaptive iteration control or uncertainty-aware selection, since the agents converged after the third iteration and gains flattened after N=4.
• Scaling to open-domain retrieval: Integrating an external retriever to generate a manageable candidate pool, since directly applying PRISM to pools of thousands of passages is limited by LLM context windows.
• Domain adaptation: Tailoring retrieval strategies and prompts for scientific, biomedical, or legal corpora, and improving recovery of weakly connected or subtle reasoning chains, where the paper notes occasional misses and redundancy.
• Reducing reader-side errors: Error analysis shows a substantial share of remaining failures (about 37% on HotpotQA, 33% on 2Wiki, 53% on MuSiQue) come from the QA model rather than the retriever, raising the question of how retrieval improvements and reader improvements should be co-designed.

Target Audience

Researchers and practitioners working on retrieval-augmented generation, multi-hop question answering, and LLM agent design. It is most useful for readers with some familiarity with dense retrieval, reranking, and benchmarks such as HotpotQA and MuSiQue, since the paper assumes knowledge of metrics like Exact Match, F1, precision, and recall. Engineers building production RAG pipelines will find the modular, training-free architecture and the reported performance-to-cost comparisons directly actionable, while those focused on deep multi-hop reasoning methodology will be most interested in the ablation and error analysis sections.

Authors’ abstract

Retrieval plays a central role in multi-hop question answering (QA), where answering complex questions requires gathering multiple pieces of evidence. We propose PRISM, an agentic retrieval framework that leverages large language models (LLMs) in a structured loop to retrieve relevant evidence with high precision and recall. PRISM decomposes retrieval into three specialized agents: a Question Analyzer that breaks complex queries into sub-questions, a Selector that identifies the most relevant context for each sub-question (focusing on precision), and an Adder that brings in any missing evidence (focusing on recall). The iterative interaction between the Selector and Adder produces a compact yet comprehensive evidence set, avoiding both brittle error propagation and noisy context accumulation. It achieves higher retrieval accuracy while filtering out distracting content, enabling downstream QA models to surpass full-context answer accuracy while relying on significantly less irrelevant information. Experiments on four challenging multi-hop QA benchmarks, including HotpotQA, 2WikiMultiHopQA, MuSiQue, and MultiHopRAG, demonstrate that our approach consistently outperforms strong baselines.