PluriHopRAG: Exhaustive, Recall-Sensitive QA Through Corpus-Specific Document Structure Learning

AI summary

Overview

Research area: Natural Language Processing — Retrieval-Augmented Generation (RAG) and question answering over document corpora.

Technical level: Intermediate. Readers benefit from familiarity with RAG pipelines, retrieval, embeddings, and LLM fine-tuning, though the core problem is stated in accessible terms.

Scope: The paper defines a new class of questions ("pluri-hop"), introduces a diagnostic benchmark (PluriHopWIND) built from wind-industry technical reports, and proposes a retrieval architecture (PluriHopRAG) for exhaustive, recall-sensitive QA over large, repetitive report corpora.

What This Paper Is About

Most RAG systems are built for single-hop or multi-hop questions, where retrieval can stop once a small set of relevant passages is found, or for summarization-style questions that have no single exact answer. The authors identify a third setting — questions that require checking every document in a corpus, where omitting even one relevant passage changes the answer, and where there is exactly one best answer. They call these pluri-hop questions, formalize them, and show that existing RAG approaches fail on them. They then build a system that learns how information is distributed across a specific corpus and filters documents cheaply so that full-corpus checking stays affordable.

Key Contributions

1. A formalization of pluri-hop questions defined by three conditions: recall sensitivity (omitting one relevant passage produces a wrong answer), exhaustiveness (it cannot be inferred from retrieved context whether the evidence set is complete, so in principle all documents must be checked), and exactness (there is only one best answer; all others are incomplete or contain incorrect information).

2. PluriHopWIND, a multilingual diagnostic benchmark of 48 English questions constructed from 191 real wind-industry technical reports in German or English, covering oil laboratory analyses, turbine inspections, and service activities. Documents vary in length from 1 to 50 pages and are anonymized and pseudonymized.

3. PluriHopRAG, a retrieval architecture with two design principles: structure-aware query decomposition (an LLM fine-tuned on synthetic examples splits a query into document-level subquestions based on corpus-specific document structure) and cheap document-level filtering (a cross-encoder discards irrelevant documents after chunk retrieval but before costly LLM reasoning).

4. An empirical comparison against naive RAG, graph-based RAG (GraphRAG), and multimodal RAG (VisdomRAG) on PluriHopWIND, plus generalization results on the Loong benchmark of financial, legal, and scientific reports.

Main Findings

• Existing methods plateau below 40% F1. Naive, graph-based, and multimodal RAG methods reach at most 40% statement-wise F1 on PluriHopWIND.

• PluriHopRAG improves F1 by 18–52% across base LLMs. The paper reports a 18% relative improvement (0.4 to 0.47) with Claude 4 Sonnet as the base model and a 52% relative improvement with GPT-4o (0.27 to 0.41). Note that the text's stated F1 values do not exactly match the F1 values listed in Table 3, where PluriHopRAG scores 0.44 (Claude 4 Sonnet) and 0.36 (GPT-4o).

• PluriHopRAG leads the PluriHopWIND comparison. On PluriHopWIND, PluriHopRAG with Claude 4 Sonnet reaches precision 0.47, recall 0.57, F1 0.44; with GPT-4o it reaches precision 0.48, recall 0.39, F1 0.36. The next-best method in both cases is naive RAG with Cohere Rerank 3.5 as reranker: F1 0.38 with Claude 4 Sonnet and F1 0.25 with GPT-4o. VisdomRAG scores F1 0.19 (Claude 4 Sonnet) and 0.21 (GPT-4o); GraphRAG scores F1 0.30 and 0.21 respectively.

• Reranking matters for naive RAG. In the ablation, per-page chunking with reranking performs best among naive RAG variants (F1 0.40 with Claude 4 Sonnet, 0.27 with GPT-4o), compared with per-page chunking alone (0.26 and 0.14), character-count chunking (0.17 and 0.12), and character-count chunking with reranking (0.31 and 0.21). The reranker selects k′=20 chunks out of k=80.

• Fine-tuning the query decomposer is important. A fine-tuned GPT-4o-based query decomposer reaches precision 0.48, recall 0.39, F1 0.36, versus a few-shot prompted version at precision 0.50, recall 0.31, F1 0.30 — a 20% relative increase in F1.

• PluriHopWIND is unusually repetitive (distractor-rich). Using the r@k measure, PluriHopWIND shows 8–20% higher repetitiveness at k=2 than other datasets, and a 13–41% relative gap at k=50. As k increases, PluriHopWIND's repetitiveness drops the least (13%), indicating that even large top-k retrieval returns highly similar distractors.

• The cross-encoder filter is effective. Compared to human labels of document relevance, the filter correctly removes almost 50% of documents manually assessed as irrelevant, while removing only 10% of relevant documents.

• PluriHopRAG generalizes to Loong. Overall score 68.53 with GPT-4o, versus 51.71 for long-context reasoning (a 33% relative increase), 45.08 for NaiveRAG (a 52% increase), and 59.93 for StructRAG with Qwen2-72B (a 14% increase). Per-set scores for PluriHopRAG are 81.75 (10K–50K), 75.20 (50K–100K), 68.23 (100K–200K), and 56.48 (200K–250K).

Methodology in Plain English

The authors first generate benchmark questions in two steps: they manually write 2–7 single-hop question-answer pairs per document, targeting information from visually rich elements such as tables and diagrams, and then have an LLM aggregate these into pluri-hop questions that require aggregating many single-hop answers, are useful to a wind energy technician, require exhaustive document search, and create high distractor presence (for example, using documents from 2020–2022 for questions and 2018–2019 as distractors). Resulting question-answer pairs and document citations are manually corrected.

To quantify distractor density, they compute repetitiveness r@k: the average cosine similarity between each text chunk's embedding and its k nearest neighbors. They sample N=100 documents from PluriHopWIND and four other multi-hop datasets, chunk each into 500-character segments with 100-character overlap, embed them with OpenAI's text-embedding-3-large, and scan k ∈ {1, 2, 5, 10, 20, 50}.

The PluriHopRAG pipeline works as follows. A query decomposer — an LLM fine-tuned on 100 synthetic examples — rewrites the user question into document-scope intermediate questions and also produces a hypothetical summary of a document that would be relevant to answer it. Candidate documents are retrieved by comparing document summaries against that hypothetical summary (K > 198). Within each candidate document, chunks are retrieved per intermediate question. The concatenation of those chunks is then compared against the hypothetical summary using a cross-encoder (Cohere Rerank 3.5); if the score falls below a threshold of τ = 0.1, the document is discarded before any expensive LLM reasoning. Documents that pass are answered question by question, and the intermediate answers are aggregated into a final answer with document citations. Fine-tuning used OpenAI's supervised fine-tuning service on GPT-4o with 3 epochs, a learning rate multiplier of 2, and batch size 1.

Evaluation on PluriHopWIND uses statement-wise precision, recall, and F1, where reference and predicted answers are split into atomic statements and a GPT-5 few-shot prompt judges whether each predicted statement can be inferred from the reference. On Loong, per domain, 100 questions are reserved for fine-tuning the query decomposer and the remaining 1300 questions are evaluated, using GPT-4o as the base model and the original Loong scheme of an LLM judge (GPT-4) assigning scores from 1 to 100.

Why This Matters

The work argues that pluri-hop questions are widespread in real life — anywhere recurring report data accumulates — yet are poorly represented in existing benchmarks, which either have clear stopping conditions (multi-hop benchmarks such as HotpotQA, 2WikiMultiHopQA, MultiHopRAG) or violate exactness and recall sensitivity (summarization benchmarks such as NarrativeQA). It also shows that top-k retrieval is structurally mismatched to this setting, and that learning a corpus's document structure is a viable alternative.

Real-world applications named in the paper include:

• Healthcare: lab results, e.g. finding a patient's lowest and highest eGFR values with dates across 2022–2024.
• Education: student progress reports, e.g. identifying students who failed two or more terms between Fall 2022 and Spring 2025.
• Energy and utilities: turbine inspection reports, e.g. finding which turbine in a windpark had the most gearbox-wear reports of moderate severity or above.
• Retail and supply chain: supplier compliance reports, e.g. identifying suppliers with quality-check failures in three or more separate audits, as well as legal and contracts compliance audits listing clauses ever marked non-compliant.

For industry, the relevance is scalability: exhaustive checking of every document would be prohibitively expensive with LLM calls alone, so the cross-encoder filter is presented as a way to cut cost while retaining coverage. The research impact is the formalization of a question category distinct from multi-hop and summarization QA, along with a benchmark that exposes failure modes of current systems.

Future Directions

• Broader validation. PluriHopWIND contains 191 documents and 48 questions, which the authors describe as modest compared to other QA benchmarks; they call for validation across larger and more diverse corpora.

• Complete the StructRAG comparison. StructRAG has only been evaluated on Loong with Qwen 2-72b as the base LLM. Because of its Direct Preference Optimization procedure, the authors deemed training it on GPT-4o for Loong prohibitively expensive, leaving the comparison incomplete.

• Extend the benchmark to more domains. The results suggest pluri-hop questions are common in manufacturing, medicine, finance, and other fields beyond the wind-industry reports used here.

• Explore how much document-structure knowledge transfer requires. The paper claims that the basic logic of how information is laid out in a corpus can be imbued with very modest training set sizes, which raises the question of how this scales to less homogeneous or more rapidly changing corpora.

Target Audience

RAG and retrieval researchers evaluating exhaustive, recall-sensitive question answering; NLP practitioners building QA systems over large collections of recurring reports (medical records, financial filings, compliance audits, inspection reports); and industry engineers in manufacturing, energy, healthcare, and finance who need answers aggregated across an entire document corpus rather than a handful of retrieved passages. Readers seeking a rigorous formulation of question types and an evaluation methodology (statement-wise F1, LLM-based judging) will also find the paper relevant, as will those interested in small-scale supervised fine-tuning for domain-specific query decomposition.

Authors’ abstract

Retrieval-Augmented Generation (RAG) has been used in question answering (QA) systems to improve performance when relevant information is in one (single-hop) or multiple (multi-hop) passages. However, many real life scenarios (e.g. dealing with financial, legal, medical reports) require checking all documents for relevant information without a clear stopping condition. We term these pluri-hop questions, and formalize them by 3 conditions - recall sensitivity, exhaustiveness, and exactness. To study this setting, we introduce PluriHopWIND, a multilingual diagnostic benchmark of 48 pluri-hop questions over 191 real wind-industry reports, with high repetitiveness to reflect the challenge of distractors in real-world datasets. Naive, graph-based, and multimodal RAG methods only reach up to 40% statement-wise F1 on PluriHopWIND. Motivated by this, we propose PluriHopRAG, which learns from synthetic examples to decompose queries according to corpus-specific document structure, and employs a cross-encoder filter at the document level to minimize costly LLM reasoning. We test PluriHopRAG on PluriHopWIND and the Loong benchmark built on financial, legal and scientific reports. On PluriHopWIND, our method shows 18-52% F1 score improvement across base LLMs, while on Loong, we show 33% improvement over long-context reasoning and 52% improvement over naive RAG.