DeFT-RAG: Incrementally Decomposing Multi-Hop Questions into Self-Contained Sub-Questions with Reusable Answers

AI summary

Overview

Research area: Natural language processing, specifically retrieval-augmented generation (RAG) for multi-hop question answering, with a secondary focus on lightweight fine-tuned decision models.

Technical level: Advanced. The paper assumes familiarity with dense retrieval, prompt-based decomposition, graph-based RAG and LoRA fine-tuning, though its core idea can be grasped without that background.

Scope: The paper proposes DeFT-RAG, a RAG framework that answers multi-hop questions by generating one self-contained sub-question at a time, offloading tracking and yes/no judgments to rules and a fine-tuned 4B decision model, and caching sub-question answers so later questions can reuse them.

What This Paper Is About

Multi-hop questions require chaining facts that are spread across several documents and that pass through intermediate "bridge entities" the question never names, so a single retrieval pass usually misses part of the evidence chain. Graph-based RAG retrieves only once from the entities the question names, and iterative RAG recovers the chain step by step but overloads the LLM with planning, dependency tracking, rewriting and bookkeeping, each of which produces its own kind of error. DeFT-RAG's goal is to remove most of that burden from the LLM, recover the full evidence chain more often, and make the intermediate answers reusable across questions instead of discarding them.

Key Contributions

1. Incremental question decomposition. Instead of planning sub-questions up front, DeFT-RAG generates exactly one self-contained sub-question at a time from the original question plus the QA pairs found so far. Because each sub-question names every entity it asks about and depends on no other sub-question, there is nothing to plan, no dependency to get wrong, and no sub-question to rewrite.

2. Moving the remaining LLM work outside the LLM. A QA table is updated by rules with no model call, and a fine-tuned lightweight decision model (DM) makes the yes/no decisions: whether a retrieved passage is relevant, whether the original question can be answered directly, and whether the accumulated QA pairs are enough to stop.

3. A QA cache for answer reuse. Because each sub-question is self-contained, it has the same meaning and answer regardless of which original question produced it. DeFT-RAG checks a cache of earlier QA pairs before retrieving, and reuses an answer when a stored sub-question means the same thing.

4. An empirical evaluation on three benchmarks with two backbones. DeFT-RAG is compared against thirteen baselines on MuSiQue, 2WikiMultiHopQA and HotpotQA (1,000 questions each) with gpt-4o-mini and Qwen3.6-35B-A3B, and code and prompts are released.

Main Findings

• Best accuracy across all six settings. With gpt-4o-mini, DeFT-RAG scores 0.629 (MuSiQue), 0.852 (2Wiki), 0.874 (HotpotQA) and 0.785 average accuracy; with Qwen3.6-35B-A3B, 0.636, 0.867, 0.871 and 0.791. It is the only system that is first or on par with the first on every benchmark.

• Largest gains where chains are longest. On MuSiQue it beats the second-best system, Hi-Q, by +5.2 points with gpt-4o-mini (0.629 vs. 0.577) and +5.3 with Qwen3.6 (0.636 vs. 0.583), both significant. On 2Wiki it leads LogicRAG by 2.9 points with Qwen3.6 (significant) and ties Self-Ask with gpt-4o-mini (0.852 vs. 0.849). On HotpotQA it leads Hi-Q by 3.4 points with gpt-4o-mini and ties Hi-Q and LinearRAG with Qwen3.6 (0.871 vs. 0.864 and 0.857).

• More of the evidence chain, in far fewer words. On MuSiQue with gpt-4o-mini, the LLM receives the full evidence chain for 46% of 3-hop and 18% of 4-hop questions using 390 words per question, the fewest of any system and less than half of vanilla RAG's 870 words. Systems that cover more chains on some hop counts pay far more: Self-Ask covers 83% of 2-hop chains with 2,590 words, and LinearRAG covers 19% of 4-hop chains with 4,110 words.

• Fixed top-k retrieval is a bottleneck in its own right. On single-step MuSiQue questions (n=92), widening from the top 5 to the top 10 passages raises how often all needed passages are present from 62% to 88%, but accuracy barely moves (0.620 to 0.630); keeping only the gold passages from the top 10 raises it to 0.793. 2Wiki shows the same pattern.

• Passage selection helps. Removing the DM's relevance filter and giving the LLM the top-5 passages drops accuracy by 2.3, 3.7 and 5.3 points on MuSiQue, 2Wiki and HotpotQA over 300 test questions per benchmark (0.791 vs. 0.753 average); the HotpotQA drop is significant (p = 0.002).

• Fine-tuning the decision model improves its judgments. AUC rises from 0.850 to 0.933 on the MuSiQue direct-answer decision, from 0.931 to 0.979 on MuSiQue relevance, and from 0.996 to 1.000 on 2Wiki relevance, while the enough-facts decision changes little (for example 0.903 to 0.904 on MuSiQue). End to end on the first 300 MuSiQue questions with gpt-4o-mini, accuracy rises from 0.587 to 0.623.

• Answer reuse is close to free. Over all 1,000 MuSiQue questions with gpt-4o-mini and one shared cache, 23.1% of generated sub-questions are answered from the cache, cutting LLM calls by 12.9%, with no accuracy loss (0.642 with reuse vs. 0.629 without, p = 0.18).

Methodology in Plain English

The design rests on one observation: iterative RAG systems fail largely because they ask one LLM call to do too many things at once. The authors enumerate four error types that follow from this — planning errors (guessing bridge entities before they are found), dependency errors (searching for a sub-question before the fact it depends on is known), rewriting errors (failing to substitute a found entity into a pending sub-question), and tracking errors (losing a fact already present in a long, growing prompt). Their example failures are drawn from a 3-hop MuSiQue question, where HippoRAG 2 stops after the first bridge entity, LogicRAG searches for a city before identifying the performer, and IRCoT has all three gold passages in its prompt yet answers with the wrong terminal.

DeFT-RAG reduces the LLM to three text-generation tasks. One writes the next self-contained sub-question given the original question and the QA table. One answers a sub-question from the passages selected for it. One writes the final answer from the QA table. Everything else is handled outside the LLM. A QA table, updated purely by rules, records each answered pair, remembers sub-questions that returned nothing so they are not asked again, and detects repeats by normalized wording. A fine-tuned decision model — built on StartLux-Decision-4B (Qwen3.5 architecture, 4B parameters) and trained with LoRA (rank 16, alpha 32, dropout 0.05, all linear layers; 32.5M of 4.24B parameters trained, learning rate 10⁻⁴, one epoch, 2,262 steps, about 6.4 hours on two NVIDIA RTX A5000 GPUs) on 48,036 training and 7,806 development examples derived only from the benchmarks' training splits — returns yes-probabilities for three decisions. Thresholds are 0.5 for relevance and enough facts, 0.7 for direct answer (a wrong direct answer is hard to recover from, whereas a needlessly looped question can still succeed), and 0.8 for cache matching.

The loop works like this: retrieve w = 2k = 10 passages for the original question; let the DM decide whether the question can be answered directly, and if so answer it. Otherwise, iterate up to T = 6 times: write a sub-question, let the DM keep only relevant passages (at most five, 2.1 to 2.4 on average, matching the baselines' top-5), answer it, add the pair to the table, and check whether enough is known. If the step limit is reached, fall back to answering from the relevant passages among the top 2w retrieved for the original question. All systems share the same backbones, the same embedding model (Qwen3-Embedding-4B), temperature 0 and output limit, with baselines keeping their own published settings; accuracy is graded by a different model than the one being tested, using the SimpleQA prompt, and differences are tested with the exact McNemar test.

Why This Matters

Impact on research. The paper reframes multi-hop RAG as a division-of-labor problem rather than a prompting problem: it shows that a fine-tuned small model can absorb the classification decisions that iterative pipelines usually leave to a large LLM, and that self-contained sub-questions make intermediate results portable across questions. The evidence-coverage analysis (46% of 3-hop chains at 390 words) gives the field a second axis — context efficiency — alongside accuracy, and the top-5 versus top-10 experiment is a clean demonstration that retrieval breadth alone does not translate into answer quality.

Real-world applications:

• Enterprise or legal question answering over document collections where answers require joining facts from several contracts, filings or memos.
• Biomedical and scientific literature search, where a question about a drug's effect may require chaining a compound to a gene, a gene to a pathway and a pathway to an outcome.
• Customer-support and product knowledge bases, where the same intermediate facts (a company's headquarters, a part's compatibility) recur across many user questions and would benefit directly from the QA cache.
• Agentic or tool-using assistants that must plan a sequence of lookups under latency and cost constraints, where fewer tokens per step and fewer LLM calls per query matter as much as accuracy.

Industry relevance. The two headline efficiency numbers — fewer input tokens than any other iterative method and 12.9% fewer LLM calls from caching — speak to inference cost and latency, which are usually the binding constraints in deployment. Answer reuse across queries is particularly relevant to production traffic, where recurring questions are the norm rather than the exception.

Future Directions

• Cache management at scale. The cache is evaluated on a single ordered pass over 1,000 MuSiQue questions. How it behaves under much larger caches, with staleness, conflicting answers, or eviction policies, is not reported and is a natural next question.

• Semantic matching of sub-questions. The cache relies on normalized-surface matching plus a DM judgment at a strict threshold of 0.8, and the paper notes that a wrong reuse propagates an error into another question. Better calibrated or verifier-backed matching is an open direction.

• Beyond a fixed step limit. DeFT-RAG caps the loop at T = 6 steps and then falls back to direct answering; whether adaptive stopping or a learned step budget improves the long-chain cases (18% coverage at 4 hops, the weakest of the reported hop counts) is unexplored in the presented content.

• Portability of the decision model. The DM is fine-tuned on labels derived from gold annotations of three benchmarks' training splits. Whether equivalent decisions can be obtained without gold supervision, or transferred to new domains and corpora, is not addressed. The provided content also does not report a dedicated limitations section, and cost comparisons beyond token and call counts are deferred to an appendix not included in the excerpt.

Target Audience

Researchers and practitioners working on retrieval-augmented generation, multi-hop question answering and LLM pipeline design will get the most from this paper, particularly those building production RAG systems who care about token cost, latency and per-step reliability alongside accuracy. It is also useful for readers interested in how small fine-tuned models can replace LLM self-classification inside multi-step pipelines. Readers new to RAG will find the error taxonomy in the introduction and the failure figure accessible, but should expect the evaluation sections to assume familiarity with benchmark conventions and retrieval metrics.

Authors’ abstract

Retrieval-augmented generation (RAG) lets a large language model (LLM) answer questions by retrieving relevant passages from a large corpus. However, RAG struggles with multi-hop questions, which require chaining facts spread across several documents through intermediate entities that the question never mentions. Graph-based RAG methods retrieve once from the entities a question names and so rarely recover the full evidence chain. Iterative RAG methods recover it step by step, but their accuracy is limited by the many tasks they ask the LLM to perform, such as planning the sub-questions, deciding their dependencies, rewriting them as answers arrive, and keeping track of what is known. We propose DeFT-RAG, a RAG framework built on three components. First, DeFT-RAG generates one self-contained sub-question at a time from the original question and the sub-questions answered so far with their answers (QA pairs). Each sub-question names every entity it asks about, which removes the need to plan, track dependencies or rewrite, and lets the LLM focus on a single fine-grained task at a time. Second, DeFT-RAG moves the remaining LLM work outside the LLM: rule-based updates to a QA table that keeps track of what is known, and yes/no decisions made by a fine-tuned lightweight decision model. Third, DeFT-RAG caches the QA pairs and reuses an answer when a later question asks a sub-question with the same meaning. On MuSiQue, 2WikiMultiHopQA and HotpotQA, DeFT-RAG achieves the best accuracy among thirteen baselines with both gpt-4o-mini and Qwen3.6 as the backbone, while giving the LLM fewer input tokens than any other iterative method. With a QA cache shared across 1,000 MuSiQue questions, 23.1% of sub-questions are answered from the cache, which reduces LLM calls by 12.9% without lowering accuracy.