Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers

AI summary

Overview

Research area: Natural Language Processing, specifically retrieval-augmented generation (RAG) for single-hop and multi-hop question answering.

Technical level: Advanced. The paper assumes familiarity with dense retrieval, contrastive training objectives (InfoNCE), late-interaction scoring (ColBERT-style MaxSim), transformer layer representations, and prompt-based LLM reasoning chains.

Scope: The paper proposes CoopRAG, a five-stage RAG pipeline in which a retriever and an LLM exchange information, and evaluates it on three multi-hop QA datasets, one single-hop QA dataset, and two factual QA benchmarks.

Publication details: arXiv:2512.10422v3 [cs.CL], 17 Dec 2025, by Youmin Ko, Sungjong Seo, and Hyunjoon Kim (Department of Artificial Intelligence / Department of Data Science, Hanyang University). Licensed CC BY 4.0. Code is listed at github.com/meaningful96/CoopRAG.

What This Paper Is About

Large language models often produce factually wrong answers, and RAG is the standard remedy, but existing RAG systems still retrieve the wrong documents and hallucinate, especially for simple and multi-hop QA. The authors argue this happens for three reasons: questions are too short to guide retrieval or reasoning; contrastive retrieval training pushes retrievers toward shallow lexical or semantic similarity ("mass-seeking behavior") instead of documents containing critical facts; and existing methods give the LLM no structured opportunity to patch knowledge it is unconfident about. CoopRAG addresses all three by having the LLM unroll a question into sub-questions and a masked reasoning chain, having a retriever answer that unrolled question, reranking results by contrasting the retriever's earlier and later layers, and having the LLM fill in the masked positions using the retrieved documents.

Key Contributions

1. Question unrolling with uncertainty masking. The LLM decomposes a question into a set of sub-questions and an evidence-triple reasoning chain in which low-confidence entities are replaced by an ⟨UNCERTAIN⟩ mask and the final answer slot is left as ⟨FILL⟩. This lets the LLM and retriever use only knowledge the LLM is certain about.
2. Unrolling-augmented retrieval (UAR). The original question is concatenated with the sub-questions and the masked reasoning chain to form an "unrolled question," which is embedded and searched (via Faiss) for the top-n documents.
3. Ranking by Contrasting Layers (RaLa). A reranking method that contrasts hidden states from lower ("premature") layers against the last layer of the retriever encoder. Layers are split into two to four buckets, one layer is randomly chosen per bucket, and documents are reranked by a MaxSim score (ColBERT-style) multiplied by a gap-aware weight ω derived from the [CLS]-token difference.
4. Difficulty-aware retriever fine-tuning and reasoning chain completion. The encoder is fine-tuned with a sample-wise reweighted InfoNCE loss where each question's weight is α = log(1 + |S|), proportional to its number of sub-questions; afterward, the LLM fills the ⟨UNCERTAIN⟩ and ⟨FILL⟩ slots from the top-k documents and then answers.

Main Findings

• Retrieval gains across the board: CoopRAG (GPT-4o-mini) outperformed all compared competitors on every retrieval benchmark. The paper reports up to 5.3% improvement on the multi-hop QA datasets and up to 35.2% improvement on the single-hop dataset over the prior state of the art (HippoRAG2). On NaturalQuestions specifically, it improves on HippoRAG2 by 35.2% in Recall@2 and 14.1% in Recall@5.
• Small LLM beats a much larger one: CoopRAG with Gemma2-9B surpassed HippoRAG2 using Llama3.3-70B. The introduction also states that CoopRAG using Gemma2-9B outperforms a prior GPT-4o-mini-based method, and that it achieves at least 15.2% higher EM on NaturalQuestions than baselines even with Gemma2-9B.
• QA performance: CoopRAG (GPT-4o-mini) reached HotpotQA EM 65.6 / F1 78.9, MuSiQue EM 52.3 / F1 67.1, 2WikiMultihopQA EM 71.7 / F1 79.2, and NaturalQuestions EM 72.0 / F1 82.3. For comparison, HippoRAG2 (Llama3.3-70B) scored HotpotQA 62.7 / 75.5, MuSiQue 37.2 / 48.6, 2Wiki 65.0 / 71.0, and NQ 48.6 / 63.3.
• Factual QA benchmarks: On SimpleQA and FreshQA, CoopRAG achieved 16.1% and 26.1% higher EM than HopRAG, respectively. Using the ChatGPT grader (Correct / Incorrect / Not Attempted), CoopRAG produced 283 correct answers versus 233 for HopRAG and 225 for HippoRAG2.
• Both weights matter, the gap-aware weight matters more: Removing both the gap-aware weight ω and difficulty weight α dropped 2WikiMultihopQA Recall@2 from 81.4% to 76.1%. On MuSiQue, removing ω reduced Recall@2 from 59.6% to 57.0%, while removing α reduced it only to 58.2%.
• RaLa improves score separation: Applying ω increased all similarity-score differences on all datasets. The relative increase in (positive) − (random negative) was 32.16% on HotpotQA, 82.41% on MuSiQue, and 38.93% on 2WikiMultihopQA. The harder (distractor) − (random negative) gap rose by 91.39% on HotpotQA, 987.9% on MuSiQue, and 94.28% on 2WikiMultihopQA.
• Uncertainty masks help retrieval and reasoning: Using ⟨UNCERTAIN⟩ improved Recall@2 by 1.4% on HotpotQA, 8.0% on MuSiQue, and 8.9% on 2WikiMultihopQA, with EM improvements on all datasets (e.g., 2Wiki EM 65.6 to 71.7). The paper also reports entropy decreases: generating ⟨FILL⟩ with uncertainty masks produced a 5.13, 6.32, and 5.62-fold entropy decrease on HotpotQA, MuSiQue, and 2Wiki respectively, and the entropy dropped by factors of 8.59, 8.88, and 9.92 right after ⟨UNCERTAIN⟩ generation.
• RaLa beats other retrievers with and without fine-tuning: Without fine-tuning, RaLa reached HotpotQA R@2 73.9 versus 68.2 for ReSCORE, 64.7 for ColBERTv2, and 57.3 for Contriever. After fine-tuning, RaLa reached 88.8% R@2 and 96.8% R@5 on HotpotQA, 7.9% and 4.5% improvements over ReSCORE, and also outperformed ReSCORE on MuSiQue and 2WikiMultihopQA.

Methodology in Plain English

The pipeline has five inference stages. First, an LLM reads the question and writes out several simpler sub-questions plus a chain of evidence triples (head entity, relation, tail entity). Wherever the LLM is unsure of an entity, it writes a mask instead of guessing; the last tail slot is a fill-in-the-blank placeholder for the final answer. Second, the question is concatenated with these sub-questions and the masked chain, and a fine-tuned MPNet encoder retrieves the top-n documents for that combined text. Document embeddings are precomputed as [CLS] vectors and stored in a vector database, with similarity search handled by Faiss.

Third, those n documents are reranked down to the top-k (k < n). This reranking, RaLa, uses the idea that early transformer layers capture surface or syntactic information while later layers capture abstract meaning. The encoder's layers are split into two to four buckets, one layer is picked at random from each bucket, and a document's score combines a ColBERT-style token-level MaxSim score at the final layer with a "gap-aware" weight ω. That weight is the difference in [CLS]-token similarity between the last layer and the chosen premature layer, so documents whose representation changes most between early and late layers get boosted.

Fourth, the LLM reads the top-k documents and fills in the masked entities and the final ⟨FILL⟩ slot, effectively repairing its own reasoning chain with retrieved evidence. Fifth, the LLM answers the question given the question, the sub-questions, the completed reasoning chain, and the top-k documents.

Training mirrors ColBERT: each training question is unrolled, paired with one positive and one hard negative document, and optimized with an InfoNCE loss. Each sample's loss is weighted by α = log(1 + |S|), where |S| is the number of sub-questions, so harder questions (which tend to decompose into more sub-questions) exert more influence on the encoder.

Why This Matters

Impact on research. The paper offers a concrete alternative to the retrieve-then-read pattern: instead of treating retrieval and generation as separate stages, it makes the LLM's expressed uncertainty the interface between them. The RaLa reranking idea — using the gap between a model's own layers as a relevance signal — is a lightweight, general trick that could be applied to other encoders and tasks, and the reported entanglement of a 9B model beating pipelines built on a 70B model suggests that pipeline design, not raw scale, is a large remaining lever.

Real-world applications.

• Enterprise and customer-support question answering, where multi-hop queries must be resolved against internal document stores and wrong answers carry a real cost.
• Scientific and legal literature search, where a question typically requires chaining several sources together and where surfaced uncertainty (the ⟨UNCERTAIN⟩ masks) is arguably more useful than a confident wrong fact.
• Search and assistant products that already use retrieve-rerank-generate pipelines and could swap in a layer-contrasting reranker.
• Resource-constrained deployments, since the reported result with Gemma2-9B indicates competitive quality without a frontier-scale model.

Industry relevance. The method is modular: the reranking stage plugs into existing ColBERT-style retrievers, and IRCoT-style multi-step retrieval is stated to be easily integrable. The authors note they use MPNet as the encoder and Gemma2 (9B, 27B), Llama3.3-70B, and GPT-4o-mini as LLMs, so the approach is not tied to a single vendor. The stated cost concerns — sequence-length limits on embedding long documents and rising MaxSim cost as token counts grow — are the main practical cautions, and the paper does not report wall-clock latency or dollar-cost figures.

Future Directions

1. Extend to knowledge base question answering and knowledge graph QA. The conclusion explicitly names validating on KBQA and domain-specific datasets, and moving beyond passage-based retrieval to knowledge graph QA, as promising directions.
2. Handle long documents. The authors state as a limitation that pretrained LMs cannot embed long documents because of sequence-length constraints.
3. Control compute cost. MaxSim operations over ColBERT-style token embeddings are acknowledged to grow with the number of tokens; how to keep the layer-contrasting scoring cheap at scale is left open.
4. Understand and reduce incorrect retrievals more broadly. The paper attributes part of the problem to the contrastive learning objective and its "mass-seeking" behavior, but states that the exact causes of incorrect retrievals in RAG are not fully understood — an open question the proposed reranking only partially addresses.

Target Audience

This paper is most useful to NLP researchers and graduate students working on retrieval-augmented generation, dense retrieval, and multi-hop question answering, and to applied ML engineers who build retrieve-rerank-generate pipelines and want a concrete reranking technique that uses a retriever's internal layers. Readers need comfort with transformer internals, late-interaction retrieval scoring, and contrastive training objectives to follow the method sections, though the high-level five-stage pipeline is understandable without that background.

Authors’ abstract

Since large language models (LLMs) have a tendency to generate factually inaccurate output, retrieval-augmented generation (RAG) has gained significant attention as a key means to mitigate this downside of harnessing only LLMs. However, existing RAG methods for simple and multi-hop question answering (QA) are still prone to incorrect retrievals and hallucinations. To address these limitations, we propose CoopRAG, a novel RAG framework for the question answering task in which a retriever and an LLM work cooperatively with each other by exchanging informative knowledge, and the earlier and later layers of the retriever model work cooperatively with each other to accurately rank the retrieved documents relevant to a given query. In this framework, we (i) unroll a question into sub-questions and a reasoning chain in which uncertain positions are masked, (ii) retrieve the documents relevant to the question augmented with the sub-questions and the reasoning chain, (iii) rerank the documents by contrasting layers of the retriever, and (iv) reconstruct the reasoning chain by filling the masked positions via the LLM. Our experiments demonstrate that CoopRAG consistently outperforms state-of-the-art QA methods on three multi-hop QA datasets as well as a simple QA dataset in terms of both the retrieval and QA performances. Our code is available.