Zero-RAG: Towards Retrieval-Augmented Generation with Zero Redundant Knowledge

AI summary

Overview

Research area: Natural Language Processing — Retrieval-Augmented Generation (RAG), specifically corpus pruning driven by overlap between a language model's parametric knowledge and an external retrieval corpus.

Technical level: Advanced. The paper assumes familiarity with dense retrieval, supervised fine-tuning (LoRA), exact-match evaluation, and multi-hop QA benchmarks.

Scope: The paper introduces Zero-RAG, a pipeline that measures how well an LLM already "masters" each passage in a Wikipedia corpus, prunes redundant passages, and adds a query router plus noise-tolerant tuning so the model can rely on its internal knowledge after pruning.

What This Paper Is About

As LLMs grow stronger, much of the knowledge in a RAG system's external corpus is already stored in the model's parameters, creating redundancy that inflates indexing and retrieval cost and can even degrade accuracy. The authors quantify this redundancy, propose a metric (Mastery-Score) to detect it, and prune the corpus while trying to keep end-task performance intact. They report that Zero-RAG removes 30% of the Wikipedia corpus and speeds up retrieval by 22% without compromising RAG performance.

Key Contributions

1. First exploration of RAG-oriented corpus pruning. The authors state Zero-RAG is the first work to prune a RAG corpus by removing knowledge redundancy between the corpus and the LLM, rather than by re-ranking or compressing retrieved context.
2. The Mastery-Score metric and a learned corpus pruner. A sentence-level score is computed by generating QA pairs from a sentence with an LLM and measuring the target model's Exact Match accuracy on them; a smaller regression model (a 7B model trained to predict the Mastery-Score of a 70B model) then predicts this score for the whole corpus.
3. Query Router and Noise-Tolerant Tuning. Two modules that let the model exploit internal knowledge under a pruned corpus: a binary classifier that decides whether retrieval is needed at all, and a fine-tuning scheme that trains the model to suppress irrelevant retrieved documents.
4. Reported release of artifacts and empirical results on four QA datasets. The authors state they will release the code, model, and pruned corpus, and report pruning, latency, and accuracy results on PopQA, HotpotQA, TriviaQA, and EntityQuestions.

Main Findings

• Large overlap between LLM and Wikipedia knowledge. Figure 1(a) reports that Llama3.3-70B achieves at least 40% accuracy on Wikipedia-related questions, which the authors interpret as significant redundancy between the external corpus and the model.
• Redundant retrieval hurts mastered questions. When questions the LLM can already answer correctly are given the corresponding passage in context, accuracy drops by about 20 points (Figure 1(b)). The authors argue this shows redundant context can distract the model.
• Pruning costs little on benchmarks. With Llama3.3-70B, Zero-RAG scores 35.78 (PopQA), 51.28 (HotpotQA), 82.69 (TriviaQA), and 63.70 (EntityQuestions) with no pruning, versus 34.80, 48.52, 82.42, and 60.43 with a 30% pruned corpus. At 70% pruning the scores are 32.91, 46.20, 82.07, and 56.29. The authors describe the average degradation as less than two points at moderate pruning and around three points at a 70% pruning ratio.
• TriviaQA is the most redundant. Removing 70% of the corpus drops TriviaQA from 82.69 to 82.07, a 0.62-point change, which the authors say indicates most TriviaQA-related knowledge in the corpus is redundant.
• Retrieval alone can lower scores; noise-tolerant tuning recovers them. For Llama3.3-70B, adding retrieval to the instruct model changes PopQA from 16.25 to 16.35, HotpotQA from 46.20 to 43.35, TriviaQA from 81.43 to 79.32, and EntityQuestions from 52.01 to 50.71. After Noise-Tolerant Tuning, the same model with retrieval reaches 38.94, 49.12, 81.50, and 65.16.
• Llama3.3-70B outperforms Llama3-70B at the same pruning ratios. The authors attribute this to Llama3.3-70B holding a larger reservoir of knowledge overlapping with Wikipedia. For Llama3-70B at 30% pruning, scores are 30.67 (PopQA), 43.30 (HotpotQA), 81.00 (TriviaQA), and 57.82 (EntityQuestions).
• Ablations isolate each module (TriviaQA / HotpotQA, Llama3.3-70B-Instruct). Zero-RAG at 30% pruning scores 82.42 / 48.52. With the Corpus Pruner removed (0% pruning) it scores 82.69 / 51.28; with the Query Router removed, 81.50 / 43.35; with Noise-Tolerant Tuning removed, 80.55 / 42.82.
• Retrieval latency falls with pruning. Table 4 reports latencies of 14.65 s (HotpotQA), 10.96 s (EntityQuestions), and 11.24 s (TriviaQA) with no pruning, averaging 12.28 s. At 30% pruning these become 10.89 s, 9.09 s, and 9.00 s, averaging 9.66 s — described as over 20% improvement on average.
• Robustness to the top-k setting. Zero-RAG (no pruning, Llama3.3-70B) scores 35.17/35.73/35.78 on PopQA, 50.98/51.16/51.28 on HotpotQA, 82.46/82.58/82.69 on TriviaQA, and 62.76/63.28/63.70 on EntityQuestions at Top5/Top10/Top20. The authors state performance is not overly sensitive to the value of k.
• Smaller models also carry redundancy. With Llama3-8B, pruning 30% of the corpus lowers EM recall by about four percentage points; Zero-RAG (no pruning) scores 21.29 (PopQA), 35.21 (HotpotQA), 72.08 (TriviaQA), 48.71 (EntityQuestions), versus 18.91, 31.89, 67.93, and 43.19 at 30% pruning.
• Total time consumption falls with pruning. Table 7 reports HotpotQA total time falling from 717.13 s to 531.94 s, EntityQuestion from 1924.47 s to 1476.23 s, and TriviaQA from 1655.54 s to 441.80 s when moving from 0% to 30% pruning.
• A separate table reports Qwen-2-72B results. Table 8 lists scores of 11.06/46.32/78.80 for the base row, 32.76/49.18/81.76 for Zero-RAG, and 29.13/45.17/80.90 for Zero-RAG with a 30% pruned corpus on PopQA/HotpotQA/TriviaQA. The authors describe degradation as averaging less than three points at moderate pruning, though the table's row label ("Llama3-70B-Instruct") differs from the stated Qwen-2-72B column header.
• Case study. The sentence "Queen Victoria became Empress of India in 1876" was pruned with a Mastery-Score of 1.0; Llama3-70B-Instruct answered all four generated questions correctly (100%).
• Reporting note. The direction of the pruning threshold is stated inconsistently: Section 3.1 says a sentence is pruned when the predicted score is below the threshold and describes 0 as "complete redundancy" and 1 as "critical information," while Appendix A.1 removes sentences whose scores exceed the threshold, and the case study removes a sentence with a high Mastery-Score.

Methodology in Plain English

The approach has four stages.

1. Measure mastery. For a given sentence, an LLM generates n question-answer pairs whose answers are contained in the sentence. The target model answers those questions, and the Mastery-Score is the average Exact Match between the generated answers and the model's outputs. A high score means the model already handles that knowledge. The paper's motivating argument is that memorizing knowledge does not guarantee the ability to use it after supervised fine-tuning, so a behavioral QA-based measure is used instead of a loss-based one.
2. Learn a pruner. Because scoring every sentence with LLM inference is infeasible, a regression model is trained to predict the Mastery-Score for each sentence, minimizing mean squared error against ground-truth scores, with the output normalized to [0, 1]. A dynamic, percentile-based threshold sets how much of the corpus to remove. A 7B model is trained to predict the scores of a 70B model, and the Wikipedia corpus is segmented into 138,390,600 sentences using NLTK's sentence tokenizer, with QA pairs generated by GPT-4o-mini.
3. Route queries. For a downstream dataset, the noise-tolerant model's performance is used to label each query as "mastered" or "unmastered." A binary classifier trained with cross-entropy on these labels decides at inference time whether to retrieve or to let the model answer directly from internal knowledge.
4. Tune for noise. Fine-tuning uses three sample formats — query with relevant documents, query with distracting documents, and query without retrieval — with a loss that combines a retrieval-free term, a retrieval-augmented term, and a noise-suppression term. Fine-tuning uses LoRA with a learning rate of 3×10⁻⁴. Retrieval uses stella_en_400M_v5 with 20 document candidates per query.

At inference, the corpus is pruned, the query router decides whether retrieval is needed, and mastered queries are answered directly while unmastered queries retrieve from the non-redundant subset.

Why This Matters

Impact on research. The paper reframes RAG efficiency as a corpus-side problem rather than a retrieval-side or prompt-compression problem. Most prior efficiency work cited in the paper — AutoCompressors, ICAE, RECOMP, prompt compression, caching of retrieved documents, and necessity-of-retrieval classifiers such as those of Jiang et al. and Liu et al. — operates on the query or the retrieved context. Zero-RAG instead asks whether passages should exist in the index at all. The authors also position the work as complementary to existing RAG methods, since pruning can be applied underneath other pipelines.

Real-world applications:

• Enterprise search and internal knowledge bases, where indexing cost scales with corpus size and much of the content is already known to the deployed model.
• Customer-support assistants built on product documentation, where latency per query matters and redundant documentation inflates the index.
• Open-domain QA services on encyclopedic sources such as Wikipedia, the setting actually evaluated here.
• Multi-hop reasoning assistants, since HotpotQA is one of the four evaluated benchmarks.

Industry relevance. Indexing and retrieval costs scale with corpus size, and the paper reports measurable reductions: a 30% smaller corpus, retrieval latency dropping from an average of 12.28 s to 9.66 s across three datasets, and TriviaQA total processing time falling from 1655.54 s to 441.80 s. For teams paying for vector index storage and per-query retrieval compute, corpus pruning is a direct cost lever. The finding that adding retrieval to an instruct model can lower scores on some datasets (HotpotQA 46.20 to 43.35 for Llama3.3-70B) is also a caution for teams that assume retrieval always helps.

Future Directions

• Extending beyond Wikipedia. The authors list generalizability across domains as a limitation: Zero-RAG has only been studied on Wiki-based databases, and domain-specific or multimodal corpora remain untested.
• Robustness to noisy source data. The authors note that performance depends on initial data quality, and that a noisy or inconsistent source corpus could cause the pruning mechanism to remove valuable information.
• Scaling the redundancy effect with stronger models. The authors state that as models and knowledge bases expand, the amount of prunable redundant information is expected to increase, implying larger time savings in future settings.
• Resolving the threshold definition and evaluating combined pipelines. The inconsistent pruning direction between Section 3.1 and Appendix A.1 needs clarification, and the authors' claim that pruning can be combined with other RAG methods is asserted rather than empirically tested with a specific competing method.

Target Audience

Researchers and engineers working on RAG systems, retrieval infrastructure, or knowledge-base management who need to reduce indexing and retrieval cost without sacrificing QA accuracy. It is most useful to readers already comfortable with dense retrieval, LLM fine-tuning, and exact-match evaluation, and to practitioners deciding whether their retrieval corpus contains knowledge the deployed model already holds. Readers looking for a beginner-level introduction to RAG will find the paper assumes substantial background.

Authors’ abstract

Retrieval-Augmented Generation has shown remarkable results to address Large Language Models' hallucinations, which usually uses a large external corpus to supplement knowledge to LLMs. However, with the development of LLMs, the internal knowledge of LLMs has expanded significantly, thus causing significant knowledge redundancy between the external corpus and LLMs. On the one hand, the indexing cost of dense retrieval is highly related to the corpus size and thus significant redundant knowledge intensifies the dense retrieval's workload. On the other hand, the redundant knowledge in the external corpus is not helpful to LLMs and our exploratory analysis shows that it instead hurts the RAG performance on those questions which the LLM can answer by itself. To address these issues, we propose Zero-RAG to tackle these challenges. Specifically, we first propose the Mastery-Score metric to identify redundant knowledge in the RAG corpus to prune it. After pruning, answers to "mastered" questions rely primarily on internal knowledge of the LLM. To better harness the internal capacity, we propose Query Router and Noise-Tolerant Tuning to avoid the irrelevant documents' distraction and thus further improve the LLM's utilization of internal knowledge with pruned corpus. Experimental results show that Zero-RAG prunes the Wikipedia corpus by 30\% and accelerates the retrieval stage by 22\%, without compromising RAG's performance.