HiFi-RAG: Hierarchical Content Filtering and Two-Pass Generation for Open-Domain RAG

AI summary

`) as markdown plain text. Reddit trees are reconstructed with top-k comments (k=5) and two layers of nested replies (m₁=3, m₂=2). Gemini 2.5 Flash then ranks the sections using only their titles and the first 200 characters of content, and drops the rest. 4. **Two-pass generation.** Gemini 2.5 Pro first drafts an answer in one paragraph from the filtered content, then in a second turn revises its answer to match the style and length of three hand-picked question-answer examples drawn from the validation set (distinct from the first 100 pairs used for evaluation). 5. **Citation verification.** A separate Gemini 2.5 Flash call reads the final answer and the sources and returns a JSON list of source indices that directly support the claims, prioritizing the most specific matching source when duplicates exist. Evaluation used ROUGE-L and DeBERTaScore with the `microsoft/deberta-xlarge-mnli` model. The validation set had 300 samples, but because of cost and time constraints during ablation, the first 100 were evaluated. Test2025 added 100 synthesized samples. For the final Test2025 run, the three style examples were replaced with three new pairs generated by Gemini 3.0 to better match the test distribution. ## Why This Matters - **For research:** The paper argues that a structured, deterministic multi-stage pipeline can outperform more autonomous "agentic" workflows for open-domain RAG — a notable counterpoint to the trend toward agent-based retrieval. It also offers a reusable ablation template and a future-events evaluation set for separating retrieval ability from parametric memory. - **Real-world applications:** - Search assistants that must answer questions about events, products, or schedules announced after a model's training cutoff. - Customer support and help-desk bots, where the paper's "how-to" examples (tying shoelaces, leash training a puppy, Photoshop edits) reflect typical user phrasing. - Enterprise or news Q&A systems where citations must be traceable to specific sources and duplicates should resolve to the most authoritative one. - Cost-sensitive deployments that need high retrieval precision without paying frontier-model prices on every filtering step. - **Industry relevance:** The design explicitly targets the cost/latency/accuracy tradeoff by splitting work between a cheap model (Gemini 2.5 Flash, stated as 4-6× cheaper than Pro) and an expensive one (Gemini 2.5 Pro). The reported failures of agentic tooling (10× cost, timeouts) and prompt optimizers (overfitting) are practical warnings for teams building production RAG. ## Future Directions 1. **Close the knowledge-cutoff gap systematically.** The paper shows the RAG-versus-no-search gap ballooning to 44.16% on Test2025; extending and hardening the Test2025-style evaluation would let researchers track how well pipelines compensate for stale parametric knowledge over time. 2. **Resolve the rephrasing trade-off.** Query rephrasing hurt scores unless URL filtering was also applied. Finding a formulation method that preserves verbosity cues without introducing ambiguity is an open problem. 3. **Replace or improve the discarded components.** Embedding-based filtering, agentic search, DSPy/GEPA prompt optimization, and LLM-as-a-Judge checking all underperformed here; each failure has a specific diagnosed cause that invites a better solution. 4. **Better style-alignment without hand-picked examples.** The final system depends on three hand-selected or generated question-answer examples for the refinement turn; automating example selection or generating style targets on the fly is a natural next step. 5. **Reconcile qualitative and automatic metrics.** The "Checker" module improved answers qualitatively while lowering ROUGE and DeBERTaScore, raising the question of whether these metrics fully capture answer quality for RAG. ## Target Audience This paper is most useful to applied NLP engineers and researchers building open-domain RAG systems, especially those working under latency and cost constraints, and to competition participants in benchmarks such as MMU-RAGent. It is also valuable to teams deciding between deterministic multi-stage pipelines and agentic architectures, and to anyone evaluating whether retrieval or parametric memory is carrying the load in a question-answering system. The five-stage structure and the included appendix prompts make it practical to adapt, while the negative results section saves readers from repeating several expensive dead ends.

Authors’ abstract

Retrieval-Augmented Generation (RAG) in open-domain settings faces significant challenges regarding irrelevant information in retrieved documents and the alignment of generated answers with user intent. We present HiFi-RAG (Hierarchical Filtering RAG), the winning closed-source system in the Text-to-Text static evaluation of the MMU-RAGent NeurIPS 2025 Competition. Our approach moves beyond standard embedding-based retrieval via a multi-stage pipeline. We leverage the speed and cost-efficiency of Gemini 2.5 Flash (4-6x cheaper than Pro) for query formulation, hierarchical content filtering, and citation attribution, while reserving the reasoning capabilities of Gemini 2.5 Pro for final answer generation. On the MMU-RAGent validation set, our system outperformed the baseline, improving ROUGE-L to 0.274 (+19.6%) and DeBERTaScore to 0.677 (+6.2%). On Test2025, our custom dataset evaluating questions that require post-cutoff knowledge (post January 2025), HiFi-RAG outperforms the parametric baseline by 57.4% in ROUGE-L and 14.9% in DeBERTaScore.