RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition

AI summary

Overview

Research area: Information retrieval and retrieval-augmented generation (RAG), specifically adaptive/agentic RAG architectures and their evaluation (cs.IR).

Technical level: Intermediate. The paper is a system description with an accompanying qualitative user study; it assumes familiarity with RAG pipelines, retrieval and reranking, and LLM serving, but the architecture is explained clearly enough for readers with baseline IR knowledge.

Scope: A 12-author team report describing R2RAG, the routing-based RAG system that won the Best Dynamic Evaluation award in the Open Source category of the Text-to-Text track of the NeurIPS 2025 MMU-RAG Competition.

What This Paper Is About

The authors built a RAG system for a competition in which systems must answer deep-research questions in real time under user-centric evaluation. The core problem is that simple factual questions and complex multi-faceted questions need different retrieval strategies, but running larger models is infeasible on the available hardware. Their goal was a lightweight, dynamically adaptive pipeline that runs on a single consumer-grade GPU while still handling complex research tasks.

Key Contributions

1. R2RAG (Routing-to-RAG): A RAG architecture that classifies each incoming query as simple or complex and routes it to one of two pipelines — a single-pass "Vanilla RAG" path or an iterative "Vanilla Agent" path.

2. A resource-constrained model configuration: All major components use Qwen3-4B for classification, query reformulation, document assessment and answer generation, with Qwen3-Reranker-0.6B for reranking, served through vLLM on one consumer-grade GPU.

3. A qualitative focus-group study (N=20) conducted with participants from RMIT University's Centre for Human-AI Information Environments (CHAI), spanning master's students to professors, used to identify failure modes and refine prompts and components.

4. An award-winning competition entry plus a released code repository (github.com/rmit-ir/NeurIPS-MMU-RAG) and a deployment design that exposes multiple RAG variants on a single GPU server via an OpenAI-compatible API.

Main Findings

• Award outcome: R2RAG won the Best Dynamic Evaluation award in the Open Source category. The paper does not report numeric leaderboard scores or the system's final ordinal rank.

• Small models were sufficient: Unquantized Qwen3-4B performed classification and generation tasks well, and in preliminary experiments 4-bit quantized larger models (e.g. Qwen3-8B) yielded weaker performance than the unquantized 4B variant, motivating the no-quantization choice.

• The two pipelines have complementary strengths: The Preference Ratio from the qualitative study suggested similar overall performance between the two variants, but qualitative analysis showed Vanilla Agent handled complex queries more effectively while being verbose for simple ones, whereas Vanilla RAG was concise for simple queries but weaker on multi-faceted tasks.

• Automatic metrics missed user-relevant differences: The authors report that aggregate metrics did not capture differences in reasoning depth, presentation, perceived system biases, response structure and level of detail, and tone, which influenced overall user judgments. This aligns with the competition organizers' stated insight that live (arena) evaluation reveals qualitative distinctions not captured by static metrics.

• Retrieval quality and controlled evidence accumulation were central to answer quality, while structured LLM-based decision components improved robustness, according to the authors' conclusions.

• LLM-based classification was preferred over logistic regression despite higher computational cost, because the 10-minute per-query limit made the extra overhead acceptable and the LLM classifier better handles out-of-distribution queries and allows criteria changes without retraining.

Methodology in Plain English

The system starts with a query classifier that labels each query simple or complex. The authors built and tested a logistic regression classifier on 175,850 queries drawn from TREC Deep Learning, Deep-Research Questions, TREC RAG 2025 and Natural Questions, using 147-dimensional feature vectors (128-dimensional semantic embeddings plus 19 linguistic features from part-of-speech and punctuation counts). They also built an LLM-based classifier with a structured prompt that asks whether the question could be answered easily with a single Google search. They deployed the LLM-based classifier.

Simple queries go down the Vanilla RAG path: generate three query variants, search the ClueWeb22-A index in parallel (up to ten documents per query, fewer than 30 after deduplication), rerank with a pointwise yes/no reranker, truncate the top documents to a 5k-word limit, and generate an answer with inline [ID] citations.

Complex queries go down the Vanilla Agent path, which repeats the variant–retrieval–reranking loop with more aggressive settings — up to five query variants, up to 25K tokens of documents — accumulating state about previously used queries and summaries of useful documents. A document-review step using Qwen3-4B estimates information coverage and generates new queries targeting identified gaps. The loop stops when any of three conditions holds: accumulated tokens exceed 20,000, coverage equals 1, or five iterations have been reached.

Resource tuning included reducing Qwen3-4B's context window to 25,000 tokens from the native 32,768, and running at temperature 0.6 and top_p 0.95 with the developer-recommended decoding parameters.

For evaluation, the team ran a two-hour qualitative session in which 20 participants freely explored the system, submitting an average of 17 queries each, giving binary feedback, open-ended comments and verbal reflections. Feedback was analyzed via a Preference Ratio and manual inspection of comments.

Why This Matters

The paper shows that competitive, dynamically adaptive RAG can be built from compact models on a single consumer-grade GPU, challenging the assumption that strong dynamic RAG requires large models or heavy quantization trade-offs.

Real-world applications:

• Deep-research assistants that must decide on their own when a question needs one search versus several rounds of investigation.
• Cost-constrained enterprise or academic search deployments where GPU memory, not model quality, is the binding constraint.
• Real-time, interactive search systems where responses are judged by users within a live arena rather than against a static test set.
• Evaluation methodology for RAG — the qualitative protocol described here is a reusable template for surfacing user-perceived issues like verbosity, bias and response structure.

Industry relevance: The deployment design exposes multiple RAG variants behind an OpenAI-compatible API on one GPU server, assigning each configuration a unique model identifier, which removes the need to run separate servers per variant. The competition's 10-minute per-query budget reflects realistic latency pressure for production assistants.

Future Directions

• Applying the qualitative evaluation protocol at larger scale and across more diverse user populations than the 20 participants from one institution reported here.
• Determining whether the query-complexity routing decision can be made more cheaply or more accurately, since the authors chose the costlier LLM classifier over the logistic regression alternative.
• Testing whether the conclusion that unquantized 4B models beat 4-bit quantized larger models holds for other model families, quantizers and context lengths.
• Better reconciling static aggregate metrics with dynamic and qualitative judgments, given the paper's finding that preference-based feedback revealed trade-offs invisible in the metrics used.

Target Audience

Researchers and practitioners working on retrieval-augmented generation, agentic search, and IR evaluation — particularly those building systems under hardware or latency constraints, and those interested in combining automatic metrics with qualitative or live user evaluation. The paper is also relevant to competition participants and organizers designing dynamic RAG evaluations, and to engineers who want a concrete, reproducible blueprint for a two-path (simple/complex) RAG service running on a single GPU.

Authors’ abstract

This paper presents the award-winning RMIT-ADM+S system for the Text-to-Text track of the NeurIPS~2025 MMU-RAG Competition. We introduce Routing-to-RAG (R2RAG), a research-focused retrieval-augmented generation (RAG) architecture composed of lightweight components that dynamically adapt the retrieval strategy based on inferred query complexity and evidence sufficiency. The system uses smaller LLMs, enabling operation on a single consumer-grade GPU while supporting complex research tasks. It builds on the G-RAG system, winner of the ACM~SIGIR~2025 LiveRAG Challenge, and extends it with modules informed by qualitative review of outputs. R2RAG won the Best Dynamic Evaluation award in the Open Source category, demonstrating high effectiveness with careful design and efficient use of resources.