Large-Scale Biomedical RAG: Corpus Scaling, Document Representation, and Efficiency Trade-offs

AI summary

Overview

Research area: Natural Language Processing — biomedical Retrieval-Augmented Generation (RAG), specifically the interaction between retrieval-corpus scale, document representation, and operational cost.

Technical level: Advanced. The paper assumes familiarity with dense/sparse retrieval, rerankers, hierarchical indexing, and multiple-choice QA evaluation.

Scope (one sentence): A controlled, system-level evaluation of how progressively scaling a biomedical retrieval corpus and changing document representation (abstracts, flat full-text chunks, or TreeRAG hierarchies) affects retrieval quality, reranking value, downstream MCQA accuracy, latency, and index storage.

What This Paper Is About

Biomedical RAG systems are usually evaluated at one fixed retrieval-corpus size and with one document representation, so it is unclear whether design conclusions hold when the underlying literature collection grows. The authors build a controlled scaling protocol over PubMed and MIRIAD that keeps the gold source documents while adding distractor documents, then compare three representations end-to-end through retrieval, reranking, generation, latency, and storage. The goal is to determine which configurations remain effective and practical as biomedical collections grow, rather than at an isolated scale.

Key Contributions

1. A controlled corpus-scaling protocol that retains gold sources while progressively adding non-gold documents, and that explicitly separates source-document scale from the representation-dependent number of retrievable index units.
2. An end-to-end evaluation framework linking corpus growth and document representation to first-stage retrieval (Macro Recall@100), reranking (Recall@5), downstream MCQA accuracy, per-query latency, and Elasticsearch index storage.
3. Empirical evidence that small-scale conclusions do not persist, showing that advantages observed at smaller corpus scales do not consistently hold as the retrieval collection grows, which the authors use to argue that corpus scale should be an explicit evaluation variable.
4. Empirically grounded configuration guidance for choosing among abstract, flat full-text, and TreeRAG representations based on evidence format, generator behavior, corpus scale, and latency/storage constraints.

Main Findings

• Representation depends on evidence format. Abstract retrieval was strongest on MedAESQA, whose reference evidence is annotated at the abstract level, and reached the highest Macro Recall@100 (65.66%) with Qwen3-Embedding-8B at the full PubMed scale. On the five external benchmarks evaluated over full PubMed, Full-text and TreeRAG generally produced the strongest gains, with TreeRAG giving the best average improvement for Gemma-3 (+7.7 points) and MedGemma-1.5 (+9.6 points).
• Pseudo-abstracts were the weakest MIRIAD representation. Because MIRIAD does not provide separate abstracts, the authors generated LLM-based pseudo-abstracts, and Abstract consistently produced the smallest held-out gain on every generator.
• Reranking is not automatically useful. On MedAESQA, No-Reranker achieved the highest Macro Recall@5 at the full PubMed scale (25.28% with Abstract), and none of the rerankers improved on Qwen3-Embedding-8B's first-stage ranking while all added latency. On MIRIAD, reranking helped more as the pool grew, and Qwen3-Reranker-8B reached 80.00% Macro Recall@5 with TreeRAG at the large scale.
• RAG helped three of four generators on external benchmarks, with mean gains of up to 9.6 percentage points. Qwen3.5-Think was the exception, with negative average changes on all three representations (TreeRAG −0.5, Full-text −1.2, Abstract −2.0 points), despite an LLM-only average of 78.0 across the five benchmarks.
• All four generators improved on the held-out MIRIAD TEST split, by 10.2 to 30.3 percentage points. TreeRAG gave the largest gain for three of the four generators and stayed within 0.5 points of Full-text for Gemma-3.
• Bigger LLM-only baselines leave less room for retrieval gains. Qwen3.5-Think had the highest LLM-only accuracies (89.8 on MMLU-Med, 90.2 on MedQA-US, 84.3 on BioASQ-Y/N, 84.1 on MIRIAD TEST) and the smallest or negative RAG deltas; the authors note baseline strength alone does not predict whether retrieval helps.
• TreeRAG trades a modest amount of effectiveness for large efficiency savings. Relative to flat full-text chunking at the full PubMed scale, it uses approximately 67% fewer retrieval units (38.3M vs 117.4M) and reduces storage by 44–56% depending on the retriever.
• Latency favors lighter retrievers. On the full PubMed Abstract index for MedAESQA, BM25 averaged 10.8 ms per query, BGE-base-en-v1.5 53.9 ms, and Qwen3-Embedding-8B 1234.7 ms; BGE is roughly 23 times faster than Qwen3-Embedding-8B but less effective.
• Storage scales sharply with representation. At full PubMed, Abstract needed 5.9, 18.8, and 73.3 GB for BM25, BGE-base-en-v1.5, and Qwen3-Embedding-8B; Full-text needed 199.5, 481.0, and 1985.1 GB; TreeRAG needed 111.9, 228.1, and 872.0 GB.

Methodology in Plain English

The authors start from two source collections: PubMed full-text articles, which supply evidence for MedAESQA and the external benchmarks, and MIRIAD source passages, used only for MIRIAD. They build five shared scaling factors (control, tiny, small, medium, large) applied independently to both collections, plus an extra "full" setting for PubMed only, since PubMed can be expanded further — reaching about 4.3 million source documents. Crucially, they keep the documents known to contain the correct answer and only add non-relevant documents, so retrieval difficulty grows without changing the right answer. They count two different things separately: how many source documents are in the pool, and how many searchable units the retriever actually indexes, since one article can become one abstract, many chunks, or several tree nodes.

Each document is represented three ways: as a single abstract, as flat chunks of up to 1,600 characters with 200-character overlap from a recursive character splitter, or as a TreeRAG-style hierarchy. For MIRIAD, which lacks abstracts, they generate pseudo-abstracts with an LLM, leaving passages of at most 300 tokens unchanged.

They test three first-stage retrievers (BM25, BGE-base-en-v1.5 at about 0.1B parameters, and Qwen3-Embedding-8B), three rerankers applied to the top 100 candidates (BGE-Reranker-v2-M3, Jina-Reranker-v3, and Qwen3-Reranker-8B, the first two at 0.6B), and four 4B generator configurations (Gemma-3-4B, MedGemma-1.5-4B, and Qwen3.5-4B with thinking mode off and on). Because MedAESQA and MIRIAD are originally open-ended, they convert them to four-option MCQA with three generated distractors and balanced answer positions.

Retrieval is scored with Macro Recall@100 and reranking with Recall@5, matched at the source-document level so representations are compared fairly. Generation is compared under three conditions: LLM-only (question and options), RAG (top five retrieved units), and Oracle (gold evidence supplied directly, treated as a dataset-specific reference point rather than an upper bound). All pipeline choices are made on MedAESQA and the 200-instance MIRIAD development split, then frozen before testing on five external benchmarks (MMLU-Med, MedQA-US, MedMCQA, PubMedQA, BioASQ yes/no, using MIRAGE-preprocessed versions) and the 800-instance held-out MIRIAD test split. Efficiency is measured as mean per-query latency, reported separately for first-stage retrieval and retrieval-plus-reranking, and as total Elasticsearch index footprint.

Why This Matters

Impact on research: The paper shows that a common experimental shortcut — evaluating a RAG design at one convenient corpus size — can produce conclusions that reverse at scale. By making corpus scale an explicit variable and by separating source-document count from index-unit count, it gives other researchers a reusable protocol and a caution about reporting retrieval metrics as proxies for downstream quality.

Real-world applications:

• Clinical literature assistants and point-of-care question answering, where answers must be grounded in retrievable publications rather than parametric memory.
• Literature triage and evidence gathering for systematic reviews, where full-text evidence matters more than abstracts.
• Medical education and exam-style practice systems built on multiple-choice items.
• Deployment-constrained hospital or research IT settings, where storage and per-query latency budgets decide whether an 8B retriever or a hierarchical index is viable.

Industry relevance: The storage and latency numbers (terabyte-scale indexes for flat full-text chunking at 117.4M units versus 872.0 GB for TreeRAG with an 8B retriever, and a 23x latency difference between the compact and large dense retrievers) are directly actionable for teams sizing biomedical retrieval infrastructure. The finding that reranking can add cost without improving recall challenges a default assumption in many production pipelines.

Future Directions

• Isolate why TreeRAG helps. The authors state their experiments do not isolate hierarchical organization as the sole cause of TreeRAG's performance, leaving open whether the benefit comes from the hierarchy, the reduced unit count, or fewer split evidence boundaries.
• Broaden beyond PubMed and MIRIAD. The study is limited to two collections and one synthetic dataset family; whether the scaling patterns hold for other biomedical corpora and for MIRIAD-style data built differently is untested.
• Address the small and synthetic evaluation base. MedAESQA has only 40 questions, MIRIAD is synthetically constructed, and MIRIAD TEST is in-domain rather than fully external, so replication on larger human-authored clinical question sets is needed.
• Extend the scale and model grid. The full PubMed setting (~4.3 million source documents) is presented as the largest setting evaluated here, and only 4B generators were tested; whether conclusions hold for larger generators and larger collections remains an open question, as does the interaction between reranking utility and first-stage ranking quality across other corpora.

Target Audience

Researchers and engineers building retrieval-augmented systems over scientific literature, especially in biomedicine and healthcare; NLP practitioners who need to choose a document representation, retriever, or reranker under latency and storage budgets; and evaluation-focused researchers interested in corpus-scale effects and in why single-scale benchmarks can mislead. Readers without background in dense retrieval, reranking, and hierarchical indexing will find the measurement details demanding.

Authors’ abstract

We present a controlled, large-scale evaluation of how corpus scale, document representation, and reranking affect the effectiveness and efficiency of biomedical Retrieval-Augmented Generation (RAG) systems. Retrieval collections were progressively expanded while retaining the relevant source documents, enabling consistent comparisons across corpus sizes. We compared abstract, flat full-text, and hierarchical full-text representations across first-stage retrieval, reranking, and end-to-end multiple-choice question answering. Configurations selected on development data were evaluated on five external biomedical benchmarks and a held-out test set, while retrieval latency and index storage were measured to quantify operational cost. Performance patterns observed with small or controlled collections did not consistently persist as collections grew, and the usefulness of reranking depended on the corpus and retrieval setting. RAG improved three of four model configurations across the external benchmarks, with mean gains of up to 9.6 percentage points, and improved all four configurations on the held-out test set by 10.2-30.3 points. Abstract retrieval performed best when reference evidence was abstract-aligned, whereas full-text representations were generally more effective when richer evidence was needed. Hierarchical representation matched or outperformed flat full-text retrieval in most end-to-end comparisons while requiring 67% fewer indexed units and 44-56% less storage. These findings show that corpus scale should be treated as an explicit evaluation variable in biomedical RAG, and that the most appropriate design depends on evidence format, model behavior, and operational constraints.