Retrieval Augmented Generation Compositionality

AI summary

Overview

Research area: Natural Language Processing — evaluation methodology for Retrieval-Augmented Generation (RAG) pipelines.

Technical level: Intermediate. The framing is accessible, but the study uses factorial experimental design, effect-size statistics (η²), and a joint quality/latency metric (hypervolume), so some familiarity with IR metrics (NDCG@10) and LLM-as-judge evaluation helps.

Scope: A large controlled study of how 450 RAG configurations compose across eight pipeline stages, eight benchmarks, and corpora from 5K to 200K documents, plus the open-source framework (lasagna) built to run it.

What This Paper Is About

New RAG techniques — better embedders, rerankers, query rewriting, agentic verification loops — are usually published after being tested one at a time, on one benchmark, at a small corpus size. This paper asks whether those isolated gains survive when the techniques are stacked together in a real pipeline, whether the choices that help retrieval also help answer faithfulness, and whether conclusions drawn at 5K documents still hold at 200K. The authors build a stage-isolated framework so that each stage can be swapped without disturbing the others, then measure interactions directly rather than inferring them from single-component ablations.

Key Contributions

1. A compositional study of RAG. 167 configurations from a D-optimal factorial design spanning eight configurable stages, plus 138 non-factorial ablation configurations across 24 targeted experiments — 450 configurations in total across eight benchmarks and corpus sizes from 5K to 200K documents.

2. Four operational RAG profiles. speed, standard, balanced, and max-reliable, distilled from the factorial and ablation sweeps as best-observed configurations targeting distinct points on the retrieval-quality / faithfulness / latency envelope.

3. lasagna, an open-source stage-isolated framework. A typed abstract base class per component type (eight component types), typed dataclass contracts between stages, a one-line @registry.register(...) decorator to add a method, 53 built-in components, six vector backends, and adapters for the eight benchmarks. Five automated isolation tests check that toggling one component does not silently change another.

4. Direct measurement of interaction effects, which prior configuration sweeps and benchmarking libraries (BERGEN, FlashRAG, greedy best-practice searches) do not estimate — with three pre-declared hypotheses (H1 additivity, H2 profile generality, H3 scale invariance) and their outcomes reported, including the one that was not met.

Main Findings

• Retrieval gains compose sub-additively. The three largest separately measured retrieval gains — cross-encoder reranking, disabling MMR, and document-awareness prefixes — predict a +46.6% NDCG@10 lift over the default_standard baseline (NDCG@10 = 0.412). The balanced pipeline combining them realizes only +17.2%, a 63% shortfall. Including all five separately positive retrieval techniques raises the shortfall to 72%; across 17 tested combinations it ranges from 7% to 72% with a median of 41%.

• Faithfulness gains compose approximately additively. A one-factor-at-a-time estimate based on BGE-large embeddings, hierarchical chunking, hybrid-BM25 retrieval, and Speculative-RAG gating predicts a +28% gain over a reference configuration; the best full pipeline improves by +32%.

• The pre-declared H1 test was not met. H1 asked whether two-way interactions explain at least 20% of faithfulness variance; they do not. A follow-up pre-specified test confirmed the flagged interaction is real (p = 0.007) but far below that bar.

• Retrieval and faithfulness are essentially unrelated across configurations. Pooled Pearson r = −0.01 (p = 0.91): 5% of configurations pair high faithfulness with low retrieval quality, and 6% show the reverse. The near-zero relationship also appears within each benchmark separately.

• Different stages drive the two metrics. For retrieval (n = 167 configurations), the largest marginal associations are embedding (η² = 0.164), enrichment (η² = 0.144), and reranking (η² = 0.090); compression and agentic gating contribute negligibly (η² < 0.01 each). For faithfulness (n = 122 configurations), query augmentation (η² = 0.224) and agentic gating (η² = 0.217) dominate, while embedding and enrichment are negligible (η² < 0.02 each).

• Two interactions survive multiple-comparison correction, both involving the embedder: Embedding × Enrichment (η² = 0.102) and Embedding × Reranking (η² = 0.110).

• MMR hurts on these corpora. It reduces NDCG@10 by 3.6% relative on average, opposite its usual benefit on broad, heterogeneous corpora; the authors hypothesize that diversity penalties demote passages that are in fact the same supporting evidence.

• No profile wins on every metric. On a held-out SQuAD v2 / MuSiQue / HotpotQA mixture over three seeds: standard reaches NDCG@10 0.860 ± 0.003, faithfulness 0.739 ± 0.016, mean latency 5.30 ± 0.05 s, hypervolume 0.523; balanced 0.840 / 0.911 / 12.25 s / 0.453; max-reliable 0.804 / 0.916 / 14.16 s / 0.389; speed 0.541 / 0.524 / 0.70 s / 0.277. At the fixed 30 s reference, standard has the highest joint hypervolume, while the agentic profiles lead faithfulness. balanced improves NDCG@10 by 4.5% and response time by 13% over max-reliable with only a 0.005 faithfulness difference.

• Rankings flip with corpus scale (H3 supported). standard leads at 5K and 15K; balanced leads from 25K onward and exceeds standard by 13.7% relative at 200K (p = 0.003, three seeds). The 25K and 100K gaps are directionally consistent but underpowered.

• Faithfulness does not degrade with scale. NDCG@10 falls log-linearly for speed and standard, but all three quality-focused profiles' faithfulness peaks around 50K, dips at 100K, and recovers from 100K to 200K. The authors hypothesize that larger corpora supply overlapping or paraphrased content that can still ground an answer.

• Preprocessing choices rival headline effects. Across 20 configurations, chunker choice spans 0.126 NDCG@10 — a 36% relative difference, larger than most factorial main effects.

• Older and simpler sometimes wins. The 2019 MS-MARCO MiniLM-L6 cross-encoder ranks above all four recent rerankers tested; Matryoshka truncation of BGE-M3 to 768 dimensions cuts vector storage and search cost by 25% while improving NDCG@10 by 2.3%; disabling hallucination and answer-quality gates gives the highest faithfulness, with gate-triggered refinement removing 14 of 15 chunks in one multi-hop example; HyDE raises NDCG@10 by 3.1% but doubles first-hop latency, cutting the latency-adjusted gain to 0.9%.

• Lighter pipelines can match quality at far lower latency. Across three other benchmarks at 100K–200K documents, lighter pipelines retain similar measured quality at 10–20× lower latency.

Methodology in Plain English

The authors first fix the pipeline architecture: eight stages (embedding, chunking, index-time enrichment, retrieval, query augmentation, reranking, context compression, and agentic quality gates), each behind an enforced interface so a swap in one stage cannot leak into another. This stage isolation is what makes it legitimate to attribute measured differences to the component being varied.

Running every possible combination would mean 4 × 4 × 4 × 3 × 5 × 4 × 2 × 4 = 30,720 configurations, estimated at roughly 40,000 GPU-hours and about $600,000 in API costs. Instead they draw a 242-run D-optimal subset via Fedorov coordinate exchange, sized to keep all main effects and all 28 dimension-pair interactions estimable; 167 of the 242 were executed (63 GraphRAG-enriched runs were dropped after an isolated ablation showed no measurable NDCG@10 change despite roughly 2× added latency, and 12 failed on timeouts). The executed factorial alone cost 247 GPU-hours, and the whole study about $5,200 and 310 wall-clock hours.

Rather than evaluating each benchmark separately, they pool the data into one unified multi-domain corpus so a single retriever must disambiguate across domains: five BEIR sets for retrieval (NFCorpus, FiQA, SciFact, ArguAna, HotpotQA) and three long-form QA sets for generation (SQuAD v2, MuSiQue, and others), yielding 149 stratified queries per configuration. Two tracks are scored: retrieval (5,132 documents, 93 queries, NDCG@10 with chunks de-duplicated to parent documents) and end-to-end generation (5,566 documents, 56 queries, RAGAS faithfulness and answer relevance judged by GPT-5.1). Roughly 358,215 individual query evaluations were recorded.

For scale, they extend the base corpus with 15K/25K/50K/100K/200K increments under the same proportional sampling protocol, with distractors drawn from the same five BEIR domains and no distractor overlapping any query's relevant set. Headline claims rest on ordinal comparisons because a two-judge sensitivity check (GPT-5.1 vs. Claude Sonnet 4.5) showed judge choice shifts absolute faithfulness but preserves profile ranking; a third judge (Claude Opus 4.7, 220 examples) agreed moderately (Pearson r = 0.44, Cohen's κ = 0.35) and supported the same relative trends.

Caveats the authors state themselves: the headline profile comparisons rest on one constructed English multi-domain test bed with at most 93 retrieval and at most 200 generation queries per experiment, the hypotheses were pre-declared and hashed internally rather than filed with a third-party registry, and the paper-specific artifacts (design matrix, per-query raw scores, analysis scripts) are not in the public repository but available from the authors on request.

Why This Matters

Impact on research: The paper challenges the standard reporting practice in RAG — a single-component ablation on one benchmark at ≤50K documents — by showing that such gains are not reliable predictors of performance inside a complete pipeline, and that the loss depends on which metric you care about. It argues for compositional evaluation as the reporting standard and supplies the tooling to make that practical.

Real-world applications:

• Enterprise search and knowledge assistants serving heterogeneous document stores where a single retriever must handle biomedical, financial, scientific, argumentative, and multi-hop queries from one vector store.
• Cost- and latency-budgeted deployment, where the 10–20× latency reduction at similar measured quality for lighter pipelines directly changes serving economics.
• Retrieval over large production corpora (100K–200K+ documents), where the 13.7% rank flip between profiles shows that small-scale tuning can select the wrong design.
• Domain-specific QA systems (biomedical, legal, customer support) that need grounded, verifiable answers rather than merely relevant retrieved passages.

Industry relevance: The practical recipes are directly actionable — chunker choice worth 0.126 NDCG@10, a 25% storage/search cost cut from Matryoshka truncation at 768 dimensions with a 2.3% NDCG@10 improvement, the finding that disabling certain quality gates improves faithfulness, and the observation that a stronger generator substitutes for a heavier pipeline (a stronger generator helps the lightest pipeline far more than the heaviest). The framework's plug-in interface means an organization can benchmark its own component against a fixed modern stack for about the price of a small experiment, and the paper reports per-configuration cost and latency to support constrained budgets.

Future Directions

• Estimate all 28 pairwise interactions directly. The authors note this needs 232 parameters, more than their design could support; a larger design or a different identifiability strategy would let interaction structure be measured rather than inferred from composition shortfalls.

• Re-validate small-corpus rankings at target scale. The 5K–200K crossover raises the question of how well these findings transfer to million-document corpora, for which the paper reports projections rather than measurements.

• Test whether the retrieval/faithfulness decoupling is intrinsic. The near-zero correlation (r = −0.01) between the two metrics suggests a need for objectives or reranking signals that serve grounding directly, rather than assuming better retrieval yields better answers.

• Extend beyond the current test bed. The paper's own limitations point to broader language coverage and more queries per experiment, and to understanding why faithfulness peaks near 50K and recovers from 100K to 200K — a mechanism the authors hypothesize about but do not isolate.

Target Audience

RAG practitioners and applied ML engineers choosing pipeline components for production; NLP researchers who publish retrieval or generation components and need a stronger evaluation protocol; and evaluation-methodology researchers interested in factorial design, interaction effects, and LLM-as-judge robustness. Readers who want a concrete recipe rather than a new model will find the four profiles and the ablation appendix directly useful; readers looking for a new architecture will not, since the contribution is the evaluation methodology and the framework rather than a novel RAG technique.

Authors’ abstract

Retrieval Augmented Generation (RAG) grounds large language models in external knowledge. Modern pipelines combine embedding, chunking, retrieval, reranking, query augmentation, context compression, and agentic verification, yet these techniques are typically evaluated one at a time. It remains unclear whether their isolated gains survive composition, whether retrieval quality and answer faithfulness favor the same design choices, and whether conclusions from small corpora hold at large scale. We pre-declare three hypotheses and introduce LASAGNA, an open-source framework for controlled RAG evaluation. We test 450 configurations across eight benchmarks and corpora ranging from 5K to 200K documents. Three findings emerge. First, individually strong techniques from different phases of a RAG pipeline do not necessarily compose: three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together. Second, component gains stack differently for retrieval and faithfulness, and optimizing one can hurt the other. Third, the best composition changes as the corpus grows: the combination that wins on a small corpus may not remain best at large scale. These findings show that gains measured in isolation are not reliable predictors of realistic performance, and that new RAG techniques should be tested with the full surrounding stack. We call on researchers to do so and supply LASAGNA to simplify these experiments.