Agentic RAG Evaluation: Budget Allocation Across Questions, Trajectories, and Reads

AI summary

Overview

Research area: Information retrieval (cs.IR), specifically the evaluation methodology of agentic retrieval-augmented generation (RAG) systems.

Technical level: Intermediate. The paper is readable without deep statistical training, but it relies on variance-component reasoning, generalizability theory, and standard IR evaluation conventions (standard error, paired comparisons, bootstrapped confidence intervals).

Scope: A single paper that measures how a fixed evaluation budget should be split across questions, search trajectories, and repeated reads when comparing two agentic RAG search policies on HotpotQA and MuSiQue.

What This Paper Is About

When researchers compare two retrieval-augmented generation policies, they have to decide how to spend a limited evaluation budget: ask more distinct questions, let each question follow more search trajectories, or have the reader answer each evidence packet more times. Each choice changes how precisely the comparison can be made and how much it costs in model tokens and search requests.

The authors run matched-cost allocations of these three resources on HotpotQA and MuSiQue, then test whether a small prior variance audit can predict the precision of larger allocations. The goal is practical guidance on which budget axis to expand, and how far an audit can be trusted to extrapolate.

Key Contributions

1. A matched-cost measurement of three budget allocations. Three designs on a 446-question panel — A (446 questions / 2 trajectories / 1 read), B (174/2/5), and C (298/3/1) — were fixed in advance to a 34.35M model-token target, and their realized precision and complete costs were measured.

2. Two audit-based precision forecasts tested against new outcomes. A nested variance-component model and a Q-only baseline, both fitted on a separate 96-question audit, predict the A/B/C standard errors within 4.0% and 3.5% respectively.

3. A direct probe of extrapolation beyond the audit. All trajectory subsets and per-trajectory read subsets in a 48-question depth control and all read subsets in B were enumerated to test whether the two-trajectory audit carries any forecasting advantage when trajectory count changes.

4. Full cost accounting and a one-read efficiency analysis. Search requests, model tokens, caching behavior, and search-price sensitivity are reported, together with a fixed-budget D-study objective that quantifies the variance penalty of using one read instead of the fitted optimum, and a matched temperature control on answer consistency.

Main Findings

• Broader question coverage wins on model tokens. A, B, and C consume 34.20, 34.14, and 34.39M model tokens (a difference below 0.8%), and A lowers measured standard error by 33.3% relative to B and 12.6% relative to C. Both SE-ratio intervals favor A.

• Forecasts are accurate on the pre-specified allocations. Measured A/B/C SEs are 1.03/1.54/1.18 F1 points; the archived nested model predicts 1.07/1.59/1.19 (within 4.0%) and Q-only predicts 0.99/1.59/1.22 (within 3.5%). B retains the audit's T=2, R=5, so both predictions coincide there by construction. Refitting dataset means in the old audit shifts predictions by less than 0.7%.

• No clear extrapolation advantage past the two-trajectory audit. On 48 questions with one read, empirical SE falls from 4.21 at T=1 to 2.83 at T=5. Across both read settings the nested model's MAE is 0.18 versus Q-only's 0.37, a paired advantage of 0.19 [−0.06, 0.36], but this gain comes from T=1,2. For T=3,4,5 the errors are 0.16 versus 0.13 with an advantage interval of [−0.25, 0.25].

• Read scaling behaves differently from trajectory scaling. In B, empirical SE shrinks only from 1.67 to 1.54 as R grows from one to five, and the nested and Q-only MAEs are 0.045 and 0.039.

• The feedback contrast itself is imprecise. The broad A allocation yields a policy contrast of +1.66 [−0.27, +3.65] F1 points, and the Adaptive/Open complete model-token ratio in A is 3.68.

• Fee rankings depend on search price. Search requirements are 5,435/2,128/5,041 distinct requests for A/B/C (logical operation counts 6,235/2,433/6,249). C/A's variance-times-total-fee ratio is 1.31 [1.05, 1.61] with free search and 1.25 [1.01, 1.54] at $1 per 1,000 requests, with a search-dominated limit of 1.21 [0.97, 1.49] — so more questions beat more trajectories over this range. B/A moves from 1.01 [0.62, 1.40] toward 0.88 [0.55, 1.23], and its interval includes one at every nonnegative price, leaving the question-versus-read monetary ranking unresolved.

• One-read penalties are usually small but uncertain on Pro. Across the 15 model/panel/selector settings, fitted one-read penalties are 0–9.9% in estimator variance at a common model-token budget. Pro/RRF's fixed-T penalty is 9.9% with a bootstrap range of 0–57.1%, while Pro/RR's point penalty is only 0.26%.

• Component estimates are unstable with few trajectories. On the 48 Flash questions, raising T from two to five shifts question/trajectory shares from 69.5/16.5% to 31.6/55.0%.

• Direct one-read versus two-read check. The variance-times-token ratio is 0.736 [0.727, 0.761] for Flash and 1.134 [0.913, 1.636] for Pro/RRF (single batches 0.69–1.58). The narrow Flash interval reflects the average of two same-question paired comparisons, with a stable token ratio of 0.719 [0.715, 0.722] and reader noise contributing only 3.1% of the fitted mean variance.

• Lowering temperature reduces disagreement but not comparison precision. Temperature zero cuts answer disagreement from 14.3% to 3.4%. Typical one-read SE is 2.38 F1 points versus 2.36 at temperature 0.7, a ratio of 1.01 [0.91, 1.14], and the paired feedback change is +0.41 [−1.53, +2.42].

Methodology in Plain English

The authors compare two search policies that share their first query and first set of search results. The "Open" planner writes all four queries up front from the question alone; the "Adaptive" planner writes queries two through four using evidence already retrieved. Both end with a reader that answers from five documents, selected either by reciprocal rank fusion (RRF, summing 1/(60+r) across distinct query lists) or by round robin (RR).

Retrieval runs over FineWeb through DeepResearchGym with seven partitions, ten results per query, and at most five 5,000-character document prefixes per packet. The planner and reader are DeepSeek-V4.1-Flash and DeepSeek-V4-Pro-0813, used with non-thinking requests, temperature 0.7, and a 256-token output cap.

The measurement target is the answer-F1 difference between Adaptive and Open, averaged over selectors, with each question first averaging its trajectories and reads. Standard errors decompose into question, trajectory, and read variance components via V(Q,T,R) ≈ σ²_Q/Q + σ²_T/(QT) + σ²_R/(QTR), with negative component estimates truncated at zero. Paired 95% percentile confidence intervals resample complete questions within datasets 10,000 times, and the cost-side analysis refits variance and cost jointly in 3,000 stratified question bootstraps.

The prediction side uses a separate archived audit of 96 questions with two trajectories and five reads. Its parameters and the variance formula were frozen before the new executions, and forecasts were computed retrospectively so that no parameter was chosen using the new outcomes. The Q-only baseline simply scales the audit's per-question standard error by 1/sqrt(Q). For efficiency, a fixed-budget objective (Tσ²_Q + σ²_T + σ²_R/R)(c_T + Rc_R) is minimized over R from 1 to 20, and the reported loss is the excess estimator variance from using one read at the same model-token budget.

Why This Matters

Impact on research. Evaluation budgets for agentic RAG are usually spent without measurement of what each resource buys. This paper supplies a matched-cost comparison in which the answer is measured rather than assumed, and it shows that a cheap audit can predict precision on the allocations it was designed for while failing to certify extrapolation to different trajectory counts. That distinction between interpolation and extrapolation is a methodological caution for anyone who calibrates an evaluation from a pilot study.

Real-world applications:

• Benchmark design and leaderboard construction — deciding whether to add questions or repeated runs when a new agentic RAG system must be ranked reliably.
• Enterprise RAG deployment evaluation — teams comparing in-house retrieval policies can use the reported model-token and search-request costs to budget a comparison, and the search-price sensitivity curves to see when retrieval fees rather than model fees dominate.
• Vendor and model selection — the Pro/Flash split shows that guidance from a cheap model workflow does not automatically transfer to an expensive one, which matters when procurement decisions rest on small evaluation panels.
• Prompt and decoding configuration decisions — the temperature control supports the view that lowering temperature reduces answer disagreement without necessarily sharpening the policy comparison.

Industry relevance. The paper's cost side is directly operational: distinct search requests, cache-aware model fees, and a variance-times-fee ratio are the quantities that determine whether an evaluation is affordable. The finding that one-read Flash evaluation is efficient, while the 96-question Pro panel gives insufficient information for a reliable read-allocation default, tells practitioners where to spend and where to gather more evidence first.

Future Directions

• Extend the audit beyond two trajectories. The paper explicitly calls for an audit with more than two trajectories plus a check of the empirical precision curve, since the depth subsets showed no reliable forecasting advantage for the nested model when extrapolating T.
• Execute independent equal-dollar allocations. Allocations were matched on model tokens; the authors state that equal-dollar designs were not executed, so the monetary comparison rests on sensitivity curves rather than a direct experiment.
• Resolve the question-versus-read fee ranking. The B/A fee interval includes one at every nonnegative price, leaving open what a properly powered equal-dollar comparison of more questions versus more reads would show.
• Re-test the audit under different operating conditions. The evidence covers one policy comparison, one model family, one FineWeb endpoint, and two QA datasets with F1 around 30, and the fee results depend on the recorded interleaved cache conditions — all boundaries the authors flag as untested.
• Diagnose Pro further. The Pro panel's one-read penalties range up to 9.9% with a bootstrap range to 57.1%, which the paper treats as insufficient basis for a read-allocation default for that route.

Target Audience

• IR and NLP evaluation researchers designing benchmarks, pilot studies, or power analyses for agentic RAG comparisons.
• Practitioners running RAG evaluations who must choose between adding questions, trajectories, or repeated reads under a fixed token or dollar budget.
• Statisticians and methodologists interested in generalizability-theory allocation applied to multi-stage LLM pipelines with real serving costs.
• Graduate students in information retrieval who want a worked example of complete cost accounting and bootstrapped variance estimation for a paired policy contrast.

Authors’ abstract

Evaluation budgets in agentic retrieval-augmented generation span questions, search trajectories, and repeated answers. We measure allocation precision, reading efficiency, and cost boundaries using a retrieval-feedback comparison on HotpotQA and MuSiQue. At 34.14--34.39M model tokens, broader question coverage lowers standard error by 33\% versus five reads and 12.6\% versus three trajectories. Archived nested and Q-only forecasts predict these allocations within 4.0\% and 3.5\%, respectively. Depth subsets establish no clear forecasting advantage beyond the two-trajectory audit. One-read variance penalties relative to the fitted optimum at the same token budget are 0--9.9\%, with substantial Pro uncertainty. Under recorded model fees, more questions beat more trajectories at search prices of \$0--1 per 1,000 requests; question-versus-read fee rankings remain unresolved. Temperature zero cuts answer disagreement from 14.3\% to 3.4\% while comparison precision stays similar. \par\medskip\noindent\textbf{Keywords:} Agentic RAG; Evaluation budget; Generalizability theory; Repeated sampling.