RAG Evaluation and Debugging

Later fixes cannot repair evidence that never entered the pipeline. An answer passes through six stages, and each one asks its own question. Corpus coverage: does the authoritative answer-bearing source exist and remain usable? Candidate retrieval: did that source enter the high-recall candidate set? Reranking and filters: did authority, permission, or ranking keep the right evidence? Context assembly: did decisive passages survive token budget and ordering? Generation and support: did the answer use the evidence faithfully and abstain when it should? Product workflow: did latency, review, and downstream action produce the intended outcome? Diagnosis starts at the first of those stages that failed, because every later stage inherits the failure.

Context assembly is the stage teams most often treat as neutral plumbing. It is not. A 2024 paper called "Lost in the Middle" ran multi-document question answering and key-value retrieval experiments, and reported where in the window the evidence has to sit: “In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.”

Read that as a debugging instruction. The retriever can succeed, the decisive passage can be inside the prompt, and the answer can still be wrong because of where in the window the passage landed. Buying a long-context model does not remove the effect. Position is therefore a variable your evaluation has to hold fixed and vary deliberately, exactly like the embedder or the top-k budget. A pipeline that logs which passages were assembled, but not in what order, cannot tell a retrieval failure from an ordering failure.

There is no one “RAG score” that represents the whole system. The clearest demonstration comes from products that were shipped and then measured. The first preregistered empirical evaluation of proprietary RAG-based legal research tools appeared in 2025, in the Journal of Empirical Legal Studies. It tested vendor claims of "hallucination-free" citations, and found this: “we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time”. These are retrieval-grounded commercial systems, in a domain with the best corpus in the world. The number that exposes them is a claim-support rate, not an answer-quality rate.

Grounding needs two numbers, not one. A 2023 audit scored four commercial generative search engines — Bing Chat, NeevaAI, perplexity.ai and YouChat — by human evaluation, keeping citation recall and citation precision apart: “on average, a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence”. Half the sentences unsupported and a quarter of the citations misattached are two different defects with two different fixes. A single “grounded?” flag would have averaged them into one meaningless figure.

So instrument three families and keep them apart. Retrieval metrics measure evidence availability: recall at the candidate budget, precision or relevance density, ranking metrics and no-answer behavior, sliced by source, language, and query type. Grounding metrics measure claim support: claim-level entailment, citation correctness and completeness as separate rates in the manner of the 51.5% and 74.5% split, unsupported-claim rate, and conflict and abstention behavior. Workflow metrics measure operational value and cost: latency and token usage, human-review burden, resolution and escalation outcomes, and incidents, appeals, and user corrections.

The case metadata determines whether a failure can be interpreted at all. Two of these categories are not optional extras. Abstention and distractor resistance have each been shown to collapse under measurement while ordinary accuracy stayed high, which is why they belong in the frozen set rather than in the postmortem.

Store the normalized query, filters, candidate IDs, scores, reranking features, selected spans, context order, prompt version, model version, citations, and validation results. Sensitive content can be protected or sampled under policy. Context order earns its place on that list on the evidence above: if the position of a passage inside the window can move an answer, a trace that records the selected spans but not their sequence has thrown away the variable that explains the failure.

Without a stage trace, teams tune prompts to compensate for corpus and retrieval failures. That can improve a demo while leaving the underlying evidence gap untouched. It can also produce the pattern the legal-tools audit found, where the answers look like research and the citations do not hold. A trace is what lets you say which of the two happened.

Require evidence from both frozen tests and fresh production-shaped cases. Freeze a protected set first, so that prompt and retriever tuning cannot reach the final gate. Score each stage separately — corpus, candidate recall, selection, grounding, workflow outcomes — rather than reporting one average. Inspect the critical slices: authority, time, language, tenant, risk, and no-answer behavior. Then calibrate the graders, run change attribution to identify which corpus, index, model, prompt, or policy version moved, and define rollback and reindexing so that compatible components can be restored while incident evidence is preserved.

Step 4 is the one with public evidence behind it. The TREC 2024 RAG Track deployed four relevance-assessment approaches in situ, across 77 runs from 19 teams: NIST's decades-old fully manual process, and three LLM-based variants using the open-source UMBRELA tool. UMBRELA's automatic judgments correlated highly with fully manual assessments at run level, on nDCG@20, nDCG@100 and Recall@100. Human assessors applied relevance criteria more strictly than UMBRELA did. The hybrid middle did not pay: “Surprisingly, we find that LLM assistance does not appear to increase correlation with fully manual assessments, suggesting that costs associated with human-in-the-loop processes do not bring obvious tangible benefits.” Two of the eight authors, Soboroff and Dang, are NIST staff, and NIST lists the paper in its own publication record.

That result tells you what an automated judge may decide, and what it may not. Ranking candidate configurations against each other at run level is a decision the evidence supports. Judging whether one particular passage is relevant is not. Neither is setting an absolute pass threshold on a strictness the grader applies more loosely than your reviewers do. A high-stakes release gate should not rest on an unvalidated grader alone.

A high average answer score is not enough if the team cannot tell missing evidence from poor ranking or unsupported synthesis. Every number in this lesson exists because someone measured a stage instead of an average. 51.5% against 74.5% separated support from attachment. 86% F1 falling to 66% separated answering from abstaining. 75% F1 falling to 36% separated comprehension from distractor resistance. 0.95 against 0.70 separated a grader's strong dimension from its weak one. In each case the coarser metric would have reported a system that worked.

Stage-aware evaluation turns the system into something that can be improved deliberately. The next lesson compares prompting, retrieval, tools, and fine-tuning. Diagnosis should determine which mechanism changes, not fashion or vendor defaults.