Generative AI
Retrieval-Augmented Generation as an Evidence Pipeline
Map a RAG system from corpus and query through retrieval, context assembly, generation, support checks, and abstention.
By the end you can
- Define RAG as a composition of retrieval and generation components
- Separate parametric model knowledge from external non-parametric evidence
- Identify retrieval recall, context use, and claim support as distinct quality layers
- Decide when RAG is appropriate and when a database query or rules engine is better
Visual
RAG is a chain of evidence transformations
Every stage needs its own measurements and its own trace. The size of what each stage decides is known rather than rhetorical. The second box is not a menu of interchangeable options.
Swapping the retriever inside that box moves the whole system. Karpukhin and seven colleagues trained a dense dual-encoder retriever on a small number of question–passage pairs, and reported in 2020: “When evaluated on a wide range of open-domain QA datasets, our dense retriever outperforms a strong Lucene-BM25 system greatly by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA benchmarks.”
Nine to nineteen points of absolute top-20 passage retrieval accuracy is the spread hiding inside a single box of this diagram, between two ordinary choices for that box. Nothing downstream can recover a passage the retriever never returned. Nothing upstream can force the generator to use one it did. That is why the stages are drawn apart and measured apart.
- 1
Interpret the request
Extract entities, time, jurisdiction, intent, and missing constraints.
- 2
Retrieve candidates
Search sparse, dense, graph, database, or hybrid indexes.
- 3
Rerank and filter
Apply relevance, authority, freshness, permission, and diversity rules.
- 4
Assemble context
Select passages, labels, citations, and ordering within a token budget.
- 5
Generate or abstain
Produce claims conditioned on evidence and explicit uncertainty policy.
- 6
Verify and record
Check support, citations, tool results, and the complete retrieval trace.
Comparison
RAG, database lookup, and fine-tuning solve different problems
Choosing RAG by default can add unnecessary ambiguity. The structured column of this comparison is not free either.
Turning a natural-language question into a query over a schema has its own reference benchmark. Yu and eleven colleagues published Spider in 2018 and described it in one line: “It consists of 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables covering 138 different domains.” Eleven college students annotated it.
The same abstract records how hard the task was when the benchmark appeared: “the best model achieves only 9.7% exact matching accuracy on a database split setting”. Deterministic filters, joins and precise provenance are real advantages of the structured route. They all arrive after the question has been translated into the right query — a step with its own measured error rate. The choice between these three columns is a choice about where the uncertainty is placed, not about whether it exists.
RAG
Retrieve unstructured evidence and generate a contextual response.
- Useful for changing document collections
- Supports source-linked explanations
- Depends on retrieval and context quality
- Does not guarantee claim support
Structured query
Fetch exact fields, records, aggregates, or current state.
- Strong for well-defined schemas
- Supports deterministic filters and joins
- Produces precise provenance
- May need generation only for explanation
Fine-tuning
Change model behavior through weight updates.
- Useful for stable tasks and formats
- Does not provide a live evidence store
- Requires curated training and regressions
- Can complement rather than replace retrieval
External evidence changes the information boundary
A pretrained model carries statistical information in its parameters. RAG adds a runtime store that can be updated, permissioned, inspected, and cited without retraining the model.
The generator still decides how to use the retrieved text. It may ignore decisive evidence, combine incompatible sources, or continue beyond what the documents support. The boundary that moves is the boundary of what the system can be shown; the boundary of what it actually uses moves only if you measure it.
Case
Two systems that kept the knowledge store outside the weights
The arrangement got its name at NeurIPS in 2020. Lewis and colleagues paired a pretrained generator with a dense vector index of Wikipedia, reached through a neural retriever. Table 1 of the paper says what that store was worth in exact match. RAG-Sequence reaches 44.5 on Natural Questions, 45.2 on WebQuestions and 52.2 on CuratedTrec, with RAG-Token at 44.1 on Natural Questions. REALM, on the same test sets, reaches 40.4, 40.7 and 46.8. The closed-book T5-11B+SSM, which has no store at all, reaches 36.6 on Natural Questions and 44.7 on WebQuestions. The authors summarise it themselves: “On all four open-domain QA tasks, RAG sets a new state of the art (only on the T5-comparable split for TQA).”
The distance from 36.6 to 44.5 on Natural Questions is the whole argument of this lesson stated in one row of one table. The same class of generator, with an external evidence store attached.
REALM had already moved the retriever into pretraining itself. Guu and four colleagues published it in 2020. It backpropagates through a retrieval step over millions of documents, and outperformed previous explicit- and implicit-knowledge methods on three open-domain question answering benchmarks by 4–16% absolute accuracy. Both systems keep the store outside the weights. They differ in when the retriever is taught to use it.
Key idea
Retrieval success and answer success are separate claims
A retriever can return the answer-bearing passage while the generator overlooks it. The generator can also produce a correct answer from prior knowledge even when retrieval failed. The first of those two has been measured, and the result is worse than it sounds.
Where the evidence sits in the context window turns out to matter on its own. Liu and six colleagues found a U-shaped position effect in “Lost in the Middle”, published in 2024, and wrote: “For example, when relevant information is placed in the middle of its input context, GPT-3.5-Turbo’s performance on the multi-document question task is lower than its performance when predicting without any documents (i.e., the closed-book setting; 56.1%).”
The retriever had done its job. The answer-bearing document was in the context window — in the middle of it — and that was worse than handing the model nothing. Where a passage sits in the assembled context is therefore a design decision with a measured cost attached, not a formatting detail of the prompt.
End-to-end scoring alone hides these cases. Evaluate candidate recall, context precision, claim support, citation alignment, and abstention separately.
A RAG answer is reliable only when the evidence was found, selected, used, and represented faithfully.
Analogy
An open-book exam still requires finding and using the right page
Students in an open-book exam may consult the whole library, and the books help only if the right source is found, read correctly, and applied to the question that was actually asked. RAG supplies external material on the same terms. A student who cannot find the page writes a visibly thin answer. A generator that cannot find it writes a confident synthesis and attaches a nearby citation that does not support it.
That last sentence has been counted in shipped products. Four commercial generative search engines — Bing Chat, NeevaAI, perplexity.ai and YouChat — were hand-audited in 2023. Nelson F. Liu and two colleagues reported: “We find that responses from existing generative search engines are fluent and appear informative, but frequently contain unsupported statements and inaccurate citations: on average, a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence.”
On average only 51.5% of generated sentences were fully supported by their citations. Only 74.5% of citations supported the sentence they were attached to. These were products with the books open and the links printed on the page.
External knowledge improves the opportunity for support; the system must still demonstrate actual support.
Steps
Decide whether RAG belongs in the architecture
Start from the evidence contract rather than from how popular the pattern is. Step 5 is the one with a court record behind it.
Two lawyers and their firm filed a brief citing six judicial opinions that did not exist. ChatGPT had produced them, quotes and citations included. On 22 June 2023, in Mata v. Avianca, Judge P. Kevin Castel opened his sanctions order this way: “Peter LoDuca, Steven A. Schwartz and the law firm of Levidow, Levidow & Oberman P.C. (the “Levidow Firm”) (collectively, “Respondents”) abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question.”
Castel imposed a $5,000 Rule 11 penalty on the two attorneys and their firm, jointly and severally — “a penalty of $5,000 paid into the Registry of the Court”. He also ordered them to write to each judge falsely named as the author of a fake opinion: “Varghese”, “Shaboon”, “Petersen”, “Martinez”, “Durden” and “Miller”.
That is what step 5 buys. A specified support behavior is the difference between a system that reports it cannot find the authority and a system that produces one, in a filing, under a name.
1. Identify changing knowledge
List facts and documents that must update without retraining.
2. Define source authority
Specify who controls versions, permissions, and retirement.
3. Test structured alternatives
Use databases, APIs, or rules for exact current state.
4. Define retrieval metrics
Set recall and precision targets for the operating query distribution.
5. Specify support behavior
Require citations, claim alignment, clarification, or abstention.
6. Price the system
Include indexing, storage, latency, model tokens, and maintenance.
RAG creates inspectable knowledge access and a larger failure surface
RAG is valuable when evidence changes, access must be controlled, or answers should link back to sources. It is not a universal substitute for exact queries or domain-specific adaptation. The larger failure surface has been measured on closed, professional-grade products rather than on research prototypes.
The first preregistered evaluation of commercial RAG legal-research tools put 202 queries through them. Magesh and five colleagues published it in 2025 in the Journal of Empirical Legal Studies: “While hallucinations are reduced relative to general-purpose chatbots (GPT-4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time.”
The full text is more specific: “Lexis+ AI's answers are accurate (i.e., correct and grounded) for 65% of queries”, and “Westlaw's AI-Assisted Research is accurate 42% of the time”. Retrieval was in place. The corpora were curated. The vendors advertised hallucination-free citations. Those are the rates.
The next lessons inspect corpus construction, retrieval methods, reranking, grounding, and evaluation. Whether it works depends on the complete chain rather than on the acronym.
Key takeaways
- RAG combines external retrieval with generation and introduces failures at every transition; the retrieval step alone spans 9%-19% absolute top-20 accuracy between a Lucene-BM25 baseline and a dense retriever (Karpukhin et al., 2020).
- Parametric model knowledge and runtime evidence have different update, permission, and audit properties: RAG-Sequence reaches 44.5 exact match on Natural Questions against 36.6 for the closed-book T5-11B+SSM (Lewis et al., 2020).
- Retrieval recall, context quality, claim support, citations, and abstention require separate evaluation — the right document placed mid-context put GPT-3.5-Turbo below its own closed-book score of 56.1% (Liu et al., 2024).
- A database or rules engine is often better for exact structured and current state, but the question-to-query step is its own hard task: 9.7% exact matching accuracy for the best model on a database split of Spider (Yu et al., 2018).
- Citation presence does not establish that a passage supports the attached claim: across four commercial generative search engines only 51.5% of sentences were fully supported and only 74.5% of citations supported their sentence (2023 audit of Bing Chat, NeevaAI, perplexity.ai and YouChat).
- Use RAG when its evidence and maintenance benefits justify the added latency and complexity — commercial legal research tools still hallucinated between 17% and 33% of the time (Magesh et al., 2025), and Mata v. Avianca ended in a $5,000 Rule 11 penalty.