A Living Benchmark for Information Retrieval from Electronic Health Records

AI summary

Overview

Research area: Clinical natural language processing and information retrieval, specifically evaluation of large language models deployed within electronic health record (EHR) systems.

Technical level: Intermediate. Readers should be comfortable with retrieval-augmented generation (RAG), long-context inference, agentic retrieval, and automated LLM-as-a-judge evaluation, but no specialized clinical or mathematical background is required.

Scope: The paper describes a scalable, clinician-validated pipeline that auto-generates question–answer pairs from longitudinal clinical notes to produce BRIE (the Benchmark for Retrieving Information in EHRs), and uses it to evaluate nine large language models under five inference configurations.

What This Paper Is About

Clinicians increasingly query LLM assistants embedded in EHR systems to review patient charts, but evaluating whether those systems retrieve the right information is hard: existing benchmarks are expensive to build by hand, limited in coverage, and decay as their contents leak into training data. This paper builds a generator that automatically produces clinically grounded question–answer pairs from de-identified notes, has clinicians validate the generator once, and then reuses it to create and refresh a benchmark without further annotation. Using that benchmark, the authors measure how state-of-the-art models and retrieval strategies fail on realistic chart-review questions.

Key Contributions

1. A validated, scalable benchmark generator. A two-stage LLM pipeline extracts and de-duplicates facts from longitudinal clinical notes, then generates question–answer pairs grounded in the admission History & Physical (H&P) note, with every answer tied to a permitted set of supporting facts. Clinicians validate the generator rather than authoring questions, so it can be rerun on new records.

2. BRIE, a released benchmark with verified answers and supporting evidence. Generated from 63,878 de-identified clinical notes across 68 patients at Stanford Health Care (median 400 notes per patient, range 102–9,319), the benchmark was filtered to 508 clinician-validated questions spanning reasoning complexity, clinical topic, and temporality.

3. A three-axis clinical retrieval taxonomy. Questions are conditioned on Reasoning (single-hop vs. multi-hop) and Topics, with Temporality assigned after generation based on where the supporting evidence sits in the longitudinal record, measured in tokens. This enables stratified rather than aggregate evaluation.

4. Demonstrations of evaluation that static benchmarks cannot support. The authors generate multiple clinician-validated answer interpretations for a subset of questions to test single-reference bias, and regenerate the benchmark from admissions two years later to test whether difficulty and failure modes are preserved without clinician filtering.

Main Findings

• Generated questions and answers were judged high quality. Across 675 generated question–answer pairs, 83.7% of questions (95% CI, 80.9–86.3) were judged clinically relevant and 97.8% (95% CI, 96.6–98.8) consistent with the patient chart; 73.3% (95% CI, 70.1–76.6) had realistic phrasing. Answers were 92.0% consistent with the chart (95% CI, 90.2–94.2), 89.8% complete (95% CI, 87.4–92.0), and 94.2% relevant (95% CI, 92.4–95.9). Supporting fact lists were 88.0% supporting (95% CI, 85.6–90.5) and 99.7% complete (95% CI, 99.3–100.0). The paper reports nineteen clinicians in the abstract and discussion but states that fifteen physicians with two independent reviewers per pair were recruited in the validation section.

• Omission, not hallucination, is the dominant failure mode. Fact recall ranged from 0.28 to 0.78 across all model–inference combinations. Under Recent inference, Claude Opus 4.7 reached 0.78 (95% CI, 0.76–0.80), Gemini 2.5 Pro 0.73 (95% CI, 0.70–0.75), and GPT 5.4 0.72 (95% CI, 0.70–0.74). In a random sample of 50 queries, 99.2% of extracted model response facts (95% CI, 98.5–99.6) were faithful to the clinical record, so lower fact precision (range 0.17–0.63) reflected additional chart-grounded content rather than fabrication.

• Open-weight models were competitive. Kimi K2.6, Qwen 3.5 397B, and Qwen 3.5 27B each reached approximately 0.74 recall (95% CI, 0.72–0.76, 0.72–0.77, and 0.72–0.76, respectively), while cost-efficient proprietary models fell further behind, most sharply Gemini 2.5 Flash Lite at 0.43 (95% CI, 0.40–0.46).

• Dense retrieval matched long-context recall at lower cost. Dense retrieval matched or exceeded Recent inference on fact recall while processing an average of 329,697 (69%) fewer tokens, and outperformed BM25. It raised average fact recall by 0.18, 0.06, and 0.03 for Gemini 2.5 Flash Lite, Claude Haiku 4.5, and GPT 5.4 Nano; BM25 changed recall by +0.07, –0.03, and –0.07 for the same models. Capping retrieval at the top 50 documents (~50K tokens) would reduce the inference cost of processing the full benchmark by approximately $837.43, or 86%, at Claude Opus 4.7 pricing; 65 of the 68 BRIE patients carry more than 50K tokens of source notes.

• Agentic retrieval did not improve recall. Agentic retrieval did not beat Dense, and for Claude Opus 4.7 and Claude Haiku 4.5 it underperformed Recent inference, reducing fact recall by 0.08 and 0.04.

• Multi-hop and longitudinal questions drive the largest failures. Multi-hop minus single-hop recall gaps were –0.20 for Agentic, –0.14 for BM25, –0.13 for Dense, –0.12 for Recent, and –0.10 for Recent-180K. When the earliest supporting fact appeared more than 180K tokens before the query (63 questions), recall fell by an average of –0.16 for Recent and –0.25 for Recent-180K, with no significant drop for the other inference methods.

• Topic matters. Under Recent inference, recall was lower for comorbidities (0.68; 95% CI, 0.67–0.69) and disease progression (0.67; 95% CI, 0.66–0.68) than for imaging (0.76; 95% CI, 0.75–0.77) and diagnostic testing (0.76; 95% CI, 0.74–0.77).

• Single-reference scoring underestimates capability. Of 100 sampled questions, 87 yielded at least one additional valid interpretation, averaging 3.18 valid interpretations per question (range 1–5), with 279 valid generated interpretations in total for those 87 questions. Scoring against the most favorable validated reference increased fact recall by 0.12 (95% CI, 0.12–0.13), a 31.4% relative improvement (95% CI, 28.9–33.93%), and fact precision by 0.14 (95% CI, 0.14–0.15), a 45.4% relative improvement (95% CI, 42.7–48.0%). Top and bottom methods were unchanged, but Kimi with Recent-180K moved from 10th to 4th in recall.

• The benchmark can be regenerated without losing difficulty. Across BRIE (n=508), BRIE_unfiltered (n=675), and BRIE_new (n=1,000, built from 2026 admissions two years later), mean fact recall with Qwen 3.5 27B was 0.73 (95% CI, 0.72–0.74), 0.73 (95% CI, 0.73–0.75), and 0.70 (95% CI, 0.70–0.71); fact precision was 0.46 (95% CI, 0.45–0.47), 0.46 (95% CI, 0.45–0.46), and 0.42 (95% CI, 0.41–0.43). Kolmogorov–Smirnov tests found no significant distributional differences in nearly all comparisons, with the exception of temporally distant facts.

Methodology in Plain English

The authors define the target task as answering a clinical question about a specific patient at a specific point in time, using only information available in the record up to when the admission H&P note was written, and returning a free-text answer supported by evidence. To build questions automatically, they run a two-stage pipeline: an LLM first extracts and de-duplicates candidate facts from all notes written before the H&P, and then those facts plus the H&P are given to Gemini Pro 2.5 in a single prompt that produces question–answer pairs. Because answers must be supported exclusively by the extracted facts and not by the H&P itself, the questions test retrieval from the prior record rather than recall of a summary. Questions are conditioned on reasoning type (single-hop vs. multi-hop) and clinical topic; temporality is assigned afterward based on token distance to the supporting evidence. Clinicians then review the generated items against 14 criteria covering the question, answer, and fact set, and disagreements about relevance were adjudicated by a third physician.

For evaluation, exhaustive manual grading was infeasible, so the authors developed and validated two automated LLM-as-a-judge methods: fact recall and fact precision via fact entailment against the reference answer, plus a pairwise win-rate metric scored for completeness, relevance, and concision. Nine models were run under five inference configurations: two long-context baselines (Recent and Recent-180K, the latter capped at 180,000 tokens), BM25 sparse retrieval, dense semantic retrieval, and an agentic configuration that performs LLM-guided retrieval through iterative tool use with conditional stopping.

Why This Matters

The paper reframes how clinical LLM evaluation should be done: rather than treating a benchmark as a fixed artifact that decays, it treats benchmark generation as a reusable, clinician-validated process that can be rerun as documentation practices and models change. It also shifts attention from hallucination to omission, arguing that omissions are harder to detect because a response can look plausible while silently dropping critical information.

Real-world applications:

• Monitoring deployed EHR-embedded chat assistants. Hospitals using LLM chart-review tools can generate institution-specific question sets and track which retrieval settings their systems handle reliably.
• Admission and handoff chart review. The task formulation mirrors what a clinician does when admitting a patient, so results speak directly to how well assistants support that workflow.
• Deciding on retrieval infrastructure. The cost–performance comparison gives concrete guidance: dense retrieval matched long-context recall while processing 69% fewer tokens on average, with an estimated 86% cost reduction in the top-50-document configuration at Claude Opus 4.7 pricing.
• Building evaluation sets at new institutions. Because the generator needs one-time validation rather than per-item authoring, other health systems can build benchmarks from their own records to capture local distribution shifts.

Industry relevance: The results matter to EHR vendors and health systems deciding whether to expand context windows or invest in retrieval, to model developers targeting clinical deployment, and to any organization that needs ongoing, contamination-resistant evidence about whether a clinical AI system is safe to trust for chart review.

Future Directions

• Improving retrieval for multi-hop and longitudinal questions. Existing strategies, including Dense and Agentic, did not consistently surface evidence spanning multiple encounters; the paper identifies this as the central unresolved failure.
• Making multi-reference evaluation standard. The demonstration that single-reference scoring underestimates capability by 31.4% in relative fact recall motivates developing evaluation harnesses that credit alternative valid clinical interpretations.
• Detecting omission errors in practice. Since the dominant failure is missing information rather than fabrication, methods that flag absent-but-relevant facts are an open problem.
• Extending regeneration across institutions and years. The framework is designed to build new BRIE versions from records collected in different years or at different institutions; the paper notes the remaining limitation discussion as ongoing future work, though the truncated text does not fully enumerate those limitations.

Target Audience

This paper is most useful to clinical NLP and clinical informatics researchers building or evaluating retrieval systems; health system leaders and EHR vendors assessing deployed LLM assistants; model developers targeting clinical chart review; and benchmark designers interested in living, contamination-resistant evaluation. Clinicians involved in AI governance and validation will also find the failure-mode taxonomy directly actionable.

Authors’ abstract

Large language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how clinicians retrieve and synthesize information from patient records. Their safety and utility depend on rigorous evaluation, yet existing benchmarks are manually curated, costly to update, and rapidly become obsolete with evolving technological advancements. We present a scalable framework that automatically generates question--answer pairs from longitudinal EHR notes. Nineteen clinicians validate the benchmark generator, producing the Benchmark for Retrieving Information in EHRs (BRIE), a continuously maintainable evaluation dataset. Across nine LLMs and five inference strategies, state-of-the-art systems frequently omit clinically important information, particularly for questions requiring synthesis across multiple documents and encounters. Because the generator itself is validated, BRIE supports evaluations that static benchmarks cannot, including the generation of multiple answers that reflect variation in clinician reasoning for robust performance assessment and continuously refreshing benchmark content to guard against leakage. Our results demonstrate that scalable benchmark generation enables rigorous, up-to-date evaluation of clinical LLMs as they are deployed in rapidly evolving healthcare settings.