RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching

AI summary

Overview

Research area: Computer vision / medical imaging — specifically evaluation metrics for AI-generated radiology reports, sitting at the intersection of vision–language models, large language model (LLM) judging, and clinical reporting standards.

Technical level: Intermediate. The core idea is conceptually simple (break a report comparison into findings, match them, grade them), but the paper assumes familiarity with report generation, LLM-as-judge metrics, and rank-correlation statistics such as Kendall's τb.

Scope: The paper proposes RadMatch, a multi-stage LLM-based metric that evaluates a candidate radiology report against a reference by extracting atomic findings, matching them, and counting clinically actionable errors, and validates it on two expert-annotated chest X-ray benchmarks.

What This Paper Is About

Radiology report generation systems are increasingly drafted by AI, but the metric used to judge those drafts matters as much as the generator. Current LLM-based metrics correlate best with radiologists, yet they emit a single opaque number that tells a clinician neither what went wrong nor how dangerous it is. RadMatch addresses this by decomposing report comparison into an explicit, persisted, finding-level matching, so every penalty can be traced to a specific finding pair and scored by clinical significance.

Key Contributions

1. A three-stage decomposition of report evaluation. RadMatch separates the task into (1) extraction of structured atomic findings from reference and candidate, (2) many-to-many matching of candidate findings to reference findings by clinical equivalence, and (3) significance-aware scoring of each resulting evaluation unit. Because the matching is persisted, the metric is auditable rather than a black-box score.

2. A significance-aware scoring scheme built on message-understanding categories. Each match is typed COR (correct), PAR (partial), or INC (incorrect), and each unmatched finding as MIS (missing) or SPU (spurious). Each unit carries a significance tier — critical, urgent, notable, or routine — adapted from the American College of Radiology actionable-reporting framework, with under- and over-calling a critical entity both judged at that entity's stakes. The headline score is the count of actionable errors per report.

3. A seven-dimension error profile plus deployment-oriented safety views. Each match is characterized along clinical status, longitudinal comparison, measurement, location, severity, morphology, and certainty. Objective dimensions (status, comparison, measurements) use deterministic comparators; only free-text dimensions are judged by the LLM. The framework additionally reports triage (critical/urgent) and actionable (critical/urgent/notable) recall and precision, plus per-subset views (normal, abnormal, longitudinal, measurement).

4. State-of-the-art clinical alignment with robustness across judge models and a low-cost deployment path. RadMatch is the most clinically aligned metric on both expert benchmarks, works with open-source judges on a single 32 GB consumer GPU in FP8, and is released as open-source code with an interactive dashboard for inspecting finding-level results.

Main Findings

• Strongest agreement with radiologists on both benchmarks. With the Claude Opus 4.8 judge, RadMatch reaches |τb| = 0.79 on ReXVal — matching the agreement among radiologists themselves — and |τb| = 0.58 on the harder RadEvalExpert, more than doubling the best prior metric (CRIMSON at 0.24). Lexical and clinical-concept metrics collapse to near-zero.

• Does not require a frontier proprietary judge. With the open-source Gemma 4 31B model, RadMatch reaches |τb| = 0.75 on ReXVal and 0.53 on RadEvalExpert, and can run on-premise on a single 32 GB consumer GPU in FP8, avoiding sending protected health data to a hosted API.

• Stable across a broad band of judge capabilities (Table 1). The strongest judges — Opus 4.8, the GPT models, Gemma 4 31B, Kimi K2.6 — all fall within |τb| ∈ [0.74, 0.79] on ReXVal and [0.53, 0.58] on RadEvalExpert. Agreement degrades modestly for mid-tier judges (Qwen3.5-35B-A3B at 0.69/0.44, DeepSeek-V4-Pro at 0.68/0.47) and remains usable for GPT-5.4-mini (0.66/0.41). It collapses only for MedGemma 1.5 4B (0.47/0.29), indicating a small, medically fine-tuned model is no better suited to this task than current stronger generalist models.

• Matched-model comparisons favor RadMatch. On ReXVal, RadMatch with GPT-4.1 reaches |τb| = 0.76 versus GREEN-GPT-4's 0.64, and RadMatch with GPT-5.2 reaches 0.74, above CRIMSON's reported clinical score of 0.68.

• Section-level results. On RadEvalExpert, RadMatch with Opus 4.8 achieves |τb| = 0.58 on findings and 0.44 on impressions, each roughly 2–3× the best prior metric for that section (GREEN at 0.20 and BERTScore at 0.23, respectively). Agreement is lower on impressions because impression sentences are more interpretive, fusing several observations into one diagnostic statement.

• Stable across candidate generators (Table 4). On ReXVal broken down by candidate generator (n = 50 each), RadMatch with Opus 4.8 scores τb of 0.75 on the CheXbert subset, 0.79 on BERTScore, 0.79 on RadGraph-F1, and 0.77 on BLEU, versus CRIMSON's 0.68, 0.71, 0.61, and 0.67 on the same subsets.

• A trivial single-call baseline matches correlation but loses entirely on structure. A count baseline (bare integer) with Opus 4.8 reaches 0.83 on ReXVal and 0.54 on RadEvalExpert; an enumerate baseline reaches 0.82/0.56. These match RadMatch on raw correlation, but even the enumerated list is unstructured free text: it attaches no significance tier, no error kind, no link to specific findings, and yields no persisted record that can be recomputed or audited.

• Diagnostic views expose what a scalar hides (Table 3). With the Opus 4.8 judge, RadMatch reports 1.51 actionable errors per report on ReXVal and 3.31 on RadEvalExpert. Triage precision/recall is 0.53/0.48 (ReXVal) and 0.44/0.28 (RadEvalExpert); actionable precision/recall is 0.49/0.44 and 0.36/0.25. On ReXVal the outcome counts are 193 COR, 92 PAR, 88 INC, 153 MIS, 116 SPU; on RadEvalExpert they are 707 COR, 404 PAR, 566 INC, 1471 MIS, 1767 SPU.

• Errors concentrate in specific subsets. Actionable errors by finding subset on ReXVal versus RadEvalExpert: normal 17 vs 121, abnormal 284 vs 1943, comparison 150 vs 743, measurement 7 vs 86. Abnormal and longitudinal-comparison findings carry most actionable errors on both benchmarks, while explicitly normal findings stay largely clean — consistent with the template-collapse mode observed in current report-generation models.

• Per-endpoint strengths and weaknesses (Table 5). Against all RadEval metrics, RadMatch (Opus 4.8) is strongest on significant false prediction (τb = −0.338) and competitive on omission (τb = −0.360, where label-based metrics lead with SRR-BERT at −0.512 and CheXbert at −0.503), and by design weakest on the non-clinical "inarticulate" category.

• Latency, cost, and stochasticity are reported as limitations. About 10.7 s per pair with a reasoning judge run serially, recovering to approximately 1.8 s at eight workers. Approximately $0.06 per pair with a frontier API judge ($12 on ReXVal, $35 on RadEvalExpert), with no per-token cost on-premise. The extraction, matching, and attribute-scoring stages are themselves LLM calls and therefore stochastic.

• Failure modes are characterized. Ranking ReXVal pairs by standardized residual between RadMatch's actionable-error count and the mean radiologist significant-error count, the 20 largest residuals split evenly into over-counts and under-counts. Over-counting occurs when a report dense with chronic or benign findings is tiered notable but a radiologist reads it as incidental, and when heavy atomization turns one clinical judgement into several scored units. Under-counting occurs when finding-level matching commits to the wrong correspondence, booking a dangerous mischaracterization as a minor partial rather than an omission plus a spurious finding. The metric is most reliable on reports of moderate length with an available indication.

Methodology in Plain English

RadMatch treats a report pair as a structured comparison problem rather than a single prompted judgement.

Step 1 — Extract findings. Each report (reference and candidate, independently) is decomposed into atomic clinical observations — the smallest units to which a clinical consequence can attach, such as "small left pleural effusion" or "no pneumothorax." Each finding is a record with typed slots: a status of normal or abnormal; free-text descriptors for location, severity, morphology, and certainty; a longitudinal comparison of stable, improving, worsening, new, resolved, or none; optional measurements formatted as (value, unit, category) triples; and a significance tier. The tier is entity-driven, not polarity-driven — a confident "no pneumothorax" still concerns a critical entity and is tiered critical, so flipping it is penalized as a critical-tier error rather than dismissed as agreement. Significance is modulated by clinical context: an aortic calcification is routine in an 80-year-old but notable in a 20-year-old.

Step 2 — Match findings. A single LLM call groups reference and candidate findings that describe the same clinical entity — same anatomical site and pathology, regardless of wording — into match groups. Groups are direct when one finding pairs with one finding, and aggregate when one side carries several, so matching is many-to-many and absorbs differences in atomization style. Findings in no group become omissions or spurious findings. A crucial design choice: a status conflict does not block a match ("small pleural effusion" and "no pleural effusion" concern one entity, so they are grouped and the disagreement recorded as a single status error), whereas an anatomical site mismatch is a different entity and resolves to an omission plus a spurious finding. A residual "generic" scope marks boilerplate such as "liver unremarkable," which earns no clinical-safety credit and is excluded from scoring.

Step 3 — Score by clinical consequence. Each match, omission, and spurious finding is an evaluation unit. Status is compared directly and any disagreement recorded as a status inversion. Longitudinal comparison is graded by clinical trajectory — a difference is major only when it crosses between the benign trajectory (stable, improving, resolved) and the active one (worsening, new), and minor within a trajectory. Measurements are compared per category against clinically motivated thresholds: a major error is a size difference above 20% (with a 2 mm floor suppressing small-structure noise), a ratio difference above 30%, any change in a count, or an attenuation difference above 20 HU or one crossing a tissue-characterization boundary (−10, 10, or 20 HU, separating macroscopic fat, lipid-rich adenoma, and simple fluid). A measurement the reference records but the candidate omits is major; one the candidate adds is minor. Location, severity, morphology, and certainty are scored by an LLM, which also labels each difference major or minor. Units typed INC, MIS, or SPU form the error set; COR, PAR, and generic units are excluded. Magnitude comes separately from the unit's significance tier, taken as the maximum among its findings, so under- and over-calling a critical entity are judged at the same stakes.

The reported score is the count of errors whose tier falls in the actionable set — critical, urgent, or notable (every tier but routine) — i.e. the number of clinically significant mistakes the candidate makes relative to the reference.

Implementation. Each report pair uses four LLM calls plus a deterministic tail: two independent extraction calls, one batched matching call, one batched attribute-grading call. When a study indication is available, an optional preliminary call extracts it and injects it as clinical context into the three downstream prompts. All other work — structured comparators, message-understanding typing, and every aggregate metric — is deterministic Python. Judges run with reasoning enabled and schema-constrained JSON output, prompted with per-modality few-shot exemplars reviewed by board-certified radiologists. Every extracted finding, matching, and per-attribute verdict is persisted as JSON, so all reported scores can be recomputed from the records without re-querying the judge.

Evaluation. Two expert-annotated chest X-ray benchmarks are used. ReXVal pairs each of 50 MIMIC-CXR studies with four candidate reports — the training-set reports scoring highest against the reference under BLEU, BERTScore, CheXbert embedding similarity, and RadGraph-F1 — giving 200 pairs, with six radiologists counting errors in each pair as ground truth. RadEvalExpert comprises 624 pairs from 208 studies drawn from MIMIC-CXR, CheXpert-Plus, and ReXGradient-160K, each paired with three candidates from CheXagent, the CheXpert-Plus model, and MAIRA-2; its findings (148 studies) and impression (60 studies) sections are annotated separately, each error graded significant or insignificant. Agreement is measured as the magnitude of Kendall's τb between each metric and radiologist significant-error counts, with study-clustered bootstrap 95% confidence intervals, and error-count metrics are negated so higher always means better.

Why This Matters

Impact on research. The paper makes an argument that goes beyond its own numbers: once a capable judge saturates correlation, a trivial single-call LLM error count already matches RadMatch on agreement. At that point the value of an evaluation metric shifts from raw agreement to interpretability, actionability, and auditability. RadMatch is designed for that regime — every penalty traces to a specific candidate–reference finding pair and a named attribute dimension, so a score can be inspected and contested finding by finding rather than taken on faith. This reframes what a good evaluation metric should provide and suggests that future work should be judged on more than a correlation coefficient.

Real-world applications:

• Model development and debugging for report-generation systems. A team iterating on a vision–language report generator can see where errors concentrate — abnormal findings and longitudinal comparisons carry most actionable errors, explicitly normal findings stay largely clean — rather than tuning against a single opaque score.
• Regulatory and quality-assurance review. The persisted finding-level record and safety views (triage and actionable precision/recall) give a reviewer an auditable trail from a penalty back to a specific finding pair and named attribute, which a scalar score cannot supply.
• On-premise clinical deployment. RadMatch with an open judge such as Gemma 4 31B runs on a single 32 GB consumer GPU in FP8, so protected health data need not leave the institution, at no per-token cost.
• Radiologist triage of AI drafts. Per-report diagnostics that name the error kind — a status inversion on an urgent finding versus a hallucinated feeding tube — are directly interpretable by a clinician deciding what to trust.

Industry relevance. The paper reports approximate cost of $0.06 per pair with a frontier API judge ($12 on ReXVal, $35 on RadEvalExpert) and roughly 1.8 s per pair at eight workers, which places structured, auditable evaluation within practical reach for routine evaluation pipelines. The authors position such metrics as helping developers build and evaluate report-generation models and iterate faster, while explicitly complementing rather than replacing expert oversight — any clinical use still requires final validation by radiologist judgment.

Future Directions

1. Extending beyond chest X-ray. RadMatch supports five anatomy–modality pairs (such as chest X-ray and abdomen CT) and is designed to extend to new modalities by adding appropriate few-shot examples, but both benchmarks used here are chest X-ray, because no expert-annotated CT or MRI benchmark with radiologist error counts is currently available. Building such benchmarks is the binding constraint on validating the claim.

2. Improving the matching stage. Under-counting failures arise when finding-level matching commits to the wrong correspondence, booking a dangerous mischaracterization as a minor partial rather than an omission plus a spurious finding. Better correspondence resolution would directly target the metric's most clinically consequential error mode.

3. Reducing latency, cost, and stochasticity. RadMatch issues several LLM calls per pair (about 10.7 s serially with a reasoning judge), and its extraction, matching, and attribute-scoring stages are themselves LLM calls and therefore stochastic. Fewer calls, cheaper judges, or more deterministic stages would broaden practical use.

4. Calibrating significance tiers against reader judgment. Over-counting occurs when reports dense with chronic or benign findings are tiered notable but a radiologist reads them as incidental, and when heavy atomization splits one clinical judgement into several scored units. Refining tier assignment and atomization policy — potentially conditioned on study indication, since the metric is most reliable on moderate-length reports with an available indication — is a natural next step.

Target Audience

Researchers and engineers building radiology report generation or vision–language models who need an evaluation metric they can debug against; medical imaging and clinical NLP researchers studying LLM-as-judge metrics and clinical-concept metrics; radiologists and clinical informatics specialists involved in validating AI-drafted reports or designing safety-oriented evaluation; and applied ML practitioners interested in auditable, on-premise-runnable evaluation pipelines, given the reported 32 GB GPU footprint and roughly $0.06-per-pair API cost.

Authors’ abstract

As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. Large language model (LLM)-based metrics are now the best-correlated with radiologist judgment, yet they output a single opaque score that neither a clinician nor a model builder can easily interpret or audit. We introduce RadMatch, a multi-stage, LLM-based metric that decomposes report comparison into a structured finding-level matching with significance-aware scoring and error characterization across seven clinical attribute dimensions (status, location, severity, morphology, certainty, longitudinal comparison, and measurement). The main score is the actionable-error count, both interpretable and auditable. Candidate findings are graded correct, partial, or incorrect, and unmatched findings are counted as missed or hallucinated. Triage and actionable safety recall/precision and per-subset views add complementary, deployment-oriented lenses. Across two expert benchmarks, RadMatch is the most clinically aligned metric, matching inter-radiologist agreement on ReXVal and more than doubling the best prior metric on the harder RadEvalExpert. Relying only on few-shot prompting, it is designed to extend to other modalities and anatomies. We will release RadMatch as open-source code with an interactive dashboard for inspecting results.