MedPAO-VL: Interpretable and Protocol-Compliant Agent for Structured Radiology Report Generation

AI summary

Overview

Research area: Medical computer vision and multimodal AI — specifically automated radiology report generation from chest X-rays (CXRs), with an emphasis on explainability, uncertainty handling, and adherence to clinical reporting protocols.

Technical level: Advanced. The paper assumes familiarity with vision-language models, slot attention, retrieval/ontology pipelines, and agentic plan-act-observe loops.

Scope: The paper presents MedPAO-VL, a multimodal agent that combines an uncertainty-aware, attention-explainable vision module with an iterative self-correcting generation loop to produce structured, ABCDEF-protocol-compliant CXR reports from the MIMIC-CXR and IU-XRAY benchmarks.

What This Paper Is About

Automated systems that write radiology reports directly from images are opaque: they cannot show how a conclusion was reached, they sometimes state findings that are not visible in the image, and they ignore the standardized formats radiologists use. A separate line of work (MedPAO) can reformat existing free-text reports into structured, protocol-compliant output but never reasons from the image itself. MedPAO-VL bridges these two lines by building a system that first detects pathologies with visually traceable evidence and explicit "uncertain" states, then iteratively checks and repairs its own narrative until it covers the detected findings, and finally structures the report according to the ABCDEF chest X-ray review protocol.

Key Contributions

1. MedPAO-VL architecture. A multimodal agent that unifies visual perception with a Plan-Act-Observe (PAO) loop, producing protocol-compliant reports with step-wise auditable reasoning rather than a single opaque generation pass.

2. Vision Concept Module with an uncertainty-aware labeling scheme. Pathology prediction is formulated as a three-class problem — present, absent, or uncertain — instead of forcing unmentioned findings into definite negatives, combined with a Pathology Slot Module that yields per-pathology attention maps as a by-product of the forward pass.

3. Concept Grounded Report Generator. A multimodal generator that receives both the CXR image and the list of present/absent concepts from the vision module, so that narrative statements remain traceable to discrete, verifiable findings.

4. Evaluation across clinical, structural, and interpretability dimensions. Includes radiologist scoring of structured reports, benchmark comparisons against published models on MIMIC-CXR and IU-XRAY, ablation studies of each component, and analyses of the learned embedding space for clinical and anatomical coherence.

Main Findings

• Benchmark clinical accuracy (unstructured reports, without the PAO loop): The 50K variant achieves CheXbert-F1 of 0.617 and RadGraph-F1 of 0.427 on MIMIC-CXR, with a CheXbert Micro-F1 of 0.706 and Macro-F1 of 0.617. For comparison, the paper reports MAIRA-2 at CheXbert-F1 0.416 / RadGraph-F1 0.346 and Flamingo-CXR at 0.519 / 0.205.

• Data efficiency: The 10K variant still reaches CheXbert-F1 0.563 and RadGraph-F1 0.376 on MIMIC-CXR, which the authors contrast with MAIRA-2 (1M training samples) and Flamingo-CXR (500K samples).

• IU-XRAY results: The 50K variant obtains CheXbert-F1 0.424, RadGraph-F1 0.401, ROUGE-L 0.341, and a GREEN mean of 0.733 (STD 0.225); the 10K variant obtains CheXbert-F1 0.203 and RadGraph-F1 0.342.

• Radiologist-rated structured reports: On 50 stratified MIMIC-CXR test cases scored blindly by two senior radiologists, MedPAO-VL scored a combined mean of 3.50 ± 0.28 on accuracy and 4.28 ± 0.20 on structure, versus MedGemma-27B at 3.18 ± 0.36 accuracy and 3.98 ± 0.24 structure. The abstract summarizes these as 3.5/5 accuracy and 4.3/5 structural score.

• PAO loop improves finding coverage: The deterministic sequential baseline had a concept coverage ratio (φ⁺) of 0.602 — meaning roughly 40% of vision-module-predicted findings were silently dropped — with CheXbert-F1 0.617 and RadGraph-F1 0.427. Full PAO raised φ⁺ to 0.803, CheXbert-F1 to 0.634, RadGraph-F1 to 0.448, and GREEN to 0.61 ± 0.20, with a mean of 2.8 refinement iterations, indicating genuine re-planning rather than single-pass termination.

• Caching recovers iterative cost: Cold-cache PAO used 11.2 mean LLM calls and 295 s; warm cache kept identical quality metrics while cutting calls to 9.8 and time to 163 s, a 45% reduction.

• Vision module ablations (MIMIC-CXR): Baseline (backbone + 236 independent heads, 2.7M trainable parameters) reached AUC 0.64, F1 0.485, precision 0.42, recall 0.574. Adding the PSM (47.2M parameters) gave AUC 0.679, F1 0.484, precision 0.41, recall 0.59. Adding a shared classifier head (6.1M parameters) gave AUC 0.77, F1 0.467, precision 0.44, recall 0.497. The full proposed model with similarity loss (6.1M parameters) reached AUC 0.784, F1 0.48, precision 0.478, recall 0.567.

• Concept grounding matters more than scale: MedGemma-4B zero-shot scored BLEU-4 0.019 / ROUGE-L 0.131 / METEOR 0.253; MedGemma-27B zero-shot scored 0.026 / 0.165 / 0.234; the 4B model fine-tuned without concepts (10K) scored 0.071 / 0.227 / 0.265; the proposed concept-grounded 4B (10K) scored 0.102 / 0.298 / 0.358.

• Clinically coherent embedding geometry: For the reference pathology "likely pneumonia", "hazy bilateral parenchymal opacities" (cos = +0.985, 9.7°) and "likely loculated fissural fluid" (cos = +0.985, 3.1°) align tightly, while "pneumothorax" (cos = −0.281) and "normal pleural surfaces" (cos = −0.348) are opposed. For "mild CHF", "mild pulmonary vascular congestion" (cos = +0.893) and "tiny pleural effusion" (cos = +0.696) project positively, while "emphysema with right apical bulla" (cos = −0.360) and "low lung volumes" (cos = −0.377) are displaced.

• Emergent anatomical organization: A t-SNE projection of the 236-dimensional slot activation space (stratified subset of 4,000 slot instances per category from the MIMIC-CXR test set, cosine distance) separates Airways, Breathing, and Circulation clusters, even though the model was never trained on ABCDEF categories.

• Faithful spatial localization: In a representative complex CXR with four co-occurring pathologies, each slot attended to a distinct region — the "blunt costophrenic angles" slot to the lower lung bases, the "calcified aortic knob" slot to the upper mediastinum, the "right pleural effusion" slot to the right lower hemithorax, and the "cardiomegaly" slot broadly across the cardiac silhouette.

Methodology in Plain English

The system has three connected parts.

Seeing, with uncertainty. A frozen MedSigLIP vision encoder processes each CXR at 448x448 resolution. A Pathology Slot Module then creates 236 "slots" — one per concept in a curated vocabulary — that each search the image for evidence of their own pathology. The key design choice is that the attention normalization runs over image regions rather than over slots, so pathologists' evidence for one disease is not suppressed by competition with another; this also means each slot produces its own interpretable heat map for free. Slots are refined over several iterations using a GRU update, and a single shared classification head predicts present / absent / uncertain for every pathology. Training combines cross-entropy with a redundancy penalty that discourages all slots from collapsing onto the same image regions.

Writing, grounded on concepts. A fine-tuned MedGemma 4B receives both the image and the list of detected present/absent concepts, and writes a fluent findings narrative. Concepts keep the text auditable; the image keeps the text natural rather than a list.

Checking and structuring. The Plan-Act-Observe loop generates a draft, then a Semantic Overlap Checker extracts clinical entities from the draft and compares them semantically with the concept list. If findings are missing, the agent re-plans and re-prompts only the discrepant section, for up to a set number of iterations (3 in the ablation experiments) or until all concepts are covered. MedPAO's existing tools — concept extraction, ontology mapping, a concept categorizer, and a structured report generator — then convert the free-text report into the ABCDEF format. A Dynamic Concept Cache stores frequent concept-to-ABCDEF mappings so repeated cases skip the slower ontology and categorizer calls.

Data. From MIMIC-CXR, studies containing explicit HISTORY, FINDINGS, and IMPRESSION sections yielded roughly 53,000 CXR–report pairs. Findings were extracted, normalized to SNOMED CT with a medical entity normalization model, and split into present/absent using DeepSeek-R1-Distill-Llama-70B, then radiologist-validated and filtered to 236 concept groups, giving approximately 50,000 (CXR, report, concept) triplets. Evaluation used the MIMIC-CXR and IU-XRAY test sets. The vision module was trained with a 90:10 stratified split on the "clear lungs" pathology, 50 epochs, AdamW with cosine schedule, learning rate 5×10⁻⁵, batch size 32, and λ_sim = 3, on a single NVIDIA A6000 (48 GB) GPU.

Why This Matters

Impact on research. The paper reframes report generation as an auditable, iterative process rather than a single forward pass. Its three-class present/absent/uncertain formulation directly attacks the closed-world assumption baked into most prior training pipelines, and its finding that concept grounding in a 4B model beats a 27B zero-shot model on generation metrics challenges the assumption that scale alone drives clinical quality.

Real-world applications:

• Draft report generation in high-volume radiology reading queues, where turnaround time is a bottleneck.
• Triage and prioritization support, since the vision module emits per-pathology evidence maps and explicit uncertainty rather than a single confident label.
• Quality assurance and teaching, where the ABCDEF structure and intermediate reasoning traces can be reviewed alongside the image.
• Second-reader or discrepancy-flagging tools, because missing-finding behavior is measurable through the concept coverage ratio.

Industry relevance. The reported 45% inference-time reduction from concept caching (295 s to 163 s) is a practical deployment signal for vendors building agentic clinical documentation, and the SNOMED CT normalization plus radiologist validation make the concept vocabulary interoperable with existing health-IT ontology infrastructure. The protocol-compliance angle addresses a common procurement objection to black-box generators.

Future Directions

• Vocabulary expansion. The authors explicitly note that diagnostic reach is capped by the predefined 236 concept groups, limiting documentation of ultra-rare anomalies; dynamically expanding the clinical vocabulary is named as future work.
• Broader modalities and workflows. The conclusion proposes extending the agentic framework to multi-modal imaging and wider diagnostic workflows beyond chest radiography.
• Reducing iterative latency further. With cold-cache PAO at 295 s and warm-cache at 163 s, further work on caching strategies, convergence criteria, and iteration limits is a clear practical need; the reported ablations fixed the maximum refinement steps at 3, so adaptive stopping is an open question.
• Harder validation. The radiologist study covered 50 stratified cases with two readers; inter-rater statistics, score distributions, and scoring criteria are deferred to the supplementary material, leaving room for larger multi-site and multi-reader evaluation.

Target Audience

Researchers and graduate students in medical vision-language modeling and clinical NLP; ML engineers building agentic or tool-augmented pipelines for regulated healthcare settings; radiologists and clinical informatics specialists interested in how structured protocols such as ABCDEF can be enforced computationally; and health-IT product teams evaluating explainability and auditability requirements for report-generation systems.

Authors’ abstract

Despite the remarkable generalization capabilities of Large Language Models, generating clinically accurate and structured radiology reports from medical images remains a significant challenge. Current end-to-end approaches function as black boxes, prone to factual hallucinations and generating ungrounded findings. This lack of reliability and failure to adhere to domain-specific structuring protocols, fundamentally limits their clinical utility. We present MedPAO-VL, a multimodal agent that shifts the paradigm from monolithic generation to explainable, iterative, protocol-compliant structured report generation. Unlike deterministic models, our plan-act-observe loop functions as a goal-oriented agent: it first perceives the Chest X-Ray through an explainable vision module designed to capture inherent diagnostic uncertainty, then iteratively refines the generated report by observing semantic discrepancies and self-correcting its generation for better alignment with visually detected clinical findings. Evaluation of our framework demonstrates that this agentic reasoning improves clinical correctness and completeness, achieving an expert-rated accuracy score of 3.5/5 and a structural score of 4.3/5. By prioritizing pathology specific explainability and iterative refinement over opaque end-to-end mapping, MedPAO-VL offers a transparent and clinically viable alternative for automated structured radiology report generation.