CliQ-RRG: Clinical-Knowledge Guided Disease-aware Visual-Textual Alignment for QA-Style Radiology Report Generation

AI summary

Overview

Research area: Natural Language Processing / medical vision-language generation — specifically automatic radiology report generation (RRG) from chest X-rays.

Technical level: Advanced. The paper assumes familiarity with contrastive learning, cross-attention, retrieval-augmented generation (RAG), knowledge graphs, and clinical labeling schemes such as CheXbert.

Scope: The paper proposes CliQ-RRG, a two-stage framework that aligns chest X-ray images, report text, and predicted disease labels, then uses graph-guided retrieval plus a large language model to rewrite reports as structured question-and-answer pairs, evaluated on MIMIC-CXR, IU X-Ray, and zero-shot on ReXGradient-160K.

What This Paper Is About

Radiology reports take time and expertise to write by hand, and automatic systems that generate them tend to produce long, unstructured paragraphs in which the important diagnostic findings are buried. Existing models also frequently ignore prior scans of the same patient, do not use disease labels as supervision for aligning images with text, and rarely ground their output in external clinical knowledge.

CliQ-RRG's goal is to make generated reports shorter, more explicit, and more clinically grounded by converting them into question–answer pairs (each question targets a clinical finding, each answer is Yes or No) followed by a concise diagnostic summary, while enriching the intermediate report with evidence retrieved from a medical knowledge graph.

Key Contributions

1. First contrastive-alignment framework for QA-style RRG. The authors state CliQ-RRG is the first framework to use contrastive alignment for QA-style radiology report generation, restructuring unstructured narratives into concise question–answer pairs.
2. Disease-aware Visual–Textual Alignment with prior-guided attention. An alignment module aligns image and text representations through predicted disease embeddings, combined with a Prior-Guided Attention Module (PrAM) that fuses current and prior chest X-ray views.
3. GraphRAG-guided generation pipeline. A graph-guided retrieval stage enriches the intermediate report with graph-grounded clinical evidence, after which a fine-tuned Llama-3-8B-Instruct model synthesizes the final QA-style report with a diagnostic summary.
4. Two-tier clinical validation. A Likert-scale rating by three medical professionals on 1,000 sampled QA pairs, plus an independent assessment by a senior radiologist with over 20 years of experience on 500 stratified QA pairs.

Main Findings

• State-of-the-art positioning claimed across benchmarks: CliQ-RRG is reported to achieve state-of-the-art performance across most metrics on MIMIC-CXR and IU X-Ray, outperforming named baselines including RADAR, REVTAF, DAD-RG, MPO, the QA-style baseline CoD, RA-RRG, and MLLM-based MAIRA-2 and MedGemma across the reported metrics. The specific metric values in the comparison tables are not available in the provided text (the tables appear as placeholders).
• Clinical Efficacy advantage: On MIMIC-CXR, CliQ-RRG is reported to outperform REVTAF on Recall and F1 in CheXbert-based Clinical Efficacy metrics.
• Ablation shows label-aware alignment beats CLIP-style alignment: With the base model, adding visual and textual channels yields only marginal gains (BLEU-4: 0.197, F1: 0.482). Adding PrAM improves F1 to 0.493; combining both reaches F1 0.508. Replacing CLIP-style alignment with Di-VTC raises BLEU-4 to 0.224. Adding graph-guided retrieval reaches F1 0.577, and the full graph-guided framework produces gains of +29.0% on NLG metrics and +31.9% on CE metrics over the base configuration.
• Statistical significance: Ablation results are reported as mean ± standard deviation over five runs, with the full model (g) showing statistically significant improvement over the base model at p ≤ 0.05.
• Hyperparameter optimum: Equal weighting of the generation and classification losses, λ1 = λ2 = 1.0, is optimal across all metrics, and contrastive temperature τ = 0.07 gives peak performance among the tested values {0.03, 0.05, 0.07, 0.10}.
• Expert evaluation of QA pairs: Of 500 QA pairs assessed by a senior radiologist, 467 were fully acceptable, 31 clinically acceptable, and 2 hallucinated — a 99.6% clinical reliability rate and a 0.4% hallucination rate.
• Zero-shot generalization: Evaluated without training or fine-tuning on ReXGradient-160K (3 health systems, 79 medical sites). Surface-level metrics show a moderate decrease under distribution shift, while semantic and clinical metrics remain relatively stable. Models trained on MIMIC-CXR consistently outperform those trained on IU X-Ray. Exact values are not reported in the provided text.
• Qualitative improvement: Report examples show the base model producing generic statements such as "no acute abnormality"; adding PrAM introduces findings such as "no evidence of consolidation", Di-VTC surfaces observations like "focal consolidation, pleural effusion, or pneumothorax", predicted disease embeddings capture terms such as "cardiomegaly", and graph-guided retrieval produces phrases such as "stable and unremarkable" closer to ground truth. Grad-CAM confirms focus on relevant anatomical regions.
• Multi-state disease classification: The classifier predicts each of 13 disease categories independently over CheXbert's four states (uncertain, negative, positive, not mentioned) rather than a single shared distribution; per-state performance is deferred to Appendix I.

Methodology in Plain English

The framework has two stages.

Stage 1 — alignment. Current and prior chest X-rays are encoded with a BioMedCLIP visual encoder. A Prior-Guided Attention Module uses the current image features as a query and the prior image features as keys and values in cross-attention, producing a unified visual representation that mixes current and historical views. Reports are split into salient clinical phrases separated by [SEP] tokens and encoded with CXR-BERT. A separate classifier predicts, for each of 13 disease categories, one of four CheXbert states from the visual features, using a learnable disease embedding matrix.

The distinctive piece is a contrastive objective called Disease-aware Visual–Textual Contrast (Di-VTC). Instead of treating every image–text pair as a positive match, pairs are considered positive when their predicted disease label vectors are identical or sufficiently similar according to a threshold, and negative otherwise. A bidirectional loss pulls positive pairs together and pushes negatives apart in a shared embedding space, so alignment is organized around diagnostic similarity rather than mere co-occurrence.

The three losses — contrastive, disease classification (cross-entropy against CheXbert-derived labels), and report generation (negative log-likelihood) — are combined with weights λ1 and λ2, both set to 1.0.

Stage 2 — knowledge and restructuring. A text decoder generates an intermediate free-text report conditioned on the visual channel, textual channel, and predicted labels. That report then acts as a retrieval query against a medical knowledge graph whose entities include anatomy, findings, and diseases, with graph entities initialized using BioWordVec embeddings. Retrieval relevance combines semantic cosine similarity with a diagnostic signal: an indicator that fires when the entity's mapped disease category has a predicted label at or above a threshold τ. Top-ranked entities seed a bounded breadth-first traversal that retrieves a subgraph, and multi-hop relation paths are ranked and serialized into a graph evidence context, which is concatenated with the intermediate report. Finally, a fine-tuned Llama-3-8B-Instruct model, guided by a prompt, converts this knowledge-enriched report into Yes/No QA pairs and a diagnostic summary.

Training setup: implemented in TensorFlow, trained on an NVIDIA 64 GB GPU with AdamW, 30 epochs, batch size 16, learning rate 3e-5, weight decay 0.01.

Data and evaluation: MIMIC-CXR (377,110 scans, 227,827 reports, official split) and IU X-Ray (7,470 frontal and lateral scans, 3,955 reports, 7:2:1 splits). Narrative output is scored with BLEU, METEOR, ROUGE-L, CIDEr, and BERTScore, plus CheXbert-based Clinical Efficacy (F1, Precision, Recall). Because QA output differs from narrative text, QA pairs are evaluated with BERTScore, CheXbert-F1, and RadGraph-F1, with expert evaluation treated as the primary evidence of clinical validity. Inter-annotator agreement uses pairwise agreement and Fleiss' kappa.

Why This Matters

Impact on research. The paper argues that narrative free-text reports make key findings hard to retrieve and hard to compare, and that most prior systems neglect diagnostic labels as supervisory signals and neglect prior scans at the visual feature level. CliQ-RRG reframes report generation as structured, knowledge-grounded QA generation, and reports a zero-shot cross-institution evaluation on ReXGradient-160K covering 3 health systems and 79 medical sites, which addresses the common criticism that RRG results do not transfer beyond the dataset they were trained on.

Real-world applications.

• Drafting structured radiology findings that a clinician can quickly accept or correct, reducing documentation time.
• Longitudinal follow-up review, where prior scans of the same patient are folded into the current interpretation.
• Quality assurance and audit, since Yes/No QA pairs are easier to scan and compare systematically than paragraphs.
• Cross-site deployment support, given the reported stability of semantic and clinical metrics under institution and device shift.

Industry relevance. The framework is built from publicly available components (BioMedCLIP, CXR-BERT, BioWordVec, Llama-3-8B-Instruct) and the authors provide code at https://github.com/chandravardhan-singh-raghaw/CliQ-RRG. The reported 99.6% clinical reliability and 0.4% hallucination rate on the sampled QA pairs are the kind of figures vendors and regulators would scrutinize, though the authors are explicit that LLM restructuring is not jointly optimized with multimodal reasoning and that supervised paired data is required.

Future Directions

• Beyond 2D chest X-rays. The authors state the study focuses on 2D CXR and note extension to other anatomical regions and modalities remains future work.
• Jointly optimized multimodal reasoning. The LLM currently serves as a fixed restructuring component rather than being jointly trained, which the authors list as a limitation.
• Explicit temporal modeling. PrAM incorporates historical scans but does not encode temporal order or the time intervals between examinations, so longitudinal progression patterns are not directly modeled.
• Reducing data dependence. Training is limited to MIMIC-CXR and IU X-Ray and relies on supervised pairs of images, reports, and diagnostic labels, which the authors note may constrain scalability in data-scarce settings.

Target Audience

Researchers and graduate students working on medical vision-language models, radiology report generation, or retrieval-augmented clinical NLP; clinical informatics teams evaluating structured reporting formats; and practitioners interested in contrastive alignment methods that use diagnostic labels as supervision. Readers need a working knowledge of transformers, contrastive objectives, and knowledge-graph retrieval to follow Stage 1 and Stage 2 in detail.

Authors’ abstract

Radiology reports are essential for diagnostic reasoning and patient care, yet their manual preparation is time-consuming and cognitively demanding. Automatic radiology report generation (RRG) offers a scalable alternative, but existing models often produce lengthy, unstructured narratives that overlook diagnostic cues and multi-view information. We present Clinical-Knowledge Guided Disease-aware Visual-Textual Alignment for QA-Style Radiology Report Generation (CliQ-RRG), a unified two-stage framework for interpretable and clinically grounded reporting. In Stage 1, CliQ-RRG employs a Disease-aware Visual-Textual Alignment module that aligns image and text representations using predicted disease embeddings, reinforced by a Prior-Guided Attention Module (PrAM) to capture multi-view dependencies across current and prior scans. In Stage 2, GraphRAG-guided retrieval augments intermediate textual representations with graph-grounded clinical evidence, and a large language model restructures them into concise, interpretable question-answer (QA) pairs with diagnostic summaries. Experiments on two public chest X-ray benchmarks demonstrate that CliQ-RRG consistently outperforms prior methods across both lexical and clinical metrics, generating accurate and clinically coherent QA-style radiology reports. Code is available at: https://github.com/chandravardhan-singh-raghaw/CliQ-RRG.