TIM: Temporal Decoupling with Iterative Mutual-Refinement Model for Longitudinal Radiology Report Generation

AI summary

Overview

Research area: Medical image analysis and vision-language modeling — specifically longitudinal radiology report generation (LRRG) from chest X-rays.

Technical level: Advanced. The paper assumes familiarity with vision-language models, contrastive alignment, video encoders, LLM-based generation, and clinical label extraction with CheXbert.

Scope: A two-stage framework (TIM) that separates static pathology recognition from temporal progression modeling and then iteratively refines prior and current reports against each other, evaluated on the Longitudinal-MIMIC dataset.

What This Paper Is About

Most automatic radiology report generators describe a single timepoint and miss how disease changes between visits. Existing longitudinal methods do use prior images and reports, but they blend static disease features and temporal-change features into one representation, which muddies both. The goal of TIM is to keep those two kinds of information separate, then use the prior and current reports as mutual references to correct each other's errors.

Key Contributions

1. Temporal decoupling: A framework that splits LRRG into a static pathological recognition branch and a dynamic progression modeling branch, so spatial disease cues and cross-visit evolution are encoded independently rather than mixed in one representation space.
2. Dual-timepoint report generation: Instead of predicting only the current report, Stage I generates both prior and current reports using a frozen LLM, with each report conditioned on the other as a reference, encouraging consistent temporal reasoning.
3. Mutual report refinement: A second stage that compares the generated prior report against its ground truth using CheXbert to produce error triplets, then feeds those errors plus an aggregated descriptor of the current report back into the LLM for iterative correction.
4. State-of-the-art results on Longitudinal-MIMIC: Reported gains over both single-image and longitudinal baselines across natural language generation (NLG) and clinical efficacy (CE) metrics.

Main Findings

• Overall performance: TIM-Stage II reaches BLEU-1 0.430, BLEU-4 0.124, METEOR 0.185, ROUGE-L 0.287, precision 0.563, recall 0.505, and F1 0.511 on Longitudinal-MIMIC, the best among all compared methods in Table 1.
• Gains over single-image baselines: +0.018 BLEU-4 and +0.030 ROUGE-L over RADAR, and +0.051 F1 over GMoD.
• Gains over longitudinal baselines: +0.025 BLEU-1, +0.011 ROUGE-L, and +0.037 F1 over Diff-RRG.
• Both stages matter: Stage I alone reaches BLEU-4 0.118 and F1 0.483; Stage II raises these to 0.124 and 0.511.
• Error propagation is real and measurable: The authors report that about 70% of errors in prior reports reappear in the current report due to temporal consistency across visits, which motivates the refinement stage.
• Stage I ablation: Adding dynamic progression modeling lifts BLEU-1 from 0.396 to 0.414 and F1 from 0.404 to 0.422; adding dual-timepoint generation raises BLEU-4 to 0.114 and F1 to 0.446; the static pathology alignment loss raises F1 to 0.460; the progression alignment loss raises BLEU-4 to 0.117; combining all components gives the best scores.
• Stage II ablation: Adding prior difference information yields precision 0.551 and F1 0.487 (versus 0.522 and 0.453 in the table's first row); adding the mutual-refinement loss gives BLEU-4 0.124 and F1 0.511. (The paper's prose states an F1 of 0.483 for the first row, which differs from the 0.453 shown in Table 3.)
• Embedding analysis: t-SNE projections and distance distributions show Stage II report embeddings sit closer to ground-truth embeddings than pre-refinement ones.
• Cross-LLM robustness: Swapping among LLaMA2-7B, LLaMA2-13B, Vicuna-7B, and Vicuna-13B preserves stable performance, with larger models scoring marginally higher.
• Perception frames: Zero frames degrades performance sharply; 1 to 3 frames stay strong with two best; 5 or 10 frames degrade slightly, attributed to redundant temporal interactions.
• Refinement iterations: Performance improves from one to three iterations and then plateaus, so three is used.

Methodology in Plain English

TIM works in two passes.

Stage I takes a prior image, a current image, and the prior report. It builds two separate feature streams. The first stream looks at each image independently to find disease patterns, and is trained to match short clinical descriptions of 14 disease categories (labeled not mentioned, positive, negative, or uncertain by CheXbert, and expanded using sources such as UMLS and PubMed). The second stream treats the two images as a tiny video: two learnable "perception frames" are inserted between the prior and current images, a video encoder processes the sequence, and a gating module fuses the two intermediate frame features. This stream is aligned with text describing progression, extracted from reports using Qwen3-plus. Both feature sets go into a frozen LLM (LLaMA2-7B), which writes both the prior report and the current report, each using the other timepoint's report as a reference.

Stage II then checks the generated prior report against the real prior report with CheXbert, producing a list of misclassified diseases. Those errors are encoded into a vector, and the initial current report is compressed into a descriptor by an attention-based aggregator. The LLM receives the current image features, the error vector, the descriptor, and a refinement prompt, and rewrites the current report. A dedicated loss pushes the refined report's text embedding closer to the ground truth than the first-round report's embedding. Only the aggregator and the error encoder are trained here; everything from Stage I is frozen.

At test time the two stages alternate: the refined current report becomes the reference for regenerating the prior report, which produces new error signals for the next refinement round. This loop runs for a fixed number of iterations (set to 3).

Setup: Experiments ran on 2 NVIDIA RTX 5880 GPUs (48 GB each), using Swin Transformer as image encoder, X-CLIP's visual branch as video encoder, CLIP's text branch as text encoder, and LLaMA2-7B as the LLM. The Longitudinal-MIMIC split contains 26,156 patients/92,374 samples for training, 203 patients/737 samples for validation, and 266 patients/2,058 samples for testing. Stage I trains for three epochs at learning rate 1×10⁻⁴; Stage II for one epoch at 5×10⁻⁵; both with batch size 4 and the Adam optimizer. Balancing coefficients are λ₁ = 0.5, λ₂ = 0.1, λ_Δ = 0.1, and β = 5.

Why This Matters

Impact on research: The paper argues that the field has been conflating two genuinely different problems — spotting a lesion in one image versus describing how it changed — and shows that separating them, plus letting reports critique each other, improves both language and clinical metrics. It reframes self-refinement as a two-timepoint, mutual operation rather than a single-report revision.

Real-world applications:

• Drafting comparison reports that explicitly state whether a finding has worsened, improved, or resolved since the last exam.
• Reducing repeated errors across a patient's visit history, where a misread prior report tends to contaminate subsequent ones.
• Standardizing follow-up documentation for longitudinal workflows such as ICU or oncology monitoring.
• Serving as a triage aid that flags suspicious interval changes for radiologist review.

Industry relevance: Because the LLM is frozen and only lightweight components are trained in Stage II, the approach is comparatively cheap to adapt to new base models — a practical property for vendors who want to swap in different LLMs, which the paper's four-model comparison supports.

Future Directions

• Beyond two timepoints: The framework is defined for a prior/current pair; extending the decoupling and mutual refinement to longer visit histories is an open question, especially given that more perception frames degraded results.
• Beyond chest X-ray and MIMIC: All experiments use Longitudinal-MIMIC, so generalization to other modalities (CT, MRI) and other institutions is untested.
• Richer error signals: Stage II currently relies on CheXbert's 14 disease labels for feedback; whether finer-grained or free-text feedback yields further gains is not explored.
• Iteration cost: Gains saturate after three refinement iterations, which means each additional round of LLM calls buys little; better stopping criteria or cheaper refinement paths remain open.

Target Audience

Graduate students and researchers in medical image analysis, vision-language modeling, and clinical NLP who are already comfortable with encoder-decoder architectures and LLM-based generation. It is also useful for clinicians and health-informatics practitioners interested in how longitudinal comparison reporting can be automated, though the dense notation and multi-stage training make it a poor starting point for beginners.

Authors’ abstract

Automatic radiology report generation (RRG) aims to translate medical images into diagnostic text, reducing radiologists' workload and standardizing clinical documentation. Nonetheless, existing approaches mainly focus on single-timepoint analysis and fail to capture temporal disease evolution across longitudinal examinations. While recent longitudinal RRG (LRRG) approaches incorporate historical data, they often combine images from different time points within a single representation space, leading to blurred semantics and inconsistent temporal reasoning. In this work, we propose a Temporal Decoupling with Iterative Mutual-Refinement Model (TIM), a two-stage framework that explicitly decouples spatial pathology from temporal progression and iteratively refines reports through mutual feedback. Stage I performs temporal-decoupled representation learning, separating temporal evolution patterns from disease-specific features and generating radiology reports for both prior and current studies. Stage II introduces a mutual report refinement mechanism that identifies diagnostic inconsistencies within prior reports and iteratively rectifies both prior and current reports through error-sensitive feedback. Experiments on the Longitudinal-MIMIC dataset demonstrate that TIM surpasses existing single-image and longitudinal baselines, achieving new state-of-the-art performance across both language and clinical metrics. Code is available at https://github.com/yihengd/TIM.