Regional Attention-Enhanced Swin Transformer for Clinically Relevant Medical Image Captioning

AI summary

Overview

Research area: Medical image captioning / radiology report generation, combining vision transformers (Swin), a regional attention module, and a language decoder (BART with PubMedBERT embeddings).

Technical level: Advanced. The paper assumes familiarity with encoder–decoder vision–language models, cross-attention, token pooling, and standard captioning metrics (ROUGE, BLEU, CIDEr, METEOR, BERTScore).

Scope: The paper proposes and evaluates a Swin–BART captioning system with a lightweight learnable regional attention block on the ROCO dataset, reporting ROUGE and BERTScore gains over ResNet-CNN and BLIP2-OPT baselines alongside qualitative case studies.

What This Paper Is About

Automated medical image captioning aims to turn radiological images into diagnostic narratives that can support reporting workflows, but medical captions must describe subtle pathological findings, anatomical relationships, and imaging modalities using domain-specific terminology, which is much harder than generic image captioning. The authors build an encoder–decoder system that deliberately amplifies diagnostically salient image regions before the language decoder attends to the image. The goal is to produce accurate, clinically phrased captions while also showing which image regions drove each description.

Key Contributions

1. A learnable regional attention mechanism. A lightweight module operates on flattened Swin feature maps and assigns region-wise importance scores via a linear transformation plus softmax, intended to emphasize pathological regions and suppress normal anatomy before cross-attention.
2. A Swin–BART architecture with biomedical embeddings. A Swin Transformer Base encoder (ImageNet-pretrained) is paired with a BART-base decoder whose token embedding layer is replaced by PubMedBERT's 768-dimensional word embeddings, with PubMedBERT embeddings frozen during initial training and BART decoder layers fully fine-tuned.
3. Comprehensive evaluation on ROCO with CNN and Transformer baselines. Training and evaluation on the ROCO dataset, compared against ResNet-CNN and BLIP2-OPT, with reported improvements in ROUGE and BERTScore.
4. Interpretability via regional attributions. The attention weights are presented as heatmaps that visualize the regions driving each generated description, positioned as a transparency mechanism rather than a purely architectural choice.

Main Findings

• ROUGE improves substantially: Proposed model 0.603, versus ResNet-CNN 0.356 and BLIP2-OPT 0.255.
• BERTScore improves substantially: Proposed model 0.807, versus BLIP2-OPT 0.645 and ResNet-CNN 0.623.
• Lexical and consensus metrics are weaker for the proposed model: BLEU 0.257 (ResNet-CNN 0.311, BLIP2-OPT 0.217), CIDEr 0.215 (ResNet-CNN 0.296, BLIP2-OPT 0.231), METEOR 0.081 (ResNet-CNN 0.084, BLIP2-OPT 0.0920). The paper attributes the modest BLEU and CIDEr to the difficulty of lexical precision and human consensus in a specialized domain.
• Results are reported as mean ± standard deviation over three seeds with 95% confidence intervals. The specific standard deviation and confidence interval values do not appear in the content provided.
• The abstract states that ablations (regional attention on/off, token-count sweep), per-modality analysis (CT/MRI/X-ray), and paired significance tests were performed. The numeric results of these analyses are not shown in the content provided.
• Three qualitative case studies: a chest CT (ground truth described a large cystic mass causing cardiac compression; the model generated "Contrast-enhanced tomography of chest computed scan showing a large mass on the left wall"), a cervical spine CT (model correctly identified the lateral cervical radiograph view and cervical spondylodiscitis and a triangular fragment), and an abdominal aorta CT (model correctly identified digital angiography and the abdominal aorta but was weaker on the interventional device description).
• Dataset scale and composition: ROCO contains over 81,000 radiology images with clinical captions from CT, MRI, and X-ray modalities, sourced from PubMed Central articles. The split is 65,000 training pairs, 8,100 validation pairs, and 8,100 test pairs, with article-level separation so no article appears in more than one split.
• Clinical coverage statistics: 40% CT, 30% MRI, 30% X-ray; 65% pathological cases, including 20% with multiple findings; average caption length is 21 words with 7 medical terms per caption.
• Training configuration: 5 epochs, batch size 8, AdamW with learning rate 1e-5, weight decay 0.01, cross-entropy loss, dropout 0.1 in the projection layer, early stopping with patience 3 epochs, seed fixed to 42, run across 3 independent runs, on an RTX 3090 GPU with 48 GB of RAM.
• Decoding configuration: beam search with beam size 4, length penalty 1.1, no_repeat_ngram_size 3, and max length 128.
• Ablations reported but not numerically shown in the provided content; the abstract lists a regional attention on/off comparison and a token-count sweep.

Methodology in Plain English

The system is an encoder–decoder, like a translator between images and text.

The encoder is a Swin Transformer Base model pretrained on ImageNet. Images are resized to 224×224, ImageNet-normalized, and augmented with random flipping, rotation of ±10 degrees, brightness/contrast adjustments of ±20%, and Gaussian noise injection. Grayscale medical images are replicated into three channels. The encoder turns each image into a 7×7 grid of 1024-channel feature maps, which are then flattened into 49 positions.

The regional attention module is the paper's central innovation. A single learned linear transformation plus a bias produces one score per spatial region, passed through a softmax so the scores sum to one. These weights are applied to the 49 region vectors and summed, so the module is effectively learning which parts of the image matter for the caption. The design intent is clinical: regions with pathological findings should receive higher weights, and normal anatomy should be down-weighted.

The projection and pooling step maps the attended features from 1024 dimensions into 768 dimensions via a learned projection matrix, then adaptively average-pools the sequence down to 29 tokens. This gives the decoder a compact representation while keeping it small enough to be computationally efficient.

The decoder is BART-base, but its original token embedding layer is surgically replaced with PubMedBERT's 768-dimensional pretrained word embeddings, giving the model biomedical vocabulary from the first tokenization step. PubMedBERT embeddings stay frozen early on while the BART decoder layers are fully fine-tuned. Captions are generated autoregressively, and the decoder's cross-attention uses queries from its own previous states while keys and values come from the encoded image features. Text is tokenized with the PubMedBERT tokenizer and captions are truncated or padded to 128 tokens, which the paper says covers 95% of clinical descriptions.

Training minimizes standard cross-entropy between generated and ground-truth captions using teacher forcing. Crucially, the regional attention parameters are optimized jointly with the rest of the network, so the region weighting is learned through backpropagation of the language modeling loss rather than from any separate region annotation.

Why This Matters

Impact on research. The paper sits in the line of work coming out of the ImageCLEF medical caption challenge, where prior winners used CNN ensembles with medical concept integration (CMRE-UoG, 2022) or a frozen large language model bridged through a Q-Former (closeAI2023, using BLIP-2 with ViT-g and OPT-2.7B). This work shows a comparatively compact design can lead on semantic metrics such as ROUGE and BERTScore. It also argues for interpretability as a first-class output: the attention weights function as a built-in explanation for why a given description was generated.

Real-world applications.

• Drafting preliminary radiology report text that a radiologist reviews and edits, reducing documentation time.
• Highlighting suspicious image regions as a reading aid, giving a visual second opinion about where the model looked.
• Quality-assurance and teaching tools, where trainees compare model-generated descriptions against ground-truth findings.
• Research infrastructure for medical vision–language work, since the paper states that code is released to support reproducibility and that ROCO licensing and terms of use are followed.

Industry relevance. Medical documentation is a major cost center in radiology workflows, and vendors of clinical imaging software have strong incentive to automate report drafting. The paper explicitly positions the output as research and decision support with a human in the loop, which is the regulatory posture any such product would need. The architecture also matters commercially: using a relatively compact decoder with frozen biomedical embeddings is cheaper to train and deploy than approaches built on frozen multi-billion-parameter language models.

Future Directions

• Stronger attention variants and supervision. Extend the attention block with multi-head and gated variants, and compare against supervision derived from pathology regions rather than learning attention purely from the captioning loss.
• Stronger and more matched baselines. Add a medically tuned BLIP-2 and a ViT-G + Q-Former variant without the regional attention, to isolate how much of the gain comes from the architecture versus the Swin–BART combination itself.
• Quantify the interpretability claim. The paper proposes a pointing-game / IoU evaluation on a labeled subset, since heatmaps that look plausible are not the same as attention that actually localizes pathology.
• Broaden evaluation and address weaknesses. Evaluate on additional corpora such as MIMIC-CXR and real reports, report per-modality calibration, apply modality-aware sampling to bolster MRI, add preprocessing ablations and paired significance tests across all metrics, and explore terminology constraints or concept dictionaries to improve device and procedure descriptions.
• Open question the paper leaves: the model was trained on 5 epochs and the current design uses natural-image pretrained weights, which the authors state may not fully exploit domain-specific features.

Target Audience

Researchers working on medical vision–language models and radiology report generation will get the most from this paper, particularly those interested in attention-based interpretability and in comparing against ImageCLEF-challenge-style baselines. It is also useful for engineers building clinical documentation tools who need a compact, deployable architecture rather than a large frozen language model, and for clinicians or informaticists evaluating whether such systems can be safely placed in a human-in-the-loop workflow. Readers without a background in transformer-based captioning will need to consult the architecture details carefully, since the paper assumes familiarity with cross-attention and standard captioning metrics.

Authors’ abstract

Automated medical image captioning translates complex radiological images into diagnostic narratives that can support reporting workflows. We present a Swin-BART encoder-decoder system with a lightweight regional attention module that amplifies diagnostically salient regions before cross-attention. Trained and evaluated on ROCO, our model achieves state-of-the-art semantic fidelity while remaining compact and interpretable. We report results as mean$\pm$std over three seeds and include $95\%$ confidence intervals. Compared with baselines, our approach improves ROUGE (proposed 0.603, ResNet-CNN 0.356, BLIP2-OPT 0.255) and BERTScore (proposed 0.807, BLIP2-OPT 0.645, ResNet-CNN 0.623), with competitive BLEU, CIDEr, and METEOR. We further provide ablations (regional attention on/off and token-count sweep), per-modality analysis (CT/MRI/X-ray), paired significance tests, and qualitative heatmaps that visualize the regions driving each description. Decoding uses beam search (beam size $=4$), length penalty $=1.1$, $no\_repeat\_ngram\_size$ $=3$, and max length $=128$. The proposed design yields accurate, clinically phrased captions and transparent regional attributions, supporting safe research use with a human in the loop.