DiA-gnostic VLVAE: Disentangled Alignment-Constrained Vision Language Variational AutoEncoder for Robust Radiology Reporting with Missing Modalities

AI summary

Overview

Research area: Medical computer vision and multimodal vision-language generation, specifically radiology report generation (RRG). Technical level: Intermediate. The paper assumes familiarity with variational autoencoders, mixture-of-experts models, and transformer decoders, but its central idea (separating "shared" from "modality-specific" information so the model still works when inputs are missing) is explained in conceptual terms. Scope: The paper proposes and evaluates a single framework, DiA, for generating chest X-ray radiology reports from images plus clinical context, comparing it against prior image-only, knowledge-guided, and context-aware models on two public datasets.

What This Paper Is About

Automated radiology report generation must combine a medical image with clinical context (demographics, symptoms, prior history), but real hospital data is messy: the clinical context is often missing, and models tend to entangle modality-specific information with shared information, producing unfaithful "hallucinated" findings. The authors build a framework that explicitly separates vision-specific, language-specific, and shared latent representations, then uses that cleanly separated structure to generate reports that stay clinically accurate even when the context is absent. The goal is a compact, efficient system that matches or beats much larger LLM- and knowledge-graph-based alternatives without relying on templates.

Key Contributions

1. A tri-factor latent decomposition for vision-language data. DiA splits the latent space into a vision-specific latent Z_v, a language-specific latent Z_l, and a shared cross-modal latent Z_s, each with its own encoder, and regularizes them so they stay statistically distinct.
2. A Mixture-of-Experts (MoE) posterior for the shared latent. Instead of a Product-of-Experts (PoE) formulation, the shared encoder combines modality-specific expert posteriors with learned mixture weights, and is regularized with Jensen-Shannon Divergence (JSD) rather than KL divergence to avoid component collapse.
3. A Disentangled Alignment Constraint. Two combined terms (orthogonality and contrastive alignment, using the InfoNCE loss) enforce uncorrelated latent subspaces while keeping the shared latent semantically predictive of each modality.
4. A modality-resilient inference scheme with a compact LLaMA-X decoder. A "null" token lets the MoE router automatically down-weight an unavailable modality (pi_L ≈ 0), reducing inference to a single-modality posterior without imputation or architectural changes, while the lightweight decoder generates reports efficiently.

Main Findings

• Report generation quality on IU X-Ray: DiA achieves BLEU@4 of 0.266, ROUGE-L of 0.516, BLEU@1 of 0.616, and F1 of 0.298. The paper reports this as surpassing the best knowledge-guided model, XProNet, by 0.067 in BLEU@4 (XProNet: 0.199), and outperforming the best context-aware model, PromptMRG, by 0.087 in F1 (PromptMRG: 0.211).
• Report generation quality on MIMIC-CXR: DiA achieves BLEU@4 of 0.134, ROUGE-L of 0.369, BLEU@1 of 0.415, and F1 of 0.497. The authors note SEI holds a marginal lead in BLEU@4 (0.135 vs. 0.134) while DiA's ROUGE-L is higher, and that DiA's F1 of 0.497 nearly matches the top performer EKAGen (0.499).
• Ablation, with context available: Adding the VL-MoE-VAE to the baseline raises the MIMIC-CXR F1 by 0.105 and IU X-Ray BLEU@4 by 0.057. Adding the Disentangled Alignment constraint pushes the full model to the highest scores on all metrics (MIMIC-CXR F1 0.621, ROUGE-L 0.399; IU X-Ray BLEU@4 0.357, ROUGE-L 0.624).
• Ablation, with context missing: The baseline's MIMIC-CXR F1 drops by 0.227 under image-only input, whereas DiA drops by only 0.183, outperforming the baseline by +0.219 in that setting. On IU X-Ray, DiA's BLEU@4 remains more than 2x the baseline's (0.198 vs. 0.079). Start-to-end gains on MIMIC-CXR are 0.175 in F1 with context and 0.219 without.
• Encoder comparison (MIMIC-CXR): A pre-trained RAD-DINO + CXR-BERT setup scored lower (BLEU@4 0.121, F1 0.441) at a higher cost of 81.1 GFLOPs, versus DiA's end-to-end learned encoder (BLEU@4 0.134, F1 0.497) at 51.1 GFLOPs.
• Decoder comparison (MIMIC-CXR): LLaMA-X beat a standard Transformer (BLEU@4 0.126, F1 0.479 at 80.6 GFLOPs) and GPT-2 (BLEU@4 0.116, F1 0.419 at 86.4 GFLOPs).
• Computational cost: DiA has 589.7M parameters and 51.14 GFLOPs. Training takes 2.8 hours on IU X-Ray with 0.15-second inference, and 79.8 hours on MIMIC-CXR with 0.18-second inference, on an NVIDIA A40 GPU.
• Qualitative inspection: Attention heatmaps show DiA focusing on key clinical regions in chest X-rays both with and without clinical context provided, with generated reports showing strong alignment to ground-truth reports.

Methodology in Plain English

The pipeline starts by turning raw inputs into features: a pre-trained EfficientNetB0 with Guided Context Attention extracts image features, and a Transformer encoder processes the tokenized clinical context. A "Modality Abstractor" then lets each modality query the other through bidirectional cross-attention, so the model learns which image regions correspond to which clinical terms.

The core is a Vision-Language Mixture-of-Experts VAE. Rather than one jumbled latent representation, the model learns three: one that must contain everything needed to reconstruct the image, one that must contain everything needed to reconstruct the text, and one shared latent that captures what the two have in common. The shared latent is inferred by averaging over per-modality "experts" with learned weights, which is what makes the system robust when one modality is absent — a "null" token makes the missing modality's weight collapse toward zero.

Training adds two regularizers to the standard evidence lower bound (ELBO): an orthogonality term that whitens each latent subspace and penalizes correlations between them (approximating statistical independence in the spirit of Independent Component Analysis), and a contrastive alignment term (InfoNCE) that keeps the shared latent closely tied to both modality-specific latents. The total objective sums the ELBO, the cross-entropy generation loss, and these two regularizers weighted by lambda_1 and lambda_2 (both set to 0.3, tuned over 0.1 to 0.5). Reports are produced by a compact LLaMA-X decoder using Grouped Query Attention, Rotary Positional Encodings, SwiGLU feed-forward layers, and RMS pre-normalization.

Why This Matters

Research impact: The paper argues that explicit disentanglement, rather than more parameters or retrieval, is what makes multimodal medical models reliable under incomplete inputs. The probabilistic "null token" trick for a missing modality is a principled alternative to imputation or brittle rule-based fallbacks used by prior context-aware and retrieval-augmented systems.

Real-world applications:

• Draft radiology reports for chest X-rays in settings where the clinical order/indication field is blank (a common real-world gap).
• Prioritize or pre-screen studies in radiology worklists to help manage reading backlogs.
• Support hospitals whose imaging systems are not integrated with EHR context data.
• Provide training or second-read support in resource-constrained clinics that cannot run large LLMs.

Industry relevance: The paper explicitly positions DiA against resource-heavy LLMs and template-based prompt models, claiming better adaptability and computational efficiency. The reported 2.8-hour training time on IU X-Ray (7,470 images) and 51.1 GFLOPs footprint suggest a system that could realistically be deployed outside well-funded research environments.

Future Directions

• Validate beyond chest X-ray reporting, since the current evaluation is limited to IU X-Ray and MIMIC-CXR.
• Extend the tri-factor decomposition to additional modality types; the paper cites handling missing data in highly heterogeneous image-and-text settings as an open challenge.
• Test whether the missing-modality robustness generalizes to other missingness patterns and rates beyond the approximately 2% (IU X-Ray) and approximately 45% (MIMIC-CXR) missing-context test samples reported.
• Determine how the framework scales to longer reports; the paper uses a maximum report length of 50 words and notes that context-aware methods "struggle with longer reports."
• Conduct clinician-facing evaluation; the paper reports no human reader study, only automated NLG metrics (BLEU, ROUGE-L) and a CheXbert-derived F1 over 14 disease labels.

Target Audience

Researchers and graduate students working on multimodal medical AI, radiology report generation, or missing-modality robustness will find the framework and ablation design most useful. Practitioners evaluating deployable clinical NLP systems will benefit from the efficiency and missing-context results, and readers interested in disentangled representation learning will find the combination of orthogonality, contrastive alignment, and MoE posteriors the most transferable idea. Some familiarity with variational inference is assumed; the appendix derives the ELBO and is truncated in the supplied content.

Authors’ abstract

The integration of medical images with clinical context is essential for generating accurate and clinically interpretable radiology reports. However, current automated methods often rely on resource-heavy Large Language Models (LLMs) or static knowledge graphs and struggle with two fundamental challenges in real-world clinical data: (1) missing modalities, such as incomplete clinical context , and (2) feature entanglement, where mixed modality-specific and shared information leads to suboptimal fusion and clinically unfaithful hallucinated findings. To address these challenges, we propose the DiA-gnostic VLVAE, which achieves robust radiology reporting through Disentangled Alignment. Our framework is designed to be resilient to missing modalities by disentangling shared and modality-specific features using a Mixture-of-Experts (MoE) based Vision-Language Variational Autoencoder (VLVAE). A constrained optimization objective enforces orthogonality and alignment between these latent representations to prevent suboptimal fusion. A compact LLaMA-X decoder then uses these disentangled representations to generate reports efficiently. On the IU X-Ray and MIMIC-CXR datasets, DiA has achieved competetive BLEU@4 scores of 0.266 and 0.134, respectively. Experimental results show that the proposed method significantly outperforms state-of-the-art models.