CMI-MTL: Cross-Mamba interaction based multi-task learning for medical visual question answering

AI summary

Overview

Research area: Medical visual question answering (Med-VQA) — multimodal vision-and-language learning that answers clinical questions about medical images.

Technical level: Advanced (assumes familiarity with Mamba/state-space models, Transformer encoders, contrastive learning, and multi-task learning).

Scope: This paper introduces CMI-MTL, a cross-Mamba interaction framework with three modules (FVTA, CIFR, FFAE) evaluated on the SLAKE, VQA-RAD, and OVQA datasets, with ablation, weight-factor, interpretability, and efficiency analyses.

What This Paper Is About

Med-VQA systems must answer clinical questions about images such as X-rays, CT, MRI, and ultrasound images, but current self-attention methods struggle to align vision and language, especially for long, complex questions. Most existing methods also treat every question as a classification problem over a predefined answer set, which wastes the rich semantics of free-form, open-ended answers. The paper's goal is to build a framework that both improves cross-modal alignment and explicitly exploits open-ended questions as auxiliary supervision to boost overall performance.

Key Contributions

1. A reframed problem statement: Beyond general Med-VQA limitations, the authors target the specific problem of using open-ended questions as auxiliary information to enhance overall classification performance.
2. A new framework: CMI-MTL integrates cross-Mamba interaction inside a multi-task learning architecture, combining fine-grained visual-text feature alignment (FVTA), cross-modal interleaved feature representation (CIFR), and free-form answer enhanced multi-task learning (FFAE).
3. Empirical validation across three datasets: Experiments on SLAKE, VQA-RAD, and OVQA, plus ablation studies on each module and an analysis of the weight factors α and β.
4. Interpretability and efficiency analysis: Grad-CAM visualizations assess whether the model attends to question-relevant image regions, and a comparison against a Transformer-based baseline reports parameter count, FLOPs, and memory consumption.

Main Findings

• SLAKE (English subset): CMI-MTL reaches 83.97% overall accuracy (82.17% open-ended, 86.77% closed-ended), above the second-best reported method, M3AE, at 83.47% overall (80.78% open-ended, 87.65% closed-ended).
• VQA-RAD: CMI-MTL reaches 77.13% overall (69.54% open-ended, 81.37% closed-ended), surpassing M3AE's 76.72% and described by the authors as "closely approaching LaPA's performance, which leads by only 1%." LaPA is reported at 78.15% overall (67.87% open-ended, 84.92% closed-ended).
• OVQA: CMI-MTL reaches 78.52% overall (71.25% open-ended, 83.07% closed-ended), outperforming M3AE's 75.32% and LaPA's 72.51%.
• Best open-ended accuracy on all three datasets: The authors attribute this to the auxiliary FFAE module.
• Ablation — CMM module: Adding the cross-modal Mamba block alone raises accuracy to 83.24% on SLAKE, 73.88% on VQA-RAD, and 77.48% on OVQA, which the authors cite as evidence it captures sequential relationships in image-text pairs.
• Ablation — AHead in FFAE: Adding the auxiliary head improves open-ended accuracy by 1.03%, 5.95%, and 7.08% over the basic model on SLAKE, VQA-RAD, and OVQA respectively, supporting the hypothesis that open-ended corpora help Med-VQA learning.
• Weight factors matter: Performance declines significantly when α exceeds 0.2 and β exceeds 0.3; the authors set α = 0.2 and β = 0.3, explaining that larger values let the auxiliary head and contrastive loss disrupt the main branch's stable feature representation.
• Interpretability: Grad-CAM heatmaps show the model focusing on question-relevant regions for closed-ended questions and locating diseased regions for open-ended ones, including long-sequence answers where it localizes a subtle hand fracture. The authors also report failure modes: scattered attention on the "free air" feature in a chest X-ray with right subdiaphragmatic artifacts, and difficulty finely distinguishing tiny cerebral infarcts and atypical hand fractures.
• Efficiency: Against a Transformer reference (M3AE: 345M parameters, 81.7G FLOPs, 21 GB memory), the Mamba-based CMI-MTL uses 245M parameters, 74.2G FLOPs, and 19 GB memory. The authors attribute this to Mamba eliminating redundant parameters in the QKV projection layers of multi-head attention.
• Rare-disease coverage: The authors note that the tested data includes clinically recognized rare or difficult conditions such as posterior reversible encephalopathy syndrome (PRES), right aortic arch, pineal region mass, rare lesions in the cerebellopontine angle, and adult intussusception.

Methodology in Plain English

The system takes a medical image and a question and produces an answer. Three pieces work together.

First, the FVTA module uses a ViT image encoder and a RoBERTa text encoder, then a question-aware Q-Former ("QQ-Former") that lets a set of learnable query embeddings pull out the image regions most relevant to the question. Cross-modal contrastive learning then pulls matching image-text pairs together and pushes mismatched ones apart, measured with cosine similarity and a temperature parameter.

Second, the CIFR module addresses a weakness of Mamba: its sequential scanning is efficient but not naturally suited to cross-modal correspondence. The authors build cross-modal Mamba (CMM) blocks that interleave features from the two modalities — producing a pair of sequences — and pass each through Mamba with element-wise multiplication, fusion, and a residual sum. Two such blocks are used, and their outputs are concatenated into a fused feature vector.

Third, the FFAE module splits prediction into two heads. A classification head makes the standard prediction for both closed- and open-ended questions using binary cross-entropy. An auxiliary head, a pre-trained T5 decoder, generates free-form answers only for open-ended questions, filtered out of each batch, using a learnable attention mask and a mask-guided cross-entropy loss. The overall objective sums the classification loss, the contrastive loss weighted by α, and the auxiliary loss weighted by β.

Training details: PyTorch on a single NVIDIA GeForce RTX 4090 GPU, AdamW optimizer with an initial learning rate of 5×10⁻⁶, 50 epochs, batch size 8, images resized to 384×384, and a learnable query vector dimension of 32.

Why This Matters

Impact on research. The paper argues that treating Med-VQA purely as classification over a predefined answer set limits adaptability to free-form answers, and demonstrates that an auxiliary generation branch can improve a classification-oriented model. It also positions Mamba-based cross-modal fusion as a more efficient alternative to Transformer attention for this task, and offers Grad-CAM evidence about what the model attends to.

Real-world applications:

• Clinical decision support, where a system answers clinician questions about a patient's imaging study.
• Telemedicine, where specialist image interpretation may not be immediately available.
• Orthopedic assessment, given the OVQA dataset's focus on orthopedics.
• Radiology training and reference, since the datasets span X-rays, CT, MRI, and ultrasound across multiple organ types.

Industry relevance. The efficiency comparison (245M vs 345M parameters, 74.2G vs 81.7G FLOPs, 19 GB vs 21 GB memory) is relevant to deployment cost and latency. The authors also state the cross-modal fusion approach could spark discussion beyond medical fields, including graphics and image science, where combining visual data with textual semantic context may improve scene and object-relationship understanding.

Future Directions

• Extending the approach to larger datasets in the Med-VQA domain, which the authors name explicitly as future work.
• Conducting human-machine comparison and collaboration studies, described as a key focus for subsequent research.
• Expanding the sample size for multi-center validation and incorporating evaluation results from doctors at hospitals of different levels to analyze applicability across clinical environments.
• Exploring human-machine collaborative diagnosis models and assessing whether model assistance improves doctors' diagnostic performance.

The authors state that the initial phase focused on architecture optimization and public-benchmark verification and did not include concurrent evaluation data from actual medical practitioners.

Target Audience

Researchers and graduate students working on medical multimodal learning, vision-language models, and state-space architectures; practitioners interested in multi-task learning designs that add generation as auxiliary supervision for classification; and clinical AI developers weighing accuracy against parameter count, FLOPs, and memory for deployment.

Authors’ abstract

Medical visual question answering (Med-VQA) is a crucial multimodal task in clinical decision support and telemedicine. Recent self-attention based methods struggle to effectively handle cross-modal semantic alignments between vision and language. Moreover, classification-based methods rely on predefined answer sets. Treating this task as a simple classification problem may make it unable to adapt to the diversity of free-form answers and overlook the detailed semantic information of free-form answers. In order to tackle these challenges, we introduce a Cross-Mamba Interaction based Multi-Task Learning (CMI-MTL) framework that learns cross-modal feature representations from images and texts. CMI-MTL comprises three key modules: fine-grained visual-text feature alignment (FVTA), cross-modal interleaved feature representation (CIFR), and free-form answer-enhanced multi-task learning (FFAE). FVTA extracts the most relevant regions in image-text pairs through fine-grained visual-text feature alignment. CIFR captures cross-modal sequential interactions via cross-modal interleaved feature representation. FFAE leverages auxiliary knowledge from open-ended questions through free-form answer-enhanced multi-task learning, improving the model's capability for open-ended Med-VQA. Experimental results show that CMI-MTL outperforms the existing state-of-the-art methods on three Med-VQA datasets: VQA-RAD, SLAKE, and OVQA. Furthermore, we conduct more interpretability experiments to prove the effectiveness. The code is publicly available at https://github.com/BioMedIA-repo/CMI-MTL.