Bridging Modalities via Progressive Re-alignment for Multimodal Test-Time Adaptation

AI summary

Overview

Research area: Multimodal test-time adaptation (MMTTA) — adapting a pre-trained multimodal model to a new target domain using only unlabeled test data, within the broader fields of machine learning, domain adaptation, and parameter-efficient fine-tuning.

Technical level: Intermediate to Advanced. The paper assumes familiarity with test-time adaptation (TTA), prompt tuning, contrastive learning, and distribution-matching objectives.

Scope: The paper proposes BriMPR, a two-stage framework that re-aligns shifted modalities by calibrating each unimodal feature distribution with prompts and then refining alignment through masked cross-modal recombination and instance-wise contrastive learning, evaluated on corruption and real-world domain-shift benchmarks.

What This Paper Is About

Multimodal models must handle test data whose distribution differs from training data, and different modalities (such as audio and video) usually shift by different amounts. Existing test-time adaptation methods, which were designed for single modalities, do not correct the shallow unimodal features, so the fused multimodal representation becomes entangled and loses discriminability. BriMPR's goal is to tackle this combined problem — unimodal feature shift together with cross-modal semantic misalignment — using a divide-and-conquer strategy that adapts each modality separately and then reconnects them.

Key Contributions

1. A novel MMTTA framework that mitigates modality-wise distribution shifts in a divide-and-conquer manner, re-aligning modalities progressively rather than treating the multimodal model as a single unit.
2. Use of prompt tuning's function approximation ability to calibrate each unimodal global feature distribution toward its source distribution, with an objective that deliberately retains only the diagonal of the covariance matrix (supported by Theorem 1, which shows the mean squared error of the diagonal-vector estimate is O(d/n) versus O(d²/n) for the full covariance matrix).
3. A cross-modal masked embedding recombination (CMER) strategy that supplies calibrated pseudo-labels for combinations of masked and complete modalities, forcing a corrupted modality to derive information from the multimodal context.
4. Extensive experiments on corruption-based and real-world domain-shift MMTTA benchmarks demonstrating superiority over Source, Tent, EATA, SAR, DeYO, FOA, READ, ABPEM, and SuMi.

Main Findings

• Unimodal shift, dominant modality corrupted: On Kinetics50-C (video corrupted, severity level 5), BriMPR raises average accuracy from the source model's 60.5% to 65.9%. On VGGSound-C (audio corrupted, severity level 5) it raises accuracy from 25.0% to 36.5%.
• Unimodal shift, full corruption grids: On Kinetics50-C with audio corruption, BriMPR averages 72.0% versus 69.4% for Source and 71.9% for SuMi (the strongest competing MMTTA method on that grid). On VGGSound-C with video corruption, BriMPR averages 57.7% versus 56.2% for Source and 57.3% for SuMi.
• Multimodal shift (both modalities corrupted, severity 5): BriMPR reaches 40.9% average on Kinetics50-C versus 31.8% for Source, and 20.7% on VGGSound-C versus 9.5% for Source. The paper notes most competing methods drop sharply under this setting.
• Real-world shifts: On MOSI → SIMS, BriMPR obtains 58.2% ACC and 57.6% F1 versus 46.0% / 45.6% for Source; the paper states that only BriMPR exceeds random guess (>50%) on this task. On SIMS → MOSI it obtains 59.9% ACC and 74.9% F1 versus 59.0% / 73.6% for Source.
• Alignment objective matters: Replacing the PMGFA objective with KL-divergence collapses performance (for example, 9.1% average on VGGSound-C multimodal shift versus 20.1% for PMGFA); moment matching in squared form yields 12.1%, and in non-squared form 16.4%.
• Each component adds gains: On Kinetics50-C multimodal shift, PMGFA alone reaches 40.5%, adding IICL reaches 40.7%, and adding CMER reaches 40.9%; on VGGSound-C multimodal shift the progression is 20.1% → 20.2% → 20.7%.
• Weight assignment in CMER is not arbitrary: Swapping the weights λᵃ and λᵛ, so lower weight goes to the augmentation with the milder distribution shift, degrades results — for example, 70.0% versus 72.0% on Kinetics50-C audio and 32.1% versus 36.5% on VGGSound-C audio.

Methodology in Plain English

BriMPR starts from a pre-trained multimodal model that is split into two modality-specific encoders, a joint module, and a classifier; only newly added prompts are trained and everything else stays frozen.

Stage 1 — align each modality on its own. Because the source feature space already has the modalities well aligned with each other, the authors treat MMTTA as several separate unimodal adaptation problems. Each modality's source and target features are modeled as multivariate Gaussians. Rather than matching full covariance matrices — which are expensive and noisy to estimate in high dimensions — they keep only the diagonal (per-dimension variances), which Theorem 1 shows reduces estimation error. For each transformer layer, learnable prompt tokens are inserted into the modality-specific encoder, and the mean and standard deviation of the layer's pooled output on the current test batch are pulled toward source statistics that were pre-computed once before testing.

Stage 2 — refine alignment through interaction. In cross-modal masked embedding recombination, a portion of one modality's patches (50%) is masked, encoded, and recombined with the other modality's complete embeddings. Because Stage 1 already improved reliability, predictions on the complete multimodal pair serve as pseudo-labels for these augmented combinations, with temperature scaling that adapts based on measured distribution discrepancy. The loss weights automatically favor whichever augmented branch has the milder shift. Separately, inter-modal instance-wise contrastive learning treats different unimodal views of the same instance as positive pairs and other instances as negatives, keeping alignment tight at the instance level. The three losses are summed and minimized to update the prompts.

Experimental setups: Kinetics50-C and VGGSound-C use video and audio with 15 and 6 corruption types respectively across 5 severity levels, with pre-trained CAV-MAE as the source model; CMU-MOSI and CH-SIMS use text, video, and audio with stacked Transformer blocks as the backbone. Training uses Adam, a learning rate of 1e-4, batch size 64, 10 prompts per layer, mask ratio 0.5, τ₀ = 0.2, D₀ = 5, and τ = 0.07 (unimodal corruption) or 0.25 (multimodal corruption), with 3 random seeds on RTX-3090 GPUs.

Why This Matters

Impact on research. The paper reframes multimodal test-time adaptation as a two-level failure — shallow unimodal feature shift plus high-level cross-modal semantic misalignment — and shows that a divide-and-conquer treatment outperforms methods that only reweight modalities in the fusion module. It also extends prompt tuning beyond vision-language models to any modality, and contributes a practical, theoretically supported simplification of distribution matching (diagonal covariance only). The visualizations in Figure 1 contrast BriMPR's t-SNE behavior with EATA's and READ's on Kinetics50-C, giving a qualitative account of why unimodal correction propagates to better fused features.

Real-world applications (grounded in the paper's datasets and domains):

• Audio-visual event classification in video platforms or surveillance, where camera or microphone conditions at deployment differ from training conditions.
• Multimodal sentiment analysis and opinion mining, the domain of CMU-MOSI and CH-SIMS, where text, video, and audio streams come from heterogeneous sources.
• Sensor-fusion perception for intelligent transportation and autonomous systems, which the authors' institutional affiliations point to, since different sensors degrade independently.
• Any deployed multimodal system where the source data cannot be stored or revisited and adaptation must happen online from unlabeled test streams.

Industry relevance. Because only lightweight prompts are optimized while the rest of the model is frozen, the approach is parameter-efficient and suited to settings where retraining or accessing source data is impractical. The reported robustness under combined audio-plus-video corruption on VGGSound-C (20.7% versus 9.5% for the source model) addresses the practically common case where degradation is not confined to one channel.

Future Directions

• Extending the framework beyond the two-modality formalization and three-modality experiments, to settings with more modalities or missing modalities at test time.
• Addressing open-set conditions, which the related-work section notes is pursued by AEO (introducing unseen classes); BriMPR is evaluated only under closed-set accuracy and F1.
• Investigating the interaction between the sequence of prompt layers and the distribution discrepancy measurement, since Stage 2 relies on the discrepancy value Disc to set pseudo-label temperature and loss weights.
• Testing whether the diagonal-covariance alignment transfers to extremely high-dimensional or non-Gaussian feature distributions, where the paper's Gaussian modeling assumption may be strained.
• Reporting efficiency metrics such as latency, memory, and energy, which are not reported in the paper, to assess online deployment costs.

Target Audience

Researchers and graduate students working on test-time adaptation, domain adaptation, multimodal learning, or parameter-efficient fine-tuning; practitioners deploying multimodal models in video understanding, sentiment analysis, or sensor-fusion systems who need online adaptation without labeled target data or source-data access.

Authors’ abstract

Test-time adaptation (TTA) enables online model adaptation using only unlabeled test data, aiming to bridge the gap between source and target distributions. However, in multimodal scenarios, varying degrees of distribution shift across different modalities give rise to a complex coupling effect of unimodal shallow feature shift and cross-modal high-level semantic misalignment, posing a major obstacle to extending existing TTA methods to the multimodal field. To address this challenge, we propose a novel multimodal test-time adaptation (MMTTA) framework, termed as Bridging Modalities via Progressive Re-alignment (BriMPR). BriMPR, consisting of two progressively enhanced modules, tackles the coupling effect with a divide-and-conquer strategy. Specifically, we first decompose MMTTA into multiple unimodal feature alignment sub-problems. By leveraging the strong function approximation ability of prompt tuning, we calibrate the unimodal global feature distributions to their respective source distributions, so as to achieve the initial semantic re-alignment across modalities. Subsequently, we assign the credible pseudo-labels to combinations of masked and complete modalities, and introduce inter-modal instance-wise contrastive learning to further enhance the information interaction among modalities and refine the alignment. Extensive experiments on MMTTA tasks, including both corruption-based and real-world domain shift benchmarks, demonstrate the superiority of our method. Our source code is available at https://github.com/Luchicken/BriMPR.