Calibrated Multimodal Representation Learning with Missing Modalities

AI summary

Overview

Research area: Multimodal representation learning (arXiv listing category: Computer Vision; keywords: Machine Learning, ICML).

Technical level: Advanced. The paper develops a generative latent-variable model, an evidence lower bound, closed-form posterior and parameter updates, and three theoretical results (a theorem, a corollary, and a monotonicity corollary) built on probabilistic PCA and singular value decomposition.

Scope: The paper introduces CalMRL, a framework that calibrates multimodal alignment when some modalities are missing from an instance, by imputing the missing modalities at the representation level and jointly aligning observed and imputed representations.

What This Paper Is About

Recent multimodal representation learning methods aim to align several modalities into a unified latent space, but they generally require every modality to be present for a given instance, which does not match the many real datasets that contain only pairs (for example vision-text or audio-text). The authors show theoretically that when modalities are missing, the remaining ones are pulled toward a "local anchor" that deviates from the optimal anchor that would exist with all modalities present, a phenomenon they call anchor shift. CalMRL addresses this by modeling and imputing the missing modalities in latent space so the alignment is calibrated back toward the complete-modality anchor.

Key Contributions

1. The paper introduces Calibrated Multimodal Representation Learning (CalMRL) to address the missing-modality problem in multimodal representation learning, and analyzes it through an anchor shift perspective, including a theorem giving lower and upper bounds on that shift.

2. It proposes a generative model in which each modality's representation is conditioned on shared latents plus its own modality-specific uniqueness, and imputes missing modalities in closed form so that the anchor offset is reduced. The imputation is refined by iterating posterior inference and parameter optimization, both of which have closed-form solutions.

3. It provides theoretical grounding for the calibration (Corollary 3, showing under what condition calibration produces less anchor shift than missing-modality training) and for the optimization (Corollary 4, showing monotonic non-decreasing log-likelihood), then trains the encoders on both observed and imputed modalities.

4. It reports extensive experiments on ten evaluation datasets covering retrieval and classification, comparing against ImageBind, InternVideo-L, LanguageBind, VAST, GRAM, TRIANGLE, and PMRL, plus empirical analyses of imputation error, anchor shift before and after calibration, and a comparison against a full-modality "ideal" setting. Code is released at https://github.com/Xiaohao-Liu/CalMRL.

Main Findings

• Missing modalities inevitably perturb the anchor. Theorem 1 shows the anchor shift has a strictly positive lower bound whenever the missing modalities contribute non-zero alignment (η > 0) or the observed data fails to capture the full leading singular value (σ₁^Ω < σ₁). The upper bound implies the perturbation can be mitigated, but never eliminated, by a large spectral gap and limited energy in the missing modalities.

• Calibration reduces anchor shift under a stated condition. Corollary 3 states that calibration reduces anchor shift if and only if the imputation error ε is below a threshold involving the spectral gap σ₁ − σ₂, the number of missing modalities, σ₁^Ω, and η. The benefit is largest when alignment among modalities is strong and the missing modalities contribute relatively little.

• The bi-step optimization is monotone. Corollary 4 shows that L(θ̂^(t+1)) ≥ L(θ̂^(t)) for the observed-data log-likelihood under the generative model, backing stable convergence of the generative subroutine within the larger alternating-optimization framework.

• CalMRL leads on multimodal retrieval. In Table 1 (Recall@1), CalMRL has the best average of 54.2, ahead of PMRL↑ (53.8), VAST↑ (53.3), GRAM↑ (52.9), TRIANGLE↑ (51.6). Its increments over VAST include MSR-VTT T→V 61.1 (+10.6) and V→T 61.1 (+12.1); DiDeMo 55.4 (+6.8) and 53.7 (+6.8); ActivityNet 57.1 (+5.4) and 53.6 (+4.8); VATEX 81.3 (+5.4) and 79.2 (+4.4); AudioCaps T→A 50.1 (+16.4) and A→T 51.0 (+18.8); Clotho T→A 23.8 (+11.4) and A→T 22.4 (+9.4).

• CalMRL also leads on multimodal classification. In Table 2 (Accuracy), CalMRL reaches 26.09 (+0.85) on VGGSound, 78.91 (+1.78) on UCF101, 16.01 (+0.72) on AudioSet, and 59.75 (+1.25) on ESC50, for an average of 45.19 (+1.15). For comparison, ImageBind has an average of 42.08.

• Continual training on missing-modality data helps generally, and learning all modalities together helps more. The paper reports that models trained with all modalities available simultaneously outperform methods that extend pairwise contrastive learning (ImageBind, InternVideo-L, LanguageBind), and that training on missing-modality datasets improves virtually all models, particularly on in-domain datasets such as MSR-VTT and AudioCaps.

• CalMRL also leads when trained on a single modality-pair dataset. In Table 3, CalMRL trained with only audio-text data (↑AT) has the best average of 51.7, and with only video-text data (↑VT) the best average of 50.4.

• Calibration is measured empirically. Figure 3 reports MSEs between real and imputed representations (with Random as a reference, and "→" marking the imputation direction), Figure 4 compares anchor shift Δ before and after calibration, and Figure 5 compares performance across missing, calibrated, and full ("ideal") modalities for models trained on MSR-VTT.

• Not reported in the provided content. The text is truncated partway through Section 4.2's second conclusion, so the full answers to RQ2 ("What is the contribution of CalMRL to different missing modalities?") and RQ3 (training stability and whether the distribution of multimodal representations is maintained) include numerical results that are not present in the supplied content, as are the figures' specific values, retrieval results at top-k of {5, 10} in Appendix D, and dataset statistics in Appendix C.1.

Methodology in Plain English

The authors start from the observation that existing methods align all modalities toward a shared center, or anchor, but that this center moves when a modality is absent. They formalize the gap between the correct anchor and the observed-modality anchor, and prove it cannot be driven to zero.

Their fix is to fill in what is missing at the representation level rather than at the raw data level. Each modality's representation is modeled as coming from two parts: a shared latent code describing the common instance, plus a modality-specific offset and noise. Because the shared code is common, information from observed modalities carries over to the unobserved ones.

Because these parameters are tied together through the shared latent, a standard closed-form solution does not apply, so the authors use a two-step (bi-step) procedure. In the first step, holding the generative parameters fixed, they compute the posterior distribution of the shared latent in closed form — a Gaussian whose mean and covariance depend only on the observed modalities, so missing modalities do not enter the update directly. In the second step, given that posterior, they derive closed-form updates for each generative parameter (the modality-specific mean, the loading matrix, and the variance). Iterating the two steps refines the model, and the imputed representation of a missing modality is then computed simply as its loading matrix times the shared-latent mean, plus its own mean.

With observed and imputed representations concatenated, the encoders are trained with a representation loss that maximizes the largest singular value of the resulting matrix while regularizing with the corresponding eigenvectors, plus an instance-matching term weighted by α = 0.1. Models are first warmed up on VAST-150K with complete modalities, then continually trained on MSRVTT (vision-text) and AudioCaps (audio-text) to introduce missing modalities. Evaluation uses Recall@1 for retrieval and Accuracy for classification, with only test splits used.

Why This Matters

The work targets a mismatch that is easy to overlook: methods designed for fully aligned multimodal data are evaluated on benchmarks whose training corpora are frequently only paired. By quantifying the anchor shift and showing how to compensate for it, the paper gives both a diagnosis and a repair for missing-modality training, and it shows that models can absorb datasets they were previously unable to absorb at all.

Real-world applications implied by the tasks and datasets used:

• Video-text retrieval and captioning, where video data may lack captions, subtitles, or audio (MSR-VTT, DiDeMo, ActivityNet, VATEX).
• Audio-text retrieval and audio captioning, where recordings often have no matched textual description (AudioCaps, Clotho).
• Multimodal classification of video with sound, where the available modality set varies from clip to clip (VGGSound, UCF101, AudioSet, ESC50).
• Continual training pipelines that mix paired datasets from different modality combinations rather than requiring one fully annotated corpus.

Industry relevance: many practical systems collect vision-text, audio-text, or video-audio pairs independently. Methods like ImageBind and LanguageBind handle this by fixing one modality as the anchor, which the authors argue bottlenecks mutual improvement and depends heavily on that fixed anchor's quality. CalMRL offers an alternative that keeps encoders trainable and does not require a single designated anchor modality, which matters for teams assembling training data from heterogeneous sources.

Future Directions

• Whether anchor shift can ever be fully removed. The authors' own upper bound indicates the perturbation can be mitigated but never eliminated; determining how close calibration can get to the full-modality anchor is left open.

• Extending the alignment to arbitrary modality sets. The paper notes that the alignment objective it builds on is limited by predefined modalities and is hard to extend to arbitrary modalities — a limitation CalMRL does not claim to solve.

• Closing the gap to the full-modality "ideal" case. Figure 5 compares missing, calibrated, and full modalities; quantifying how much of the remaining gap is attributable to imputation error versus the alignment objective itself is a natural next question.

• Characterizing behavior across different missing-modality patterns. The paper frames the problem at the instance level, viewing a dataset as a set of instances missing the same modalities, and RQ2 asks about the contribution of CalMRL to different missing modalities; the full results are not in the provided content.

Target Audience

Researchers and graduate students working on multimodal representation learning, cross-modal retrieval, and multimodal classification, especially those who need to train on datasets that are only partially aligned. It is also relevant to practitioners building multimodal retrieval or classification systems from heterogeneous data sources, and to readers interested in the theory of latent-variable models and spectral alignment objectives. Readers need comfort with probabilistic PCA, variational lower bounds, and singular value decomposition to follow the derivations; the core idea of imputing missing modalities to correct a shifted alignment center is accessible without that background.

Authors’ abstract

Multimodal representation learning harmonizes distinct modalities by aligning them into a unified latent space. Recent research generalizes traditional cross-modal alignment to produce enhanced multimodal synergy but requires all modalities to be present for a common instance, making it challenging to utilize prevalent datasets with missing modalities. We provide theoretical insights into this issue from an anchor shift perspective. Observed modalities are aligned with a local anchor that deviates from the optimal one when all modalities are present, resulting in an inevitable shift. To address this, we propose CalMRL to calibrate incomplete alignments caused by missing modalities. CalMRL leverages the priors and the inherent connections among modalities to model the imputation for the missing ones at the representation level. To resolve the optimization dilemma, we employ a bi-step learning method with the closed-form solution of the posterior distribution of shared latents. We validate its mitigation of anchor shift and convergence with theoretical guidance. By equipping the calibrated alignment with the existing advanced method, we offer new flexibility to absorb data with missing modalities, which is originally unattainable. Extensive experiments demonstrate the superiority of CalMRL. The code is released at https://github.com/Xiaohao-Liu/CalMRL.