Closing the Modality Gap Aligns Group-Wise Semantics

AI summary

Overview

Research area: Multimodal representation learning, specifically contrastive alignment of image, text, audio, and video encoders in a shared latent space (the CLIP family of models).

Technical level: Advanced. The paper combines formal derivations of contrastive loss dynamics and gradient behavior with empirical evaluation across four datasets, and readers need familiarity with InfoNCE, cosine similarity spaces, and clustering metrics.

Scope: The paper argues that the "modality gap" is benign for instance-wise tasks like retrieval but harmful for group-wise tasks like clustering, and introduces two loss terms that close the gap while improving clustering performance.

What This Paper Is About

Multimodal models such as CLIP are trained to place matching pairs (for example, an image and its caption) near each other in a shared latent space, but in practice the embeddings of each modality end up clustered in separate regions — a phenomenon known as the modality gap, measured as the Euclidean distance between the centroids of each modality's representations. Whether this gap matters is disputed: prior work has reported both that closing it helps retrieval and that a larger gap correlates mildly with better downstream performance. This paper reframes the debate by separating retrieval (which depends only on relative rankings and is therefore insensitive to the gap) from clustering (which depends on absolute distances and is therefore damaged by it), then proposes an objective that drives the gap to nearly zero while improving clustering without sacrificing retrieval.

Key Contributions

1. A task-dependent diagnosis of the modality gap. The authors provide theoretical and empirical evidence that the gap is largely irrelevant to instance-wise tasks such as cross-modal retrieval, because InfoNCE only optimizes relative similarity orderings, but directly inflates within-cluster scatter for group-wise tasks such as clustering.

2. A new two-part objective. They introduce the Align True Pairs loss (L_ATP), which pulls matching pairs together across modalities relative to an anchor modality, and the Centroid Uniformity loss (L_CU), which spreads the averaged per-sample centroids over the latent space. Combined as L_gap = L_ATP + L_CU and added to the standard contrastive loss, this forms L_CL_gap.

3. A derivation of why the gap-free configuration is a global minimum. A controlled simulation with six simulated image–text embedding pairs shows the standard contrastive loss settles near 60°, while the proposed loss adds a term with gradient 2 sin θ that moves the only stationary point to 0°.

4. A multi-modal evaluation. The method is validated on two-modality settings (CIFAR10, MSCOCO) and three-modality settings (AV-MNIST, MSR-VTT), with no architectural changes or post-hoc corrections, and the authors state that the code is available at https://github.com/ispamm/ModGap.

Main Findings

• Closing the gap improves clustering, not retrieval. On MSR-VTT, reducing the gap from the standard CLIP value toward zero improved the clustering metric by +17.5 points in the authors' controlled post-hoc shifting experiment (Figure 1), while retrieval scores barely moved. The paper states that the optimal clustering performance coincides with the zero-gap configuration.

• Gap-free spaces organize by semantics rather than by modality. In MSCOCO nearest-neighbor queries, the CLIP latent space (gap 0.47) returned only same-modality neighbors for a text query, while the proposed method's near-zero-gap space returned neighbors from both image and text modalities. The same behavior held for an image query.

• True-pair cosine similarity rises substantially. On MSCOCO, mean true-pair cosine similarity (Cos TP) increased from 0.34 for CLIP (LT) to 0.77 for the proposed method, and the gap fell from 0.47 to 0.03. On MSR-VTT, Cos TP rose from 0.27 (CLIP LT) to 0.53, with the gap moving from 0.24 to 0.06. The authors note that in comparison configurations the true-pair similarity instead decreases progressively during training.

• Clustering gains across all four datasets. V-Measure: CIFAR10 67.0 (CLIP LT) to 67.9; MSCOCO 12.98 to 23.63; AV-MNIST 77.6 to 82.7; MSR-VTT 23.3 to 32.1. kNN accuracy followed the same direction in every case (81.2 to 82.4; 26.3 to 36.4; 87.0 to 89.2; 52.9 to 58.0).

• Gap values for the proposed method. CIFAR10 0.09, MSCOCO 0.03, AV-MNIST 0.09, MSR-VTT 0.07, compared with 0.86, 0.47, 0.20 and 0.29 for CLIP (LT) respectively.

• Gradient reweighting explains the mechanism. As the gap narrows, the gradient contribution of non-matching pairs increases while that of matching pairs decreases. With a large gap, negatives and positives contribute equally; with a closed gap, the hard negatives dominate optimization, shaping semantic clusters while leaving positive-pair rankings intact.

• Cross-modal retrieval results are mixed. In Table 1, the proposed method improves retrieval on CIFAR10 (82.0 to 82.4 TV R@1) and AV-MNIST (87.1 to 88.7 TV R@1, 84.2 to 89.1 TA R@1), but text-to-video R@1 is lower on MSCOCO (74.6 for CLIP LT vs. 70.3) and MSR-VTT (34.2 vs. 32.8). Text-to-audio R@1 on MSR-VTT rises from 10.3 to 11.8.

• Downstream multimodal retrieval and captioning improve in Table 2. MSCOCO multimodal R@1 rose from 72.5 (CLIP LT) to 77.3, BLEU@1 from 45.8 to 46.1, BLEU@3 from 22.5 to 23.2, and CIDEr from 153.2 to 160.9. On MSR-VTT, R@1 rose from 30.6 to 33.3, BLEU@1 from 26.7 to 26.8, BLEU@3 from 9.5 to 9.6, and CIDEr from 63.6 to 64.4. The NotAGap baseline reported gap values of 0.17 (MSCOCO) and 0.09 (MSR-VTT).

• Per-pair gap variance is very small. The appendix reports distance means of 1.153 (MSCOCO T-I), 1.163 (MSR-VTT T-V) and 1.252 (MSR-VTT T-A), with variances of 0.004, 0.009 and 0.010, and Cos TP means of 0.334, 0.319 and 0.211 with variances of 0.005, 0.012 and 0.016.

Methodology in Plain English

The authors start from a standard multimodal training setup: a batch of samples, each observed through M modalities, encoded into normalized vectors, with the InfoNCE contrastive loss pulling matching pairs together and pushing non-matching pairs apart. They then reason about geometry. Because InfoNCE only requires the true pair's similarity to exceed the best negative's similarity, retrieval succeeds regardless of how far apart the two modality clusters sit — so retrieval should be indifferent to the gap. For clustering, they show algebraically that a constant per-modality offset δ, representing the gap, adds a ||δ||² term to the expected within-class scatter without depending on the class itself. A larger gap therefore inflates every cluster uniformly, hurting homogeneity and completeness even when individual pairs are still correctly ranked.

To fix this, the method adds two losses on top of the contrastive objective. The first, L_ATP, penalizes the squared Euclidean distance between each modality's embedding and the anchor modality's embedding for the same sample, forcing true pairs together. On its own this risks collapsing the space, so the second loss, L_CU, operates on each sample's centroid — the average of all its modality embeddings — and uses a radial basis function term to push different centroids apart, spreading representations across the unit hypersphere while preserving the alignment established by L_ATP. Applying uniformity to the centroids rather than to each modality independently avoids breaking the alignment between semantically matching pairs. The two losses are summed and added to the averaged bidirectional InfoNCE loss to give L_CL_gap.

Evaluation spans four datasets (CIFAR10 with two ResNet50 encoders, AV-MNIST with three modalities, MSCOCO with EVA-CLIP ViT-G and BERT-B, and MSR-VTT with the same encoders plus BEATs for audio) and four tasks (cross-modal retrieval, clustering via k-means and k-NN measured with V-Measure and k-NN accuracy, multimodal retrieval with cross-conditioning using a data-anchor matching loss, and captioning). The truncated content does not report training hyperparameters, compute budget, or the exact temperature values used.

Why This Matters

For research, the paper reframes a long-running disagreement. Earlier work reported that closing the gap may help retrieval (Yaras et al. 2025) while other work found a larger gap mildly positively correlated with downstream performance (Schrodi et al. 2025), leaving the field without a clear mandate. This paper argues the disagreement stems from which task family is measured, and supports that claim with a formal account of why rank-based metrics should be insensitive to a constant offset while cluster-geometry metrics should not. It also extends the discussion beyond the usual image–text pair to three-modality audio–visual–text settings, which prior gap-mitigation methods (fixed temperature, post-hoc translation, shared encoders) had not addressed.

Real-world applications implied by the evaluated tasks:

• Media asset management and content organization. Grouping images, video and audio into semantic clusters (rather than modality-specific piles) is what an archive or editing system needs when a user browses by topic.
• Cross-modal search engines. A shared space where a text query returns both images and audio of the same semantics, rather than only same-modality matches, as demonstrated in the MSCOCO nearest-neighbor experiment.
• Video understanding pipelines. The MSR-VTT trimodal setup (video, audio, text across 20 categories) resembles automatic tagging and chaptering of video libraries.
• Accessibility tooling. Audio description and captioning systems depend on the same encoder–decoder structure evaluated here, where CIDEr rose from 153.2 to 160.9 on MSCOCO.

Industry relevance comes from the method's cost profile: the authors state it requires no architectural changes and no post-hoc corrections, only an added loss term, and it scales to the general n-modal case. That makes it a candidate for retrofitting existing CLIP-style pipelines rather than retraining from scratch.

Future Directions

• Reconciling retrieval trade-offs. The proposed method lowers text-to-video R@1 on MSCOCO (74.6 to 70.3) and MSR-VTT (34.2 to 32.8) in Table 1 while improving multimodal R@1 in Table 2. Understanding when and why instance-wise retrieval degrades, and whether the loss weights can be tuned to avoid it, is an open question.
• Scaling beyond three modalities. The authors state the formulation extends to the general n-modal case, but the truncated content reports experiments only up to three modalities, so behavior with four or more remains untested here.
• Evaluating on additional group-wise tasks. The paper argues that any objective optimizing within-cluster and between-cluster geometry should benefit from gap closure; testing that prediction on tasks beyond clustering (for example, semantic hashing or retrieval-augmented generation over shared spaces) would test the generality of the claim.
• Tuning the balance between L_ATP and L_CU. The authors note that L_ATP alone can collapse the space and that L_CU prevents this; the relative weighting and its sensitivity are not reported in the available content.

Target Audience

Researchers and graduate students working on multimodal representation learning, contrastive objectives, or cross-modal retrieval and clustering. It is also relevant to practitioners deploying CLIP-style encoders who need embeddings that support semantic grouping rather than only pairwise matching, and to readers following the ongoing debate over whether the modality gap should be closed or accepted. The formal derivations assume comfort with contrastive loss gradients and embedding geometry.

Authors’ abstract

In multimodal learning, CLIP has been recognized as the \textit{de facto} method for learning a shared latent space across multiple modalities, placing similar representations close to each other and moving them away from dissimilar ones. Although CLIP-based losses effectively align modalities at the semantic level, the resulting latent spaces often remain only partially shared, revealing a structural mismatch known as the modality gap. While the necessity of addressing this phenomenon remains debated, particularly given its limited impact on instance-wise tasks (e.g., retrieval), we prove that its influence is instead strongly pronounced in group-level tasks (e.g., clustering). To support this claim, we introduce a novel method designed to consistently reduce this discrepancy in two-modal settings, with a straightforward extension to the general $n$-modal case. Through our extensive evaluation, we demonstrate our novel insight: while reducing the gap provides only marginal or inconsistent improvements in traditional instance-wise tasks, it significantly enhances group-wise tasks. These findings may reshape our understanding of the modality gap, highlighting its key role in improving performance on tasks requiring semantic grouping.