The More, the Merrier: Contrastive Fusion for Higher-Order Multimodal Alignment

AI summary

Overview

Research area: Multimodal machine learning and contrastive representation learning, with applications spanning computer vision, audio, and language (categorized on arXiv under Computer Vision).

Technical level: Intermediate. The paper builds on familiar contrastive objectives (CLIP-style InfoNCE) but introduces information-theoretic machinery (total correlation, conditional mutual information) and requires comfort with retrieval and linear-probe evaluation protocols.

Scope: The paper introduces Contrastive Fusion (ConFu), a single contrastive framework for three modalities that jointly optimizes pairwise alignment and higher-order "fused" alignment, and evaluates it on synthetic (XOR, AV-MNIST), affective (MOSI, UR-FUNNY, MUStARD), and fine-grained bird (SSW60, VB100) benchmarks alongside a newly constructed Bird-MML dataset.

What This Paper Is About

Most multimodal contrastive methods such as CLIP and ALIGN align exactly two modalities at a time, which limits them to correlations recoverable from pairs and leaves them vulnerable to modality competition, where a strong modality suppresses a weaker one. Some recent approaches attempt to model three or more modalities jointly, but they typically use a single joint objective that does not explicitly preserve pairwise relationships, which makes them ineffective on single-modality tasks and incompatible with standard one-to-one retrieval. ConFu addresses this by embedding both individual modalities and their fused combinations into one shared space, so that pairwise and higher-order dependencies are optimized together rather than one at the expense of the other.

Key Contributions

1. A unified contrastive framework that generalizes CLIP-like pairwise multimodal alignment to three modalities, combining a pairwise loss over all modality pairs with a fused loss that aligns each modality against the fused representation of the other two. The two are balanced by a weighting factor λ ∈ [0, 1].

2. Bird-MML, a synthetic dataset of artificially paired image, audio, and text triplets covering 150 species, including all species present in SSW60 and VB100, intended as a controlled pretraining resource for testing multimodal complementarity.

3. Competitive retrieval and classification performance, with the framework supporting both one-to-one (1→1) and two-to-one (2→1) retrieval inside a single model, unlike Symile and TRIANGLE which require all modalities at inference time.

4. Demonstrated robustness under distracting modalities and noise-induced distribution shift, where the framework outperforms all baselines even when fusion involves corrupted or non-informative inputs.

Main Findings

• Higher-order dependencies are captured where pairwise methods fail. On the synthetic XOR task, ConFu and Symile solve the task as the synergy parameter increases, while trimodal pairwise CLIP remains at chance (~3% accuracy) and both GRAM and TRIANGLE fail to exceed 15% accuracy.

• AV-MNIST zero-shot classification favors ConFu. ConFu reaches 71.2 ± 0.2 on audio-visual fusion versus 64.9 ± 0.3 for TRIANGLE, 64.4 ± 0.5 for GRAM, and 70.9 ± 0.3 for Symile. The paper reports that fusing audio and visual inputs yields an 8% improvement over the strongest unimodal baseline, and that ConFu's vision-only score (64.6 ± 0.3) surpasses the corresponding unimodal baseline by 1.5%.

• Retrieval improvements on affective benchmarks. In Recall@10 results on UR-FUNNY, ConFu reaches 7.4 mean for 1→1 and 13.6 for 2→1 retrieval; on MUStARD, 45.3 for 1→1 and 62.6 for 2→1. The paper notes that 2→1 retrieval surpasses 1→1 retrieval on UR-FUNNY and MUStARD, indicating that two query modalities provide richer cues.

• GRAM struggles to balance both retrieval settings. GRAM is the only baseline supporting both 1→1 and 2→1 retrieval, but it fails on modalities M2 and M3 on MOSI and UR-FUNNY, which the authors attribute to the difficulty of balancing 2→1 and 1→1 alignment.

• Linear-probe classification is competitive but mixed. ConFu attains the highest accuracy on UR-FUNNY (64.9 ± 1.0) and remains competitive on MOSI (66.7 ± 2.1) and MUStARD (64.1 ± 2.5), where Symile (67.5 ± 1.3) and GRAM (64.7 ± 2.3) respectively lead.

• Fine-grained bird classification favors fusion when both modalities are informative. In the multi-frame zero-shot protocol, ConFu achieves the highest SSW60 accuracy at 71.4%, compared with 64.1% for TRIANGLE, 60.2% for Symile, and 56.9% for GRAM. On VB100, where audio is largely uninformative (~4% unimodal accuracy), the fused model maintains 18.1% (reported in the text as 18.15%) against a best visual-only baseline of 20.69%, while GRAM, Symile, and TRIANGLE decline by 5–10% relative to ConFu.

• Single-frame evaluation confirms the pattern. Averaged over 19 uniformly sampled frames, ConFu reaches 65.5 ± 2.5 on SSW60 audio-visual fusion, exceeding the best baseline by roughly 4%, and 16.4 ± 0.4 on VB100, where performance drops by 2–3% due to the weaker audio modality.

• Robustness under distribution shift. With Gaussian noise applied to the vision modality only at test time (standard deviation 0.05, roughly 20dB SNR), ConFu achieves the highest accuracy at 45.4 ± 0.1, ahead of Symile at 40.9 ± 0.3 and Tri-CLIP (V) at 35.4 ± 0.4. TRIANGLE (26.5 ± 0.2) and GRAM (25.9 ± 0.1) fall below the unimodal baseline.

• Visual dominance is present but fusion adds unique value. Error-overlap analysis on SSW60 shows 39% of correct predictions are made by both fusion and vision, purely audio-driven successes are rare (2%), fusion-audio overlap is small (5–6%), and about 5% of samples are solved only by the audiovisual model. Certain classes, such as Label 31, benefit substantially from fusion.

• A modal masking fix addresses a MOSI weakness. In the MOSI 2→1 case, fusion slightly underperforms the pairwise baseline, an issue the authors report is substantially alleviated by introducing partial modality masking.

Methodology in Plain English

ConFu takes three modalities and passes each through its own encoder into a shared latent space. From there, two kinds of contrastive loss are applied. The pairwise loss is the standard InfoNCE objective applied to each of the three modality pairs, pulling matching samples together and pushing mismatched ones apart. The fused loss adds a second ingredient: a small fusion network (a shallow MLP in the experiments) combines the encoded features of two modalities into a single fused vector, which is then aligned against the third modality using the same InfoNCE-style objective. Because the fused representation is aligned with a held-out modality, the model is forced to encode information that only emerges from the interaction of the pair — such as XOR-like relationships that no pairwise term could detect. The total training objective is a weighted sum, (1 − λ) times the pairwise loss plus λ times the fused loss, and the authors show this corresponds to maximizing a contrastive lower bound on the total correlation among the three variables. The density ratio critics in both losses are implemented as temperature-scaled dot-product similarities, so no architectural changes to the encoders are required and the only added cost comes from the lightweight MLP.

To have data suitable for this study, the authors built Bird-MML: images drawn from iNaturalist (research-grade, Creative Commons–licensed, 1,000 samples per species), audio from Xeno-Canto segmented into 10-second clips and zero-padded when needed, and text captions produced by combining InstructBLIP2 image captions, audio metadata, and short Wikipedia summaries, fused with google/gemma-2-2b-it into one multimodal caption per instance. Roughly 43% of species required limited audio reuse due to data scarcity. Evaluation then proceeded through zero-shot classification, 1→1 and 2→1 retrieval, and linear-probe classification in few-shot and full-data regimes, with downstream tests performed on naturally paired video-audio data.

Why This Matters

Impact on research. The paper argues that prior higher-order methods pay a hidden cost: because they use one joint alignment objective, they lose the pairwise structure that makes single-modality tasks and standard retrieval work. By factorizing total correlation at the loss level instead, ConFu keeps pairwise and higher-order supervision independently controllable, which the authors present as enabling balanced performance from a single model. The theoretical framing — decomposing total correlation into pairwise mutual information terms plus conditional terms over all permutations, each bounded by InfoNCE — gives the framework an information-theoretic justification rather than an architectural one, and the approach works without modifying the encoders.

Real-world applications:

• Biodiversity monitoring and acoustic survey. The Bird-MML, SSW60, and VB100 experiments target fine-grained species recognition from images and calls, relevant to passive acoustic monitoring and camera-trap networks.

• Affective computing and human behavior analysis. Evaluations on MOSI, UR-FUNNY, and MUStARD point toward systems that infer sentiment or humor from text, audio, and video jointly.

• Cross-modal retrieval and search. The unified 1→1 and 2→1 retrieval capability supports searching with one signal or with a composition of two — for instance retrieving a target from an image plus a text description, or an audio clip plus text.

• Robust multimodal sensing under degraded inputs. The noise-shift experiments (vision perturbed at roughly 20dB SNR) speak to deployments where one sensor is unreliable, and the modality-overlap analysis shows the model recovers a subset of samples that no single modality solves.

Industry relevance. The claim of minimal overhead — lightweight MLPs atop existing encoders — matters for practitioners who already have trained unimodal or pairwise models and want higher-order behavior without retraining from scratch. The explicit support for both retrieval configurations in one model reduces the need to maintain separate systems, and the reported stability under corrupted or non-informative modalities addresses a common operational failure mode in production pipelines.

Future Directions

1. Selective pruning of alignment terms. The authors state that as the number of modalities grows, computational demands may also rise, and suggest pruning certain alignment terms — for example, omitting many-to-many alignments when only single-modality queries or targets are needed — as a route to efficiency.

2. Handling partially missing modalities. ConFu currently relies on fully aligned modalities during training. Relaxing this, such as training with incomplete triplets or drawing on pseudo-pair construction ideas, is identified as crucial for broader applicability.

3. Factorized and redundancy-reduction representations. The paper proposes extending ConFu with representations inspired by recent factorization-based and redundancy-reduction approaches.

4. Adaptive regulation of modality competition. Given the observed visual dominance in the SSW60 overlap analysis, the authors suggest developing adaptive mechanisms to regulate modality competition, and investigating robustness in retrieval when one or more modalities are noisy, incomplete, or severely corrupted.

Target Audience

Researchers and graduate students working on multimodal representation learning, contrastive objectives, and cross-modal retrieval will find the framework and its information-theoretic derivation most directly useful. Practitioners in bioacoustics, biodiversity monitoring, and affective computing benefit from the dataset construction pipeline and the robustness results. Engineers building retrieval or fusion systems will find the architecture-agnostic formulation, the 1→1 and 2→1 unified retrieval support, and the modest computational overhead the most practically relevant elements. Readers without background in contrastive learning or mutual information estimation will need to consult the InfoNCE references the paper cites.

Authors’ abstract

Learning joint representations across multiple modalities remains a central challenge in multimodal machine learning. Prevailing approaches predominantly operate in pairwise settings, aligning two modalities at a time. While some recent methods aim to capture higher-order interactions among multiple modalities, they often overlook or insufficiently preserve pairwise relationships, limiting their effectiveness on single-modality tasks. In this work, we introduce Contrastive Fusion (ConFu), a framework that jointly embeds both individual modalities and their fused combinations into a unified representation space, where modalities and their fused counterparts are aligned. ConFu extends traditional pairwise contrastive objectives with an additional fused-modality contrastive term, encouraging the joint embedding of modality pairs with a third modality. This formulation enables ConFu to capture higher-order dependencies, such as XOR-like relationships, that cannot be recovered through pairwise alignment alone, while still maintaining strong pairwise correspondence. We evaluate ConFu on synthetic and real-world multimodal benchmarks, assessing its ability to exploit cross-modal complementarity, capture higher-order dependencies, and scale with increasing multimodal complexity. Across these settings, ConFu demonstrates competitive performance on retrieval and classification tasks, while supporting unified one-to-one and two-to-one retrieval within a single contrastive framework. We release our code and dataset at https://github.com/estafons/confu.