Multimodal Classification via Total Correlation Maximization

AI summary

Overview

• Research area: Multimodal (mainly audio-visual) deep learning, specifically multimodal classification, and information-theoretic training objectives. It sits at the intersection of multimodal fusion, mutual-information estimation, and contrastive representation learning.
• Technical level: Intermediate. The experimental setup and intuition are accessible, but the derivations around total correlation, MINE, and Gibbs distributions require comfort with information theory.
• One-sentence scope: The paper diagnoses modality competition through an information-theoretic lens and proposes a hyperparameter-free training loss, TCMax, that maximizes the total correlation between per-modality features and labels, evaluated on five audio-visual classification benchmarks.

What This Paper Is About

When a multimodal model is trained jointly on several modalities (for example audio and visual), one modality often converges faster and dominates the shared objective, so the fused model can end up performing worse than the best single modality alone. The authors show that this happens because joint training maximizes the mutual information between the fused feature and the label, which decomposes into a term for the fast modality plus a conditional term that has almost no room left once the fast modality is already informative. Their goal is a training objective that keeps the strengths of joint learning (cross-modal interaction) and unimodal learning (each modality trained on its own signal) while also aligning modalities, without adding extra hyperparameters or changing the model architecture.

Key Contributions

1. An information-theoretic account of modality competition. The authors decompose the joint-learning objective into I(y; z^(a)) + I(y; z^(v) | z^(a)) and show the conditional term is upper bounded by H(y) − I(y; z^(a)), so a well-fitted modality starves the others.
2. Total Correlation Neural Estimation (TCNE). Building on Mutual Information Neural Estimation (MINE), they derive a dual representation and lower bound for the total correlation among M modality features and the label, showing MINE is the special case of TCNE with two variables.
3. The TCMax loss. By setting the critic function T to the label logit of the existing prediction head, they obtain a hyperparameter-free loss that maximizes total correlation via variational bound optimization, with Propositions 1–3 establishing that the lower bound improves and that a TCMax-trained model still outputs the correct conditional label distribution, so no prediction-time modifications are needed.
4. A computational-cost analysis. The naive form requires |B|^M forward passes of the prediction head; the paper gives a negative-pair sampling variant and shows that with linear fusion the denominator decouples into per-modality sums, requiring only |B| forward passes.

Main Findings

• Modality competition is visible in the baselines. In Table 1, plain concatenation on CREMA-D reaches 68.5 multimodal accuracy while its audio branch alone reaches 58.6 and its visual branch only 26.5. Even MMPareto reports 74.4 multimodal on CREMA-D while its own visual modality alone reports 75.7, illustrating the reported phenomenon that joint learning can trail the best unimodal network.
• TCMax is strongest on four of five audio-visual benchmarks. TCMax with a shared head reports 63.5 on Kinetics-Sounds, 47.5 on VGGSound, and 56.0 on UCF101 multimodal accuracy, and TCMax with concatenation

Authors’ abstract

Multimodal learning integrates data from diverse sensors to effectively harness information from different modalities. However, recent studies reveal that joint learning often overfits certain modalities while neglecting others, leading to performance inferior to that of unimodal learning. Although previous efforts have sought to balance modal contributions or combine joint and unimodal learning, thereby mitigating the degradation of weaker modalities with promising outcomes, few have examined the relationship between joint and unimodal learning from an information-theoretic perspective. In this paper, we theoretically analyze modality competition and propose a method for multimodal classification by maximizing the total correlation between multimodal features and labels. By maximizing this objective, our approach alleviates modality competition while capturing inter-modal interactions via feature alignment. Building on Mutual Information Neural Estimation (MINE), we introduce Total Correlation Neural Estimation (TCNE) to derive a lower bound for total correlation. Subsequently, we present TCMax, a hyperparameter-free loss function that maximizes total correlation through variational bound optimization. Extensive experiments demonstrate that TCMax outperforms state-of-the-art joint and unimodal learning approaches. Our code is available at https://github.com/hubaak/TCMax.