PaSE: Prototype-aligned Calibration and Shapley-based Equilibrium for Multimodal Sentiment Analysis

AI summary

Overview

Research area: Multimodal Sentiment Analysis (MSA) — predicting emotional polarity or intensity from text, audio, and visual signals — with a focus on the machine learning problem of modality competition during fusion.

Technical level: Advanced. The paper combines entropic optimal transport, prototype-based contrastive learning, cooperative game theory (Shapley values), and gradient modulation, and assumes familiarity with multimodal fusion architectures and representation learning.

Scope: The paper proposes PaSE, a three-part framework (prototype-guided calibration, entropic-optimal-transport cross-modal alignment, and dual-phase optimization with Shapley-based gradient modulation) and evaluates it on CMU-MOSI, CMU-MOSEI, and IEMOCAP against 14 baselines plus several LLMs and MLLMs.

What This Paper Is About

Multimodal sentiment analysis assumes that combining text, audio, and vision should beat any single modality, but in practice dominant modalities (usually text) transmit stronger gradient signals and suppress weaker ones — a phenomenon the paper calls modality competition. The authors build PaSE to keep that competition in check, so that audio and visual signals meaningfully contribute instead of being drowned out. The goal is both higher accuracy on sentiment benchmarks and more balanced, cooperative use of all three modalities.

Key Contributions

1. A unified framework (PaSE) that combines prototype-aligned calibration with Shapley-optimized equilibrium to explicitly address modality competition in multimodal sentiment analysis.

2. An entropic optimal transport (EOT) cross-modal prototype alignment mechanism that enforces structural consistency and bidirectional matching between modality-specific class prototypes, aiming to reduce semantic discrepancies across modalities.

3. A dynamic fusion and optimization scheme: a Prototype-Gated Fusion (PGF) module for extracting shared representations, plus Shapley-based Gradient Modulation (SGM) that reweights each modality's gradients according to its estimated marginal contribution.

4. Extensive evaluation on three benchmarks (IEMOCAP, CMU-MOSI, CMU-MOSEI) reporting state-of-the-art results, ablation studies for each module, comparisons of fusion strategies, modality-combination tests, and comparisons against general-purpose LLMs and MLLMs.

Main Findings

• Best reported results on CMU-MOSI: PaSE reaches Acc-2 of 86.40/88.32, F1 of 86.34/88.25, Acc-7 of 50.92, Corr of 0.847, and MAE of 0.695 (the two Acc-2/F1 forms correspond to including versus excluding zero as negative/non-negative).

• Best reported results on CMU-MOSEI: Acc-2 of 86.07/88.10, F1 of 86.21/87.96, Acc-7 of 55.76, Corr of 0.831, and MAE of 0.523.

• Gains over the strongest prior baseline: relative gains of 0.89% in Acc-2 and 1.25% in Acc-7 over MSAmba (the most recent state-of-the-art model cited) on CMU-MOSI.

• IEMOCAP results by emotion class (F1): Happy 91.5, Sad 88.6, Angry 89.4, Neutral 73.2, with the paper reporting an average improvement of 2.93% over prior methods. Weighted accuracy (WA) on IEMOCAP is 80.47.

• SGM matters most in ablation: removing Shapley-based Gradient Modulation drops performance by 2.85% on MOSI and 3.07% on MOSEI; removing cross-modal alignment drops it by 1.52% and 1.41%; removing prototype-guided calibration drops it by 1.07% and 1.19%.

• Ablation on fusion strategy: replacing the Prototype-Gated Fusion module lowers scores, and PGF outperforms simple sum ("Sum" performs worst), MLP concatenation ("Con"), and attention-based fusion ("Att") across the reported metrics.

• Modality-combination analysis on MOSI (negative/positive Acc-2 only): Audio 60.01, Visual 61.43, Text 84.70, Audio+Visual 63.35, Text+Audio 86.71, Text+Visual 87.14, and all three modalities 88.32 — a 3.62% improvement over the best single modality.

• Fusion gains: adding the visual modality to text yields a 2.76% improvement in the paper's Figure 1 analysis, compared with GCNet's 0.60% under the same condition; the full-modality setting achieves an average gain of 4.02% over bimodal variants.

• Prototype-Gated Fusion is context-sensitive: on IEMOCAP, anger samples receive the highest average weight on audio (0.392), consistent with prosodic cues, while neutral expressions are dominated by text (0.681).

• SGM reduces text dominance: without SGM the text modality overwhelmingly dominates fusion; with SGM, text dominance is significantly reduced and visual and audio contributions increase, alongside consistently improving fusion accuracy.

• General-purpose LLMs and MLLMs underperform specialized models in this domain: Qwen2-7B reaches 40.15 WA on IEMOCAP, Llama-3-8B 38.60, Internlm-2.5-7B 39.49, mPLUG-Owl 55.17, Qwen2-VL-7B 34.23, LLaVA-OV-7B 40.09, VideoLLaMA2-7B 38.86, and GPT-4o-mini 46.33, all below PaSE's 80.47 on a BERT-base backbone.

• Robustness under missing modalities: the modality-combination results indicate the model retains competitive performance with partial inputs.

• Reported prototypes and training setup: momentum coefficient γ = 0.98 for prototype updates, λ = 0.01, μ = 0.1, α = 0.1, β = 0.05, Adam optimizer with learning rate 1e-5, batch size 64, 200 epochs, trained on a single NVIDIA A100 GPU.

• Dataset splits reported: CMU-MOSI has 1,284 training, 229 validation, and 686 test samples; CMU-MOSEI has 16,326 training samples (the truncated content does not report its validation or test counts). Results are averaged over five runs with different random seeds.

Methodology in Plain English

PaSE keeps text, audio, and vision on separate branches for as long as possible before fusing them.

First, each modality is calibrated on its own. The model computes a "prototype" for every sentiment class in every modality — a centroid of the features belonging to that class in the current mini-batch — and updates these prototypes with a moving average rather than recomputing them from scratch. A contrastive-style loss then pulls each sample toward the prototype of its own class and away from other classes. Because each modality is trained against its own semantic structure, weaker modalities can sharpen their features without being overwritten by the dominant one.

Second, the modalities are aligned to each other. Class prototypes from two modalities are treated as discrete distributions, and the alignment is framed as an optimal transport problem: find the cheapest way to match one modality's prototypes to the other's. Since exact optimal transport is computationally prohibitive, the authors use an entropy-regularized relaxation. Matching is computed in both directions and averaged, with an extra term penalizing disagreement between the forward and backward transport matrices, plus a structure-preserving term that pushes each transport matrix toward the identity so class semantics stay intact.

Third, training is split into two phases. In the early phase, modality weights are set from the entropy of each modality's predictions via a softmax, and a gating mechanism conditioned on both the modality embedding and its class prototype produces the fused representation — this lets a strong modality lead early on. Once validation entropy stabilizes, training shifts to the balancing phase. Here each modality is assigned a Shapley value, computed from a utility function that blends inter-modal and intra-modal loss terms, to measure its marginal contribution. Normalized Shapley values become per-modality modulation factors that scale the gradient update: weaker modalities get amplified learning rates, dominant ones get suppressed.

Why This Matters

Impact on research: The paper reframes multimodal fusion as a non-cooperative game, quoting prior work that describes multimodal learning this way. Its distinctive move is to use Shapley values — a principled quantification of marginal contribution from cooperative game theory — instead of the indirect signals (gradient norms, loss magnitudes) used by earlier gradient-modulation methods. It also adds optimal transport as a cross-modal alignment tool alongside prototype-based calibration, and documents that general-purpose LLMs and MLLMs with billions of parameters still trail a lightweight BERT-base model on these affective benchmarks.

Real-world applications:

• Affective dialogue systems that need to read frustration, satisfaction, or confusion from a caller's words, tone, and facial expression.
• Mental health and wellbeing monitoring, where subtle acoustic and visual cues may carry emotional signal that text alone misses.
• Social media and public-opinion analysis, where posts pair text with images, video, or audio.
• Human-computer interaction and virtual agents that adapt responses based on multimodal emotional state.

Industry relevance: The method uses standard, widely available feature extractors (Facet for visual, COVAREP for acoustic, BERT-base for text) and a single A100 GPU, keeping it accessible relative to large MLLM deployments. Robustness under missing modalities is directly relevant to production systems where a camera or microphone feed may be unavailable. The finding that LLMs lag on fine-grained affective tasks supports continued investment in specialized, modality-aware architectures rather than relying on general-purpose models.

Future Directions

• Beyond three-modality settings: the framework is defined over the modality set {text, audio, vision}; how Shapley estimation and optimal transport scaling behave with more or fewer modalities, or with additional sensor streams, is not reported.

• Automating the phase transition: the switch from the fusion phase to the Shapley balancing phase is triggered when validation entropy stabilizes, with a warm-up period to prevent early overfitting; more principled or adaptive switching criteria remain an open question.

• Better understanding of competition versus collaboration: the paper states that the intricate dynamics between modality collaboration and competition remain underexplored, so a deeper theoretical account of when dominance helps versus hurts is still missing.

• Closing the LLM gap: the large margin between PaSE and both LLMs and MLLMs on IEMOCAP raises the question of whether dedicated adaptation or modality-aware optimization could bring general-purpose models up to specialized performance.

Target Audience

Researchers and graduate students working on multimodal learning, affective computing, and sentiment analysis; practitioners building fusion systems where one input channel tends to dominate; and readers interested in applying cooperative game theory (Shapley values) or optimal transport to representation learning. The paper is best suited to readers comfortable with loss formulation, transport plans, and gradient-based training — the ablation and modality-combination tables are the most directly useful parts for applied work.

Authors’ abstract

Multimodal Sentiment Analysis (MSA) seeks to understand human emotions by integrating textual, acoustic, and visual signals. Although multimodal fusion is designed to leverage cross-modal complementarity, real-world scenarios often exhibit modality competition: dominant modalities tend to overshadow weaker ones, leading to suboptimal performance. In this paper, we propose PaSE, a novel Prototype-aligned Calibration and Shapley-optimized Equilibrium framework, which enhances collaboration while explicitly mitigating modality competition. PaSE first applies Prototype-guided Calibration Learning (PCL) to refine unimodal representations and align them through an Entropic Optimal Transport mechanism that ensures semantic consistency. To further stabilize optimization, we introduce a Dual-Phase Optimization strategy. A prototype-gated fusion module is first used to extract shared representations, followed by Shapley-based Gradient Modulation (SGM), which adaptively adjusts gradients according to the contribution of each modality. Extensive experiments on IEMOCAP, MOSI, and MOSEI confirm that PaSE achieves the superior performance and effectively alleviates modality competition.