ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis

AI summary

Overview

Research area: Multimodal sentiment analysis (MSA) — predicting human sentiment from text, audio, and video signals — with a specific focus on consistency-aware learning and token-level efficiency.

Technical level: Advanced. The paper assumes familiarity with Transformer encoders, cross-modal attention, divergence measures (KL, Jensen–Shannon), and token pruning/compression.

Scope (one sentence): The paper proposes ConsMSA, a framework that formalizes intra- and inter-modality semantic distribution consistency via a Jensen–Shannon-divergence-based score, and uses it simultaneously as a training regularizer, a differentiable token-reweighting gate, and a token-pruning criterion for sentiment prediction.

What This Paper Is About

Multimodal sentiment models encode text, audio, and video into token-level representations, but many of those tokens are redundant (neutral or repeated signals within a modality) or outright conflicting (e.g., positive words delivered with negative prosody, as in sarcasm). This injects noise into fusion and wastes computation on tokens that do not support a coherent sentiment decision. ConsMSA's goal is to explicitly measure how consistent each token's predicted sentiment distribution is with the other tokens in its own modality and in the other modalities, then use that measure to regularize training, reweight features, and prune tokens — aiming for both higher accuracy and heavy compression without accuracy loss.

Key Contributions

1. Holistic consistency paradigm. A framework (ConsMSA) that explicitly formalizes intra-modality and inter-modality semantic distribution consistency, treating redundancy and cross-modal conflict as a single underlying phenomenon — token-level semantic inconsistency — rather than as separate problems requiring separate mechanisms.

2. An information-theoretic consistency metric. The Intra- and Inter-modality Consistency Score (I²CS), built on Jensen–Shannon (JS) divergence, which compares token-induced sentiment distributions in a shared semantic space. JS is chosen because it is symmetric, finite, and bounded, unlike KL divergence.

3. Consistency-aware importance signals. The I²CS score is coupled with a predictive relevance term (the max probability of a token's latent sentiment distribution) to form Signal(h_i) = Rel(h_i) − I²CS(h_i). This single signal drives three mechanisms: (i) a relevance-weighted consistency loss, (ii) a differentiable sigmoid gating mask for soft reweighting, and (iii) a TopK hard-selection operator for token compression.

4. Strong empirical results with efficiency. State-of-the-art results on CMU-MOSI, CMU-MOSEI, and CH-SIMS across both BERT-based (T+A+V) and RoBERTa-based (T+A) settings, with competitive accuracy retained when only 10% of tokens are kept.

Main Findings

• Best results across all reported metrics and backbones. In the BERT-based (T+A+V) setting, ConsMSA reaches Acc-7 47.96, Acc-2 86.06, F1 86.06, MAE 0.698 on CMU-MOSI and Acc-7 54.92, Acc-2 86.67, F1 86.65, MAE 0.525 on CMU-MOSEI, against the prior strongest listed baseline MDF (48.25 / 85.82 / 85.90 / 0.692 on MOSI; 54.80 / 86.61 / 86.41 / 0.525 on MOSEI). In the RoBERTa-based (T+A) setting it reaches Acc-7 51.46, Acc-2 90.09, F1 90.08, MAE 0.557 on MOSI and Acc-7 56.00, Acc-2 88.41, F1 88.39, MAE 0.485 on MOSEI, versus MMML at 50.34 / 89.69 / 89.67 / 0.580 and 55.74 / 88.02 / 88.15 / 0.492.

• Gains transfer across text encoders. Replacing BERT with RoBERTa improves the textual backbone, yet ConsMSA still adds gains on top — evidence that the contribution is not tied to a specific encoder.

• CH-SIMS improvement over the prior SOTA. Under the matched T+A protocol against MMML, ConsMSA obtains Acc-2 83.59, Acc-3 70.68, Acc-5 50.77, F1 83.64, MAE 0.331 versus 82.93 / 69.37 / 49.38 / 82.90 / 0.332 — absolute gains of 1.31% in Acc-3 and 1.39% in Acc-5.

• Pruning can beat full retention. On MOSI, hard selection at a retention ratio of 0.8 (Acc-2 90.85, F1 90.84, MAE 0.553) outperforms soft selection at ratio 1.0 (Acc-2 90.09, F1 90.08, MAE 0.557), which the authors interpret as I²CS-driven pruning acting as a "semantic purifier."

• Optimal retention differs by dataset. The best ratio is 0.6 on CH-SIMS (Acc-2 83.59, F1 83.70, MAE 0.337) versus 0.8 on MOSI, which the authors attribute to differing redundancy in Chinese versus English linguistic data.

• Robustness under extreme compression. At a 0.1 retention ratio, MOSI still reaches Acc-2 89.18, F1 89.17, MAE 0.576 and CH-SIMS reaches Acc-2 81.18, F1 81.32, MAE 0.355 — comparable to full-sequence performance.

• I²CS beats common masking heuristics on MOSI. Compared with the proposed masking signal (Acc-7 51.46, Acc-2 90.09, F1 90.08, MAE 0.557), alternatives score: Entropy 50.29 / 88.72 / 88.75 / 0.578; Attention 50.48 / 88.63 / 88.62 / 0.572; Random 49.27 / 88.33 / 88.31 / 0.575; KL-div 50.32 / 89.30 / 89.20 / 0.612; Cosine 47.96 / 88.90 / 88.71 / 0.628; L2 47.23 / 88.42 / 88.06 / 0.698. Cosine and L2 even underperform random masking.

• Ablations show every component matters (MOSI, T+A). Removing IntraCS gives 47.52 / 87.20 / 87.21 / 0.629; removing InterCS gives 46.79 / 88.87 / 88.83 / 0.626; removing relevance from Eq. 6 gives 45.48 / 87.96 / 87.99 / 0.668; removing relevance from Eq. 9 gives 46.36 / 87.80 / 87.82 / 0.647; removing gating gives 45.77 / 87.20 / 87.21 / 0.653; removing the modality-specific loss gives 47.38 / 87.50 / 87.51 / 0.643; removing the I²CS loss gives 48.83 / 86.59 / 86.62 / 0.634.

• Consistency visibly improves after training. Token-level visualizations show weak intra- and inter-modality consistency before optimization and substantially higher consistency afterward (lower divergence values indicate higher consistency).

• Fine-grained accuracy is limited by long-tail classes. On MOSI, most sentiment categories exceed 50% accuracy, but "HN" and especially "HP" are much weaker; those classes contain only 46 and 20 samples respectively, which constrains 7-class accuracy.

Methodology in Plain English

ConsMSA starts by encoding the raw audio and text into token-level features (the paper's main formulation uses text and audio; the benchmark tables also report T+A+V results).

Step 1 — Turn every token into a sentiment guess. Each token is passed through a small projection head followed by a softmax, producing a probability distribution over sentiment categories. A token now "votes" on the sentiment, in the same shared semantic space as every other token. Its confidence — the maximum probability in that distribution — is its relevance.

Step 2 — Score consistency with JS divergence. For each token, the model compares its vote to (a) all other tokens in the same modality, averaged over T−1 tokens (IntraCS), and (b) tokens in the paired modality or modalities, averaged over N = M−1 modality pairs and T tokens (InterCS). These are combined as I²CS = α·IntraCS + β·InterCS, where α and β balance redundancy suppression against conflict mitigation. Lower I²CS means higher consistency. JS divergence is used because it is symmetric, finite, and bounded — properties KL divergence lacks.

Step 3 — Combine with confidence. A token is useful if it is both confident and in agreement with the rest. The paper defines Signal(h_i) = Rel(h_i) − I²CS(h_i), rewarding confidence and penalizing disagreement.

Step 4 — Use the signal three ways.

• As a loss: ℒ_I²CS averages I²CS weighted by relevance, so confident tokens are pushed harder toward agreement and uncertain tokens are not over-penalized.
• As a soft gate: weights w_i = σ(γ · Signal(h_i)) multiply token features, so coherent, confident tokens contribute more to fusion.
• As a hard filter: a TopK operator keeps only the highest-weight K tokens, giving an explicit compressed sequence. Crucially, this is the discrete version of the same criterion used for soft gating, not a separate heuristic.

Step 5 — Train jointly. The final objective is ℒ_joint = λ₁ℒ_task + λ₂ℒ_I²CS + λ₃ℒ_MS, where ℒ_task is an ℓ1 regression loss on the fused prediction and ℒ_MS applies unimodal supervision to text and audio heads so fusion cannot rely on spurious cross-modal correlations. Optimized tokens are concatenated and passed to a fusion module and predictor.

Setup details: PyTorch on a single NVIDIA A100 GPU with 40GB memory; RoBERTa text encoder producing 1024-dimensional embeddings; Data2Vec-Audio producing 768-dimensional audio tokens; AdamW with initial learning rate 5×10⁻⁶, batch size 8, early stopping with patience of 8 epochs; BERT is also used as a text encoder for comparison with prior SOTA such as MDF.

Why This Matters

Impact on research. The paper reframes multimodal noise as a single measurable quantity — semantic distribution inconsistency in prediction space — rather than a collection of separate architectural problems (disentanglement, alignment, attention design). It also connects efficiency work to representation learning: the same score that regularizes training also ranks tokens for pruning, which prior token-selection work (attention-, entropy-, or learned-pruning-based, scored in feature space) does not do. The finding that distance-based criteria such as cosine and L2 similarity fall below random masking is a useful negative result for anyone building pruning heuristics.

Real-world applications:

• Customer-feedback analytics: triaging call-center or product-review recordings where tone and words may disagree, and where long inputs make aggressive token compression valuable.
• Social-media and brand monitoring: running sentiment models at scale on video content under tight compute budgets, since the method holds up at 10% token retention.
• Human–computer interaction and conversational agents: detecting sarcasm and affect mismatch so a system does not misread a user's emotional state.
• Clinical or behavioral screening support: flagging affective inconsistency in recorded interviews, where robust handling of conflicting cues matters more than raw accuracy.

Industry relevance. The compression results matter for deployment: retaining 10% of tokens while keeping comparable accuracy directly reduces inference cost and memory for Transformer-based multimodal pipelines, and the method is backbone-agnostic (demonstrated with both BERT and RoBERTa text encoders), lowering the cost of adopting it on top of an existing stack.

Future Directions

1. Long-tail and fine-grained sentiment. The confusion analysis on MOSI shows "HN" and "HP" underperforming, with only 46 and 20 samples respectively. Class-balanced or reweighted variants of the consistency objective are an obvious next step, and the paper explicitly motivates future exploration in this direction.

2. Adaptive, per-input retention. The optimal retention ratio differed across datasets (0.6 for CH-SIMS, 0.8 for MOSI). Learning the ratio per example, rather than fixing a global K, would remove a dataset-specific hyperparameter.

3. Better divergence choices. KL-divergence-based masking was the closest competitor (Acc-7 50.32, Acc-2 89.30, F1 89.20, MAE 0.612 on MOSI) but still trailed the JS-based signal. The margin is modest, leaving room for other bounded divergences or learned consistency metrics.

4. Extending the consistency formulation beyond two modalities in the main evaluation. The paper's core formulation is presented for text and audio with N = M−1 modality pairs; the main comparison on CH-SIMS is restricted to T+A for protocol alignment with MMML, with T+A+V deferred to the Appendix — full tri-modal analysis and generalization to other modality sets remain open.

Target Audience

Researchers and graduate students working on multimodal learning, affective computing, and efficient Transformer inference; practitioners deploying sentiment or emotion models under compute constraints; and engineers interested in token-pruning and feature-selection methods that are aligned with the downstream decision boundary rather than with feature-space saliency. Readers should be comfortable with softmax distributions, KL/JS divergence, and standard multimodal fusion architectures — the paper is not beginner-level.

Authors’ abstract

Multimodal sentiment analysis (MSA) aims to predict human sentiments by integrating signals from different modalities such as text, video, and audio. However, raw multimodal sequences often suffer from semantic inconsistency--exhibiting redundancy or conflicts within and across modalities--which hinders robust understanding and increases computational cost. To this end, we introduce ConsMSA, which explicitly formalizes semantic distribution consistency across both $\textit{intra}$- and $\textit{inter}$-modality, providing a consistency-aware mechanism for robust and efficient multimodal sentiment prediction. Specifically, ConsMSA projects multimodal token features into a shared sentiment space to compute an Intra- and Inter-modality Consistency Score ($I^2CS$). By coupling this score with predictive relevance, we formulate consistency-aware importance signals that are utilized: (i) as a consistency regularizer to align latent distributions during training, (ii) to derive semantic-aware weights for adaptive multimodal token reweighting, and (iii) as a practical criterion to prune redundant or conflicting tokens. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate that ConsMSA achieves state-of-the-art performance while remaining robust under aggressive token compression--retaining only 10% of tokens yields comparable accuracy. These results establish semantic distribution consistency as a promising foundation for synergizing predictive robustness with computational efficiency.