Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis

AI summary

Overview

• Research area: Multi-modal sentiment analysis (MSA) — predicting both the polarity and the intensity of sentiment from text, audio, and video — with a focus on natural language processing, multi-modal fusion, and large language models.
• Technical level: Advanced. The paper assumes familiarity with cross-attention, sparse mixture-of-experts, Mamba/state-space models, and fine-tuning multi-modal LLMs.
• Scope: The paper proposes and evaluates TEXT, a text-routed sparse mixture-of-experts model that uses MLLM-generated explanations and a new temporal alignment block to improve multi-modal sentiment analysis across the MOSI, MOSEI, CH-SIMS, and CH-SIMSv2 datasets.

What This Paper Is About

Multi-modal sentiment analysis models must combine text, audio, and video, but the modalities disagree: in the paper's running example from MOSI, only the text modality predicts the correct polarity, so audio and video can actively mislead the model. The authors argue that two tools are underexplored for fixing this — using explanations (generated by multi-modal large language models) to align the modalities, and designing a temporal alignment block suited to short videos rather than to the long-video or generic settings that Mamba and temporal cross-attention were built for. TEXT is their answer: it generates explanations with MLLMs, aligns audio and video with those explanations through cross-attention, aligns audio against video with a lightweight temporality-oriented block, and then routes a sparse mixture-of-experts using text as the gate, finishing with a gate-fusion classifier.

Key Contributions

1. TEXT aligns encoded audio and video through a novel temporality-oriented neural network block that combines and simplifies the benefits of Mamba and temporal cross-attention.
2. TEXT augments data for MSA via a multi-modal LLM that produces explanations, and these explanations are aligned with the audio and video representations.
3. TEXT introduces a new text-routed sparse mixture-of-experts module, in which routing decisions are driven by the text modality.
4. TEXT is reported as the winner among all tested models on four datasets across six metrics (the paper states it wins on at least four of the six metrics).

Main Findings

• Best overall across four datasets: TEXT achieves the best performance across MOSI, MOSEI, CH-SIMS, and CH-SIMSv2 among all tested models, which include three recently proposed approaches (ALMT, KuDA, DEVA) and three MLLMs (Qwen2.5-vl, GPT-4o, VideoLlama3-7B), winning at least four of six metrics.
• CH-SIMS MAE improvement: TEXT lowers MAE on CH-SIMS to 0.353, versus 0.449 for ALMT, 0.408 for KuDA, and 0.424 for DEVA — described as a 13.5% decrease relative to recently proposed approaches, and better than Qwen's 0.404.
• Other CH-SIMS wins: TEXT reaches 50.15% Acc-5 versus KuDA's 43.54%, and 86.75% F1 versus KuDA's 80.71%.
• MOSI wins: TEXT achieves 88.72% Acc-2 versus GPT-4o's 86.74%, and 88.76% F1 versus GPT-4o's 86.68%.
• Exceptions to the pattern: KuDA is best on Acc-7 for MOSI and MOSEI and on Acc-5 for CH-SIMSv2; DEVA leads on Acc-5 for MOSEI. The authors conjecture that fine-grained scoring needs additional (possibly domain-specific) sentiment knowledge.
• Text is the dominant modality: In the MOSEI ablation, text alone with explanations reaches 52.84% Acc-7 while full TEXT reaches 52.29%, and text outperforms audio and video on Acc-5 and Acc-7 across metrics.
• Audio and video contribute comparably: The ablation results for text+video and text+audio are similar, suggesting both supply distinct but equally valuable contextual cues (prosody in audio, visual signals in video).
• Explanations matter: Removing explanations causes roughly a 2% decline across most metrics.
• Temporal alignment is the most crucial component for MAE: The ablation study identifies temporal alignment as most crucial for MAE; replacing it with Mamba or TCA causes a roughly 2% drop, and replacing it with concatenation yields an MAE of 0.580.
• SMoE is comparable in value to explanations: The authors attribute this to keyword-sensitive expert activation, where experts trained for specific topics are activated by text containing the corresponding keywords, which may also foster cross-modal consistency and interpretability.
• Case study: For the MOSI example with human label 1.400, TEXT predicts 1.390 (deviation 0.010), versus ALMT 1.080 (0.320), GPT-4o 0.800 (0.600), VideoLLaMA3 2.000 (0.600), and Qwen2.5-vl 2.500 (1.100).
• Explanation benefit for audio in the case study: With explanations, audio-only prediction has a deviation of 0.380; without explanations the audio-only deviation rises to 1.440.
• Nonetheless, GF and SMoE had minimal impact in that single case: Without GF or SMoE the deviation increases by 0.050.
• MLLM weakness on Chinese data: The authors note GPT-4o's power diminishes on Chinese datasets and cite work suggesting datasets may be memorized by MLLMs.

Methodology in Plain English

TEXT is built from six modules. First, it creates explanations in two stages: VideoLLaMA 3, fine-tuned on the EMER-fine dataset, generates raw explanations with a prompt that separates audio, video, and comments; then Qwen 3 refines those raw explanations into "fine" explanations using a checking prompt (the second-stage prompt is called the reasoning prompt).

Each modality is then encoded separately: subtitles and explanations with BERT, audio with Librosa, and video with OpenFace. Because these encodings have different lengths, the model uses 50 tokens per uni-modal encoding plus one learnable token for feature aggregation, producing a 51-dimensional embedding. In the explanation alignment blocks, audio, video, and subtitles are each aligned with their matching explanation or comment through cross-attention, producing aligned representations E_t, E_a, and E_v.

The temporal alignment block then combines the audio and video representations. It uses one Conv1d plus two linear operations with layer normalization and the Sigmoid Linear Unit activation, combining branches by element-wise multiplication and addition and concatenating the results into E_av. The authors stress that this block is simpler than Mamba and temporal cross-attention because it involves neither cross-attention nor a state-space model.

The aligned audio-video representation is then fed, together with the text representation, into a text-routed sparse mixture-of-experts, where text is the routing key. Finally, an MLP with gate fusion acts as the classifier, and only the tokens used for feature aggregation feed the final decision.

Experiments compare TEXT against ALMT, KuDA, DEVA, Qwen2.5-vl, GPT-4o, and VideoLlama3-7B on MOSI (2,199 subjective words and video clips), MOSEI (22,856 YouTube monologues and video segments covering 250 topics from 1,000 speakers), CH-SIMS (2,281 human-labeled video clips with scores from -1 to 1), and CH-SIMSv2 (the updated version of CH-SIMS). Metrics span classification (weighted F1, Acc-2, Acc-3, Acc-5, Acc-7) and regression (MAE and Pearson correlation), with Acc-2 and F1 computed in two settings: negative/non-negative (including zero) and negative/positive (excluding zero). All reported results are stated to be statistically significant using a T-test at p < 0.05. Code is available at https://github.com/fip-lab/TEXT.

Why This Matters

Research impact. The paper argues that alignment, not just fusion, is the crucial link between representation learning and multi-modal fusion, and it shows that MLLM-generated explanations can be used not only as extra text but as a supervision signal for aligning audio and video. It also transplants two techniques with limited prior use in MSA — sparse mixture-of-experts and gate fusion — into the task, and reports that a task-specific temporal block can beat Mamba and temporal cross-attention on short-video sentiment.

Real-world applications (drawn from the paper's stated application areas):

• Healthcare applications that rely on interpreting patients' emotional states across modalities.
• Human-computer interaction and human-machine interaction systems that must read subtle emotional nuance.
• Fraud detection from multi-modal content.
• Trading systems that incorporate sentiment signals.

Industry relevance. Because TEXT routes experts by text, it keeps the text modality dominant while still drawing on audio and video only when they help — a design that maps well onto production systems where text is the most reliable signal and audio/video are noisy. The reported MAE reduction on CH-SIMS (0.353) and the strong Chinese-dataset results also matter for markets where Chinese-language sentiment analysis is the deployment target, and the paper explicitly notes that GPT-4o's advantage fades on Chinese datasets.

Future Directions

• Eliminating cumulative error. The authors name this as their future work, since the pipeline depends on multiple MLLMs and errors from VideoLLaMA can propagate.
• Multilingual expansion. The paper currently covers Chinese and English; the authors plan to expand TEXT to use more MLLMs in multiple languages.
• Improving fine-grained scoring. The paper observes that KuDA and DEVA win on some fine-grained metrics (Acc-7 on MOSI and MOSEI, Acc-5 on CH-SIMSv2 and MOSEI) and conjectures that fine-grained scoring requires additional, possibly domain-specific, sentiment knowledge — an open problem TEXT does not solve.
• Hyperparameter and interpretability questions. The paper reports that three layers are optimal for SMoE on CH-SIMS, and suggests keyword-triggered expert activation may improve interpretability, leaving the scope of that benefit unquantified.

Target Audience

Researchers and graduate students working on multi-modal sentiment analysis, multi-modal fusion, and mixture-of-experts architectures; practitioners building emotion- or sentiment-aware systems in healthcare, human-computer interaction, finance, or fraud detection who need to combine text, audio, and video; and readers interested in how multi-modal LLM explanations can be repurposed as an alignment signal rather than only as auxiliary text. Readers should be comfortable with cross-attention, sparse experts, and LLM fine-tuning to follow the method sections.

Authors’ abstract

Human-interaction-involved applications underscore the need for Multi-modal Sentiment Analysis (MSA). Although many approaches have been proposed to address the subtle emotions in different modalities, the power of explanations and temporal alignments is still underexplored. Thus, this paper proposes the Text-routed sparse mixture-of-Experts model with eXplanation and Temporal alignment for MSA (TEXT). TEXT first augments explanations for MSA via Multi-modal Large Language Models (MLLM), and then novelly aligns the epresentations of audio and video through a temporality-oriented neural network block. TEXT aligns different modalities with explanations and facilitates a new text-routed sparse mixture-of-experts with gate fusion. Our temporal alignment block merges the benefits of Mamba and temporal cross-attention. As a result, TEXT achieves the best performance cross four datasets among all tested models, including three recently proposed approaches and three MLLMs. TEXT wins on at least four metrics out of all six metrics. For example, TEXT decreases the mean absolute error to 0.353 on the CH-SIMS dataset, which signifies a 13.5% decrement compared with recently proposed approaches.