SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

AI summary

Overview

• Research area: Multimodal Sentiment Analysis (MSA) — predicting human sentiment from language, visual, and acoustic signals when those signals are partially missing, with a focus on using Large Language Models (LLMs) as a source of high-level semantics. The paper is listed under Natural Language Processing (arXiv:2609.30238v1 [cs.CL], 24 Sep 2026).
• Technical level: Advanced. The work assumes familiarity with transformer adapters, contrastive/alignment losses, and eigendecomposition of kernel Gram matrices.
• Scope (1 sentence): The paper proposes SemMSA, a framework that generates compact latent sentiment semantics with a frozen LLM and aligns them with all modalities via an anchor-free spectral objective, evaluated on MOSI, MOSEI, and SIMS under missing rates from 0.0 to 0.9.

What This Paper Is About

Multimodal sentiment models are usually trained assuming language, vision, and audio are fully observed, but in real deployments parts of each modality are corrupted or missing — and performance collapses when that happens. Existing fixes either try to reconstruct the missing features (which the authors say recovers low-level patterns and can hallucinate sentiment-contradicting content) or fuse whatever remains (which can overfit modality co-occurrence and become unreliable under severe missingness). SemMSA instead compensates at the semantic level: it uses a frozen LLM to iteratively produce continuous sentiment-relevant latent states, then aligns semantics, language, visual, and acoustic representations together without choosing any single modality as an anchor.

Key Contributions

1. SemMSA framework — a latent semantic-aided framework that provides high-level semantic compensation with LLMs and jointly aligns heterogeneous representations via an anchor-free spectral objective. The authors state this is "the first attempt to utilize LLMs to perform semantic-level compensation for MSA."

2. Cross-modal Semantic Refinement (CSR) — lightweight visual and acoustic adapters with learnable prompts and self-/cross-attention map nonverbal features into the frozen LLM embedding space, after which CSR iteratively refines sentiment-relevant semantic states in a token-efficient latent process without decoding explicit text.

3. Cross-modal Spectral Alignment (CSA) — builds a kernel Gram matrix over the normalized semantic, language, visual, and acoustic representations and enhances its dominant spectral component, capturing global nonlinear relationships without anchor-modality dependency; an instance-level spectral separation constraint preserves cross-sample discriminability and mitigates representation collapse.

4. State-of-the-art results across missingness settings — the authors report consistent state-of-the-art performance on MOSI, MOSEI, and SIMS under diverse missingness, with an average accuracy improvement of 1.4%.

Main Findings

• Intra-modal missingness (MOSI): SemMSA reaches Acc-2 of 74.36/73.91, F1 of 74.18/73.82, Acc-5 of 40.93, Acc-7 of 36.49, MAE of 1.011, and Corr of 0.550. The paper reports improvements over the best baseline TF-Mamba of 2.43% on Acc-5 and 2.23% on Acc-7, and the lowest MAE at 1.011.

• Intra-modal missingness (MOSEI): SemMSA obtains Acc-2 of 79.61/79.38, F1 of 80.62/79.87, Acc-5 of 48.12, Acc-7 of 47.06, MAE of 0.648, and Corr of 0.601, which the authors present as evidence of scalability to larger-scale data.

• Intra-modal missingness (SIMS): SemMSA achieves Acc-2 of 75.46, F1 of 77.50, Acc-3 of 58.84, Acc-5 of 35.68, MAE of 0.474, and Corr of 0.501, surpassing P-RMF by 0.087 in Corr.

• Stability as missingness grows: Figure 3 shows F1, MAE, Acc-2, and Corr across missing rates from 0.0 to 0.9. Most baselines degrade clearly as the missing rate increases, while SemMSA remains consistently superior and more stable. Detailed results at each individual missing rate are said to be in the Appendix.

• Inter-modal missingness (MOSEI, F1): SemMSA attains the best result in every condition — 83.61 for {l}, 66.59 for {a}, 65.91 for {v}, 85.17 for {l,a}, 84.96 for {l,v}, 72.68 for {a,v}, and 86.32 for {l,a,v} — giving an average of 77.89 versus 75.18 for the second-best method, CorrKD, a margin of 2.71% on average. The paper notes that all methods degrade in unimodal settings and that language consistently outperforms visual and acoustic modalities.

• Ablation — CSR is the largest single gain: Adding CSR alone improves Acc-2 and F1 by 5.68% and 11.11% on average over the baseline. Adding CSA on top raises MOSI Acc-5 from 38.84 to 40.48 and Acc-7 from 35.03 to 36.08. The full model (CSR + L_csa + L_sep) is best on both MOSI and SIMS, which the authors attribute to the complementarity of intra-instance cross-modal coherence (L_csa) and inter-instance discriminability (L_sep).

• Adapter design beats heavier projections: On SIMS, a linear projection gives Acc-2 72.84 and F1 73.62, MLP gives 73.58 and 74.36, a standard Transformer (~85M parameters) gives 74.52 and 76.21, and Q-Former (~100M) gives 74.83 and 76.68, versus SemMSA's 75.46 and 77.50. The authors' own adapter uses only 4.7M parameters.

• Spectral alignment beats pairwise alignment: On SIMS, InfoNCE reaches Acc-2 73.42 / F1 75.39, CMD 73.77 / 75.18, PMRL 74.63 / 75.98, and volume-based loss 74.31 / 76.35, all below SemMSA's 75.46 / 77.50. InfoNCE and CMD are described as failing to model joint structure across all three modalities, while PMRL and volume-based losses remain limited to linear space and anchor modality.

• Efficiency on MOSI: SemMSA has 113M trainable parameters, 4.8 task GFLOPs, 13.0 GB memory, 30.6 ms latency, Acc-2 74.36/73.91 and F1 74.18/73.82, compared with LNLN (116M, 9.0 GFLOPs, 12.8 GB, 25.1 ms) and P-RMF (117M, 9.7 GFLOPs, 13.5 GB, 67.0 ms). The paper reports a 50.5% reduction in task GFLOPs versus LNLN and a 54.3% reduction in inference time versus P-RMF.

• Confusion matrices (MOSI, Figure 4): Under missing rates of 0.1, 0.5, 0.7, and 0.9, LNLN tends to concentrate predictions around middle sentiment classes, whereas SemMSA keeps clearer diagonal patterns across the seven classes (0-6 denoting strongly negative through strongly positive).

• LLMs work better as semantic generators than as direct predictors: Using Qwen2.5-Omni-7B for direct prediction yields only 69.84/71.36 Acc-2 and 69.92/71.58 F1 on MOSI, while using LLMs for semantics yields clear gains. Qwen3-8B achieves the best F1 and Acc-3 on SIMS, Qwen2.5-Omni-7B gives superior Acc-5 and MAE on MOSI, and Llama3.1-8B lacks Chinese support so SIMS results are not reported for it.

Methodology in Plain English

The pipeline has three stages.

1. Encode what is available. Each modality (vision, audio, language) is passed through a frozen modality-specific encoder. Missing visual and acoustic segments are replaced with zero vectors and missing language tokens with [UNK]. Training applies instance-wise Bernoulli masking, with 50% of samples kept complete.

2. Turn nonverbal signals into language-model tokens. Small visual and audio adapters, each with M = 8 learnable prompt embeddings updated by a lightweight transformer with B = 2 blocks, use cross-attention to pull evidence out of the visual or audio sequence and self-attention to capture intra-modality dependencies. Learnable positional embeddings are injected because the frozen encoder features carry no temporal order. The adapter outputs are linearly projected into the LLM's embedding dimension and concatenated in a fixed order with the embedded language tokens, forming a "multimodal prefix" in the frozen LLM's space.

3. Refine semantics in latent space, then align spectrally. Rather than generating text, CSR reads the LLM's last-layer hidden state at the final position, treats it as a new continuous token, appends it to the prefix, and repeats this O = 4 times. This yields a short sequence of latent semantic states that are pooled and projected into a compact sentiment-aware representation. Meanwhile, CSA normalizes the semantic, visual, acoustic, and language representations, maps them through an RBF kernel into a Reproducing Kernel Hilbert Space, and builds a 4×4 kernel Gram matrix. Eigendecomposition gives eigenvalues that are treated as logits; a softmax objective pushes the largest eigenvalue to dominate, meaning all four representations concentrate along one shared nonlinear direction. Because no modality is designated as the anchor, the objective does not collapse when one modality is severely missing. A separation loss takes each instance's dominant eigenvector, projects it back into representation space, and penalizes similarity between different instances' dominant directions so instances do not all collapse to one point.

The four aligned representations are summed element-wise, passed through a linear classifier for the sentiment score, and the total loss combines mean squared error with the spectral alignment and separation terms. Implementation settings: 200 epochs, batch size 64, AdamW with learning rate 1×10⁻⁴ plus warm-up, cosine annealing, and early stopping; input sequence length T = 8; hidden dimension d = 128; frozen Qwen3-1.7B as the LLM; RBF bandwidth σ = 1.0; temperature τ = 0.1; all experiments on an NVIDIA RTX 6000 Ada. Results are averaged over random seeds 1111, 1112, and 1113. MOSI and MOSEI are annotated with sentiment scores in [-3, +3]; SIMS is a Chinese dataset annotated in [-1, +1]. The paper says detailed dataset statistics appear in the Appendix, which is not included in the provided content.

Why This Matters

Research impact. The paper reframes missing-modality MSA from a feature-recovery problem to a semantic-compensation problem, arguing that reconstructing low-level patterns can hallucinate features that contradict the actual sentiment evidence. It also argues that pairwise contrastive alignment to a single anchor is structurally limited, and offers a spectral, anchor-free alternative. If the reported gains hold, this points toward latent-space LLM use — avoiding costly autoregressive text decoding — as a practical design pattern for robust multimodal learning.

Real-world applications.

• Social media and review monitoring: Sentiment at scale from user videos and posts where audio drops out, frames are occluded, or text is truncated.
• Customer-service and call-center analytics: Assessing caller affect when parts of the audio stream are corrupted or the transcript is incomplete.
• Human-computer interaction and virtual agents: Adapting to user emotion when a camera is blocked, a microphone fails, or speech recognition misses tokens.
• Clinical or wellbeing monitoring: Tracking affective state from multimodal recordings where sensor dropout is common — though the paper reports no clinical validation.

Industry relevance. The efficiency table matters for deployment: SemMSA reports 4.8 task GFLOPs versus 9.0 for LNLN and 9.7 for P-RMF, and 30.6 ms latency versus 67.0 ms for P-RMF, at 113M trainable parameters. The paper notes that reported memory and latency cover the full pipeline including the frozen LLM, not just trainable modules. Using a frozen LLM (Qwen3-1.7B) with only lightweight adapters and a small number of appended latent tokens also means the expensive model is not fine-tuned. The LLM comparison suggests organizations can swap in different frozen backbones — Qwen3-1.7B, Qwen3-8B, Llama3.1-8B, Qwen2.5-Omni-7B — and still obtain strong results, which reduces lock-in to one vendor's model.

Future Directions

• Push missingness further and beyond the tested regimes. Evaluation covers missing rates 0.0 to 0.9 with positions sampled independently per modality; how the spectral alignment behaves when missingness is structured, correlated across modalities, or adversarially placed is not reported.
• Broaden modality and task coverage. The framework is defined for exactly three modalities (vision, audio, language) and a 4×4 Gram matrix. Extending the spectral objective to more modalities, or to modality sets where entire streams were never available at training time, is an open question.
• Reduce dependence on LLM choice and language coverage. Llama3.1-8B produced no SIMS results because it lacks Chinese support, and the authors report that LLM choice shifts which metric is best (Qwen3-8B on SIMS F1/Acc-3, Qwen2.5-Omni-7B on MOSI Acc-5/MAE). A systematic study of how backbone semantics transfer across languages and domains is not provided.
• Explain the latent semantics. CSR deliberately avoids decoding text, so the sentiment-relevant content of the refined latent states is not human-readable. Whether these states can be interpreted, audited, or shown to avoid the "spurious generation" the paper criticizes is left open — the theoretical analysis in the appendix is truncated at the preliminaries in the provided content.

Target Audience

Researchers and engineers working on multimodal learning, affective computing, and robust NLP under missing data; practitioners deploying sentiment or emotion systems where sensor dropout is routine; and readers interested in using frozen LLMs as semantic feature generators rather than as end-task predictors. The paper is best suited to readers comfortable with kernel methods, eigendecomposition, and adapter-based LLM conditioning.

Authors’ abstract

Recent research on Multimodal Sentiment Analysis (MSA) has focused on learning from language, visual, and acoustic modalities with incomplete data to infer human sentiment. Most studies typically compensate for missing information by reconstructing modality features or designing complicated fusion mechanisms. However, these methods still suffer from spurious generation and noisy guidance due to the lack of high-level semantic grounding in partially observed multimodal evidence. To address these issues, we propose SemMSA, a latent semantic-aided framework that constructs rich sentiment-relevant semantics with LLMs, fully integrating with all modalities via anchor-free spectral alignment. It mainly consists of Cross-modal Semantic Refinement (CSR) and Cross-modal Spectral Alignment (CSA). Specifically, CSR first adaptively extracts visual and acoustic representations by corresponding adapters to form a unified multimodal prefix with language in the frozen LLM embedding space. It then iteratively produces continuous discriminative semantic states through a token-efficient latent refinement process without decoding explicit text. Next, CSA simultaneously aligns the refined semantics with all modalities by enhancing the dominant spectral component of their kernel Gram matrix. This captures global nonlinear dependencies among all representations without relying on a predefined anchor modality. In addition, an instance-level spectral separation constraint preserves cross-sample discriminability and mitigates representation collapse. Extensive experiments on SIMS, MOSI, and MOSEI benchmarks demonstrate that SemMSA achieves state-of-the-art performance.