Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoning

AI summary

Overview

Research area: Multimodal affective computing and multimodal large language models (MLLMs), framed through cognitive psychology (Theory of Mind).

Technical level: Intermediate. The psychology framing is accessible, but the method section assumes familiarity with supervised fine-tuning, chain-of-thought prompting, and group-relative policy optimization (GRPO).

Scope: The paper introduces a hierarchical, ToM-grounded benchmark (HitEmotion) for diagnosing where multimodal models fail at emotional reasoning, and a reinforcement-learning method (TMPO) that uses intermediate mental states as supervision to improve both accuracy and the faithfulness of reasoning.

What This Paper Is About

Current multimodal models can label obvious emotions but break down when a task requires inferring what someone believes, intends, or is pretending to feel. The authors argue this is because models are trained as shallow "emotional fact retrievers" rather than as simulators of mental states. They address both sides of the problem: a benchmark that organizes emotion tasks by cognitive depth so failure points can be located precisely, and a training method that explicitly rewards coherent mental-state reasoning.

Key Contributions

1. HitEmotion benchmark. A hierarchy of 24 emotion-related tasks (20,114 instances, video and image) organized into three cognitive levels: Emotion Perception and Recognition (EPR), Emotion Understanding and Analysis (EUA), and Emotion Cognition and Reasoning (ECR). Unlike prior affective benchmarks, it is grounded in Theory of Mind, includes gold reasoning chains, and evaluates model rationales, not just answers.

2. A ToM-guided reasoning chain. Structured three-tier prompting templates that walk a model through first-order mental state attribution, relational/contextual modeling, and finally causal plus second-order reasoning. The chains are wrapped in <think> and <answer> tags so deliberation and final output are learned separately.

3. TMPO (Theory-of-Mind Preference Optimization). A two-stage training pipeline: supervised fine-tuning on curated reasoning chains, then GRPO with a four-part reward mixing structure, content, process, and consistency signals.

4. Empirical evidence that ToM supervision works. The tuned model tops 16 of 24 tasks and beats proprietary baselines on the hardest cognitive tier, while producing more faithful and logically consistent rationales.

Main Findings

• Steep cognitive drop-off. Performance falls monotonically as task depth increases. At Level 1 (EPR), only three of ten tasks average above 60 accuracy. At Level 2 (EUA), only two exceed 60. At Level 3 (ECR), no task reaches an average of 60 across models.

• Closed-source models lead. GPT and Gemini series consistently outperform open-source models, with Gemini-2.5-Pro best on many perception tasks (e.g., 78.39 on FESD) partly due to natively handling video and audio. Fine-tuned open models such as Emotion-LLaMA-7B can match or beat untuned baselines on specific tasks, but a gap remains.

• ToM prompting helps strong models, harms weak ones. As a pure prompting strategy, ToM chains boost GPT-4.1, GPT-4o, and Gemini substantially—up to +16.52 on Sentiment Flip Analysis for GPT-4o and +11.33 on Emotion Interpretation for GPT-4.1. Weaker open models such as VideoLLaMA3-7B drop from 61.78 to 54.18 on FESD, suggesting insufficient representational capacity causes hallucinated reasoning chains.

• TMPO produces the largest gains. Adding GRPO on top of SFT yields consistent improvements across every task category. The full model reaches 77.12 on FESD, 67.63 on ISA, 73.13 on EER, and 72.45 on Laughter Reasoning, surpassing strong proprietary systems on cognitively demanding tasks.

• Reward components are complementary. Ablations show content reward produces the biggest single jump, consistency reward removes logical fallacies, process reward encourages ToM-specific terminology, and structure reward is a prerequisite—removing it causes "format collapse" where the model stops emitting parseable <answer> tags.

• Both ToM design choices matter. Generic "think step-by-step" prompting underperforms a cognitive-path prompt without terminology (ToM-Init), and explicitly adding ToM keywords (ToM-Full) beats both.

• Perception-focused tasks remain a weakness. On direct tasks like facial expression inference, TMPO lags leading systems—the authors attribute this to limits in the base model's raw multimodal perception, which reasoning-focused training cannot fix.

Methodology in Plain English

The researchers first assembled 24 public affective datasets and reorganized them into a single closed-label question-answering format across three difficulty tiers. They kept only official test splits to avoid leakage and had annotators cross-check a third of the data.

Because no existing dataset contained step-by-step mental-state reasoning, they generated reasoning chains with a four-step pipeline: LLM generation, filtering, enhancement, and correction—treating these chains as gold supervision.

Training runs in two stages on Qwen2.5-Omni-7B. In stage one, the model is fine-tuned to output a <think> block containing the reasoning chain followed by an <answer> block containing the prediction. In stage two, the model samples several candidate outputs per prompt, and GRPO updates the policy based on how each candidate scores on four rewards: correct formatting, answer correctness, use of ToM vocabulary, and internal logical consistency. A KL penalty keeps the tuned model close to the SFT reference so it does not drift.

Why This Matters

The work reframes emotional AI from classification toward mental-state simulation, connecting affective computing to decades of developmental psychology on how humans reason about minds. It also shows that process-level rewards derived from intermediate mental states—not just final-answer correctness—are a viable supervision signal for social reasoning.

Real-world applications:

• Mental health and counseling support. Systems that track a user's beliefs, intentions, and emotional trajectory rather than responding to isolated emotional cues.
• Human-robot and virtual agent interaction. Assistants that can detect sarcasm, implicit intent, or emotional shifts in video calls.
• Content moderation and misinformation analysis. Detecting persuasion techniques in memes and identifying when a speaker's literal words conflict with their intended meaning.
• Customer experience analytics. Reviewing video or audio interactions where tone, facial expression, and text diverge, which existing sentiment tools miss.

Industry relevance: The benchmark provides a diagnostic suite for teams building affective capabilities into multimodal products, and TMPO offers a reproducible recipe for improving open-weight models enough to close much of the gap with proprietary APIs on hard social-reasoning tasks.

Future Directions

• Extending to broader social intelligence. The authors note that ToM-style training suits information-asymmetric domains such as negotiation and intent analysis, where objective answers are rare, but leave this extension untested.
• Fixing the perception bottleneck. TMPO cannot compensate for weak raw multimodal perception on tasks like facial expression recognition; combining it with stronger perceptual backbones is an open problem.
• Calibrating ToM prompting for weaker models. Since ToM prompts help strong models and hurt weak ones, understanding when to deploy these scaffolds—or how to make them safe for smaller models—remains unresolved.
• Scaling the reasoning-chain curation pipeline. The gold chains were built with LLM assistance plus filtering and correction; whether this scales to noisier, more open-ended, or multilingual settings is unclear.

Target Audience

Researchers and engineers working on multimodal LLMs, affective computing, or social reasoning benchmarks. It is also useful for practitioners who want a concrete template for combining structured prompting with process-level reinforcement learning, and for cognitive scientists interested in how Theory of Mind concepts are being operationalized for machine evaluation. Readers without prior exposure to GRPO or multimodal fine-tuning will need background reading to follow the training section.

Authors’ abstract

Despite rapid progress in multimodal large language models (MLLMs), their capability for deep emotional understanding remains limited. We argue that genuine affective intelligence requires explicit modeling of Theory of Mind (ToM), the cognitive substrate from which emotions arise. To this end, we introduce HitEmotion, a ToM-grounded hierarchical benchmark that diagnoses capability breakpoints across increasing levels of cognitive depth. Second, we propose a ToM-guided reasoning chain that tracks mental states and calibrates cross-modal evidence to achieve faithful emotional reasoning. We further introduce TMPO, a reinforcement learning method that uses intermediate mental states as process-level supervision to guide and strengthen model reasoning. Extensive experiments show that HitEmotion exposes deep emotional reasoning deficits in state-of-the-art models, especially on cognitively demanding tasks. In evaluation, the ToM-guided reasoning chain and TMPO improve end-task accuracy and yield more faithful, more coherent rationales. In conclusion, our work provides the research community with a practical toolkit for evaluating and enhancing the cognition-based emotional understanding capabilities of MLLMs. Our dataset and code are available at: https://HitEmotion.github.io/.