VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models

AI summary

Overview

Research area: Computer vision / affective computing, specifically emotion understanding from video using video large language models (VideoLLMs).

Technical level: Advanced. The paper assumes familiarity with multimodal LLM training pipelines, reinforcement learning with GRPO, reward modeling, and tree-edit-distance metrics.

Scope: The paper introduces VidEmo, a family of emotion-centric video foundation models trained with a two-stage recipe (curriculum learning then tree-structured RL), plus Emo-CFG, a 2.1M-sample instruction dataset for fine-grained facial attribute, expression, and emotion analysis.

What This Paper Is About

Understanding emotion in video is hard because emotions are dynamic and depend on context, so models must combine low-level facial attributes into high-level emotional judgments while producing a rationale a human can follow. Existing VideoLLMs struggle at this: even Gemini 2.0 reaches only 26.3% accuracy on fine-grained sentiment analysis. The paper's goal is to build a model family that unifies basic attribute perception, expression analysis, and high-level emotion understanding in a staged, explainable way, backed by a large purpose-built dataset.

Key Contributions

1. VidEmo, an affective cues-guided reasoning framework. It combines curriculum emotion learning during pre-training with affective-tree reinforcement learning during post-training, producing fine-grained and interpretable emotion inference from dynamic video. The paper reports over 16.3% and 14.2% improvement over existing open-source VideoLLMs across 15 facial perception tasks at the 1-3B and 7-8B scales, respectively.

2. Emo-CFG, an emotion-centric fine-grained dataset. A large-scale resource of 2.1M diverse instruction-based samples with annotations spanning attributes, expressions, and emotions, plus explainable emotional question-answering, fine-grained captions, and associated rationales.

3. A three-level affective-tree reward for RL. Generated captions are parsed into an attribute → expression → emotion tree, compared to a human-annotated ground-truth tree using tree edit distance, and scored with an exponential decay reward R = exp(−λ · Edit(T_gt, T_pred)).

4. A hierarchical, search-based inference procedure. At inference the policy model samples candidates at each of the three levels (attribute perception, expression analysis, emotion inference) and selects the best via a reward-guided scoring mechanism, forming a bottom-up reasoning trajectory.

Main Findings

• Broad benchmark leadership: VidEmo-T1 is reported as superior across 15 face perception tasks, surpassing Gemini 2.0 (dated 5th Feb, 2025) on 14 of 15 tasks.
• Scale-wise gains: At the 1-3B and 7-8B scales, VidEmo-Base (3B/7B) reaches an overall average accuracy of 62.4% / 64.1%, versus the strongest baseline Qwen2.5-VL at 46.1% / 51.7%, a margin of +16.3% / +12.4%.
• Attribute perception: VidEmo averages 86.3%, above Qwen2.5-VL (7B) at 80.6% (+5.7%). Individual results include 99.7% on identity verification, 95.6% on facial shape, and 97.0% on gender prediction.
• Expression analysis: VidEmo averages 39.9%, outperforming Qwen2.5-VL (7B) by +6.8%, with reported leads in fine-grained expression classification (35.6% vs. 29.7%) and micro-expression detection (20.4% vs. 13.6%).
• Emotion understanding: VidEmo averages 69.3%, ahead of Gemini 2.0 (63.5%) and Qwen2.5-VL 7B (63.6%), with new benchmarks on instruction adherence (68.1%), fluency (69.1%), and video-text relevance (69.3%).
• Against closed models: VidEmo-T1 (7B) reaches 86.3% in attribute perception (Gemini 2.0 +9.8% behind), 39.9% on expression tasks (Claude 3 +16.6% behind), and 69.3% in emotion understanding (GPT-4o at 48.0%, a +21.3% margin).
• Effect of the reasoning stage: Moving from VidEmo-Base (7B) to VidEmo-T1 raises attribute perception from 84.7% to 86.3%, expression analysis from 39.1% to 41.3%, and emotion understanding from 67.5% to 69.3%. Head pose estimation improves +1.9% (93.4 to 96.7), facial feature prediction +1.3% (85.6 to 86.9), gender recognition +1.9% (89.5 to 90.4), micro-expression detection +2.2%, fine-grained expression recognition +1.0%, fluency +2.3% (67.9 to 70.2), video-text relevance +2.6%, and instruction adherence +1.9%.
• Downstream fine-tuning: On DFEW, VidEmo achieves UAR 64.92 and WAR 73.10; on MAFW, UAR 44.02 and WAR 54.86, versus EMO-LLaMA at 60.23 / 65.89 and 41.57 / 48.63. The paper reports an average improvement of 9.4% over the previous best, with relative gains of 7.8% UAR and 10.9% WAR on DFEW and 5.9% UAR and 12.8% WAR on MAFW.
• Ablation results: Removing all components gives an average of 51.4; adding curriculum emotion learning alone gives 61.9; adding the affective-tree reward gives 63.6; the full configuration with emotion reasoning gives 67.0 (Att 84.5, Exp 43.8, Emo 72.9).
• Dataset quality check: In a user study comparing Emo-CFG against CelebV-Text (described as the largest human-labeled video emotion dataset), preference rates were 95.5% for precision, 92.1% for rationality, and 93.0% for complementarity, all statistically significant (Wilcoxon signed-rank test, p < 0.01), based on 50 videos and 25 users.
• Task-difficulty pattern: Attribute tasks show the highest average performance. The authors note this aligns with training dynamics, where model perplexity increases in the sequential order of attribute, expression, then emotion.

Methodology in Plain English

The approach starts by building a dataset rather than a model. Emo-CFG is assembled from 17 source datasets covering head, avatar, and full-body avatar video. For already-labeled sources, GPT-4o generates instruction pairs such as multiple-choice questions, open-ended questions, and short captions. For unlabeled video, the pipeline applies a causal affective reasoning strategy: Gemini 2.0 first produces fine-grained captions describing attributes, then expressions, then emotions in sequence, and GPT-4o turns those into question-answer pairs. A separate step prompts a VideoLLM to self-reflect and produce rationales linking low-level attributes to high-level emotions. Because emotion labels are subjective, three different VideoLLMs act as a committee that votes on correctness, returning "critic" items with corrections; verified data is kept and the rest is rewritten.

Training then proceeds in two stages. First, curriculum emotion learning tunes the base model in three ordered steps—attribute tuning, expression tuning, emotion tuning—so simpler perceptual skills are in place before harder emotional judgments. Second, post-training applies GRPO (a reinforcement learning method that samples a group of outputs per query and optimizes relative rewards with a KL penalty against a reference policy). Three reward signals are mixed: a rule-based QA reward using accuracy and F1 over classification and action-unit tasks, a model-based caption reward for short action/appearance/emotion captions, and the affective-tree fine-grained caption reward. That last reward parses a generated caption into a three-level tree (attribute A → rationale → expression E → rationale → emotion M) and compares it to a human-annotated tree via tree edit distance, converting distance into a reward through exponential decay.

At inference, the model can use a hierarchical search: sample several candidate outputs at each of the three levels, score them with the reward mechanism, and carry the best forward. The authors note they disable emotion reasoning (ER) when comparing against other state-of-the-art methods so that only one model response is sampled, keeping the comparison fair.

Why This Matters

Impact on research. The paper argues that emotion recognition has been limited to a small number of categories and static attribute perception, and proposes a way to insert structured, explainable reasoning into the emotion path of a VideoLLM. It also contributes a 2.1M-sample dataset with rationales and fine-grained captions, positioning Emo-CFG as a foundational data infrastructure for emotion-centric video analysis. The reported benchmark spans 40 metrics over 6 closed-set attribute tasks, 12 open-set attribute tasks, 9 expression tasks, and 6 high-level emotion understanding tasks, giving the field a broader evaluation surface than single-label emotion classification.

Real-world applications (as named in the paper):

• Human-computer interaction
• Surveillance
• Healthcare
• Any setting requiring fine-grained, explainable emotional captions with rationale (demonstrated in the paper's qualitative examples of attribute perception, expression analysis, and emotion understanding)

Industry relevance. The work comes from a collaboration between Nankai University, Pengcheng Laboratory, and Kuaishou Technology, indicating direct interest from short-video and social-media platforms where emotion-centric automatic captioning and content understanding matter. The models are built on Qwen2.5-VL backbones at 3B and 7B, sizes that are practically deployable.

Future Directions

• Reduce counterfactual generation. The authors list susceptibility to counterfactual content and emotionally inconsistent descriptions as a key limitation.
• Add more modalities. Because emotion understanding is inherently multimodal, integrating audio or broader contextual cues could enrich affective reasoning; the authors position VidEmo as a foundation for that direction.
• Scale and refine the reasoning stage. The gap between VidEmo-Base and VidEmo-T1 suggests further gains from stronger affective-tree rewards or more search at inference; the paper does not report results beyond the T1 configuration.
• Extend the data pipeline. The pipeline relies on Gemini 2.0 and GPT-4o for labeling and a three-model committee for verification; whether these dependencies can be replaced or extended to other languages and domains is not reported.

Target Audience

This paper is most useful for researchers and engineers working on multimodal large language models, affective computing, and video understanding, particularly those building emotion-aware systems or needing an instruction-tuning dataset with rationale annotations. It also suits practitioners who want a concrete recipe for combining curriculum learning with reinforcement learning rewards in a multimodal model, and benchmark designers looking for a broad set of face perception and emotion evaluation tasks. Readers without background in RL fine-tuning or VideoLLM architectures will find the methodological sections demanding.

Authors’ abstract

Understanding and predicting emotion from videos has gathered significant attention in recent studies, driven by advancements in video large language models (VideoLLMs). While advanced methods have made progress in video emotion analysis, the intrinsic nature of emotions poses significant challenges. Emotions are characterized by dynamic and cues-dependent properties, making it difficult to understand complex and evolving emotional states with reasonable rationale. To tackle these challenges, we propose a novel affective cues-guided reasoning framework that unifies fundamental attribute perception, expression analysis, and high-level emotional understanding in a stage-wise manner. At the core of our approach is a family of video emotion foundation models (VidEmo), specifically designed for emotion reasoning and instruction-following. These models undergo a two-stage tuning process: first, curriculum emotion learning for injecting emotion knowledge, followed by affective-tree reinforcement learning for emotion reasoning. Moreover, we establish a foundational data infrastructure and introduce a emotion-centric fine-grained dataset (Emo-CFG) consisting of 2.1M diverse instruction-based samples. Emo-CFG includes explainable emotional question-answering, fine-grained captions, and associated rationales, providing essential resources for advancing emotion understanding tasks. Experimental results demonstrate that our approach achieves competitive performance, setting a new milestone across 15 face perception tasks.