EmoVerse: A MLLMs-Driven Emotion Representation Dataset for Interpretable Visual Emotion Analysis

AI summary

Overview

Research area: Computer Vision / affective computing — specifically Visual Emotion Analysis (VEA) and dataset construction for multimodal large language models.

Technical level: Advanced. The paper combines large-scale dataset curation, multi-model verification pipelines, and two-stage fine-tuning of a 3B-parameter vision-language model, plus linear probing of learned emotion embeddings.

Scope: The paper introduces EmoVerse, an open-source visual emotion dataset with over 219k images, dual categorical and dimensional emotion annotations, knowledge-graph-inspired Background–Attribute–Subject (B-A-S) triplets with subject-level grounding, an automated annotation and verification pipeline, and an interpretable model that produces continuous emotion representations with attribution explanations.

Note: the submitted metadata title differs from the title used inside the paper content ("EmoVerse: A Large-Scale MLLM-Powered Dataset for Explainable Visual Emotion Understanding"), and the content also lists Yongdong Zhang among the authors.

What This Paper Is About

Visual Emotion Analysis tries to connect what an image shows with how a person feels when viewing it, but progress is held back because most open datasets attach only one discrete emotion label to an entire image — so there is no way to tell which object, background, or visual attribute caused that emotion. EmoVerse addresses this by decomposing each image's emotional content into Background–Attribute–Subject (B-A-S) triplets, grounding elements to actual pixel regions, and labeling every image with both a discrete emotion category and a continuous, high-dimensional emotion embedding. The goal is a dataset, a reliable annotation pipeline, and a model that together make visual emotion analysis explainable rather than a black-box classification.

Key Contributions

1. EmoVerse dataset: Described as the first large-scale visual emotion dataset that offers high-dimensional Dimensional Emotion Space (DES) annotations together with fine-grained B-A-S triplets and object-level grounding. It contains over 219k images (218,522 reported in the analysis section) spanning social, artistic, natural, web-sourced, and AI-generated content.
2. Annotation and Verification Pipeline: A multi-stage pipeline combining Gemini 2.5, GPT-4o, EmoViT, and a Chain-of-Thought-based Critic Agent with human oversight, designed to produce consistent annotations with minimal human intervention.
3. Interpretable emotion model: A two-stage fine-tuning of Qwen2.5-VL-3B into a projector that maps visual cues into 1024-dimensional DES embeddings while producing detailed attribution explanations for emotions.
4. Unified CES + DES annotation schema: Every image receives Mikels' eight categorical emotion labels with confidence scores (amusement, awe, contentment, excitement, anger, disgust, fear, sadness) as well as continuous DES representations, plus bounding boxes and segmentation masks for the primary subject from Grounding DINO and the Segment Anything Model.

Main Findings

• Scale and balance: The dataset comprises over 218,522 finely annotated images — approximately 1.9 times larger than EmoSet (118,102 images) and over 9.4 times larger than the FI dataset (23,308 images). The paper states EmoVerse shows the minimum maximum difference (Δ) and variance (σ) across emotion categories compared to prior work.
• AIGC subset: Roughly 25,000 images were synthesized with the Seedream model using B-A-S triplets as seed prompts, accounting for 12.17% of the total dataset.
• Cross-dataset generalization: With ImageNet pretraining, a ResNet-50 trained on EmoVerse reaches 81.87% Top-1 accuracy on EmoSet and 72.14% on EmoVerse itself, versus 47.81% and 72.92% respectively for a model trained on EmoSet. Without pretraining, EmoVerse-trained models reach 73.50% on EmoSet and 68.97% on EmoVerse. The paper reports these as the highest cross-dataset recognition accuracy in both settings; evaluation subsampled 10,000 images per dataset across five random seeds.
• Pipeline ablation: Full-pipeline accuracy on the human-verified subset was 93.20% for emotion category, 90.56% for description, 96.16% for B-A-S triplets, 71.14% for emotion intensity, and 85.46% for bounding boxes. Removing cross-verification dropped B-A-S accuracy to 85.40% and removing the Critic Agent dropped it to 60.32%.
• Critic Agent error detection: On deliberately corrupted annotations the Critic Agent recalled 89.65% (emotion category), 97.27% (description), 85.78% (B-A-S triplet), and 78.42% (bounding box), but only 45.79% for emotion intensity — a limitation the authors attribute to the inherent subjectivity of intensity, which is why the agent only judges intensity at three discrete levels (high, medium, low).
• Fine-tuning gains (Table 5): Compared with base Qwen2.5-VL, the fine-tuned model improved BBox IoU from 74.87 to 79.24, BBox Center Distance from 93.06 to 94.31, F1 from 80.33 to 84.60, CLIP Score from 83.27 to 93.94, Emotion Accuracy from 41.20 to 73.43, and Intensity Accuracy from 86.12 to 91.20. (The paper's prose describes the CLIP gain as +6.4%, which differs from the +10.67 shown in Table 5.)
• Attribute-based training matters: Under a linear classification head on frozen embeddings, attribute fine-tuning scored 73.74% accuracy, 77.86% precision, 75.74% recall, and 76.21% F1, beating CES-only fine-tuning (67.37 / 72.20 / 69.80 / 70.72) and untrained Qwen2.5-VL (55.35 / 62.64 / 56.29 / 58.26).
• DES structure: MDS visualization of the 1024-dimensional DES embeddings shows that training with full attribution produces the most compact and clearly separable clusters.
• User study: 50 participants across diverse academic backgrounds rated five groups of 50 randomly selected images each. EmoVerse scored highest on emotion arousal (82.41), CES accuracy (81.83), and annotation accuracy (86.41), compared with Flickr (72.67 / 66.25), Emotion6 (72.17 / 68.17), EmoSet (78.08 / 76.00), and EmoArt (64.17 / 64.25 / 82.17). Flickr, Emotion6, and EmoSet do not include contextual annotations, so their annotation accuracy is not reported.

Methodology in Plain English

The researchers built the dataset from three source types: filtered existing datasets (EmoSet as the emotional foundation, EmoArt for artistic styles, Flickr30k for natural images with captions), targeted web image collection using B-A-S-derived queries such as "joyful crowd at music festival," and AI-generated images produced by systematically swapping one or two elements inside existing B-A-S triplets.

Annotation happens in layers. Each image is decomposed into a Background, an Attribute, and a Subject — a minimal "emotional knowledge unit" that allows word-level supervision tied to specific image regions. Each image also gets a Mikels eight-category label with a confidence score, and the primary subject is localized with Grounding DINO plus SAM to produce bounding boxes and segmentation masks. Continuous emotion is represented as a 1024-dimensional DES embedding produced by the authors' own model.

Rather than trusting a single annotator, two visual language models (Gemini 2.5 and GPT-4o) label background context and sentiment and are compared against EmoViT, which the authors state is more accurate at sentiment labeling and therefore weighted more heavily. A Critic Agent then acts as an independent inspector, using chain-of-thought reasoning to check whether the scene description, background caption, and emotion label are mutually consistent, labeling each sample valid, revisable, or discarded and issuing correction instructions fed back into the next iteration. A subset of samples was checked by humans as a ground-truth reference.

For the model, Qwen2.5-VL-3B is fine-tuned in two rounds — first on attribute annotations, then on emotion category labels — using cross-entropy loss. Afterward the model is frozen and used as a feature interpreter: hidden states from the last four transformer layers are adaptively weighted, pooled, and projected down through learned matrices to form the DES embedding, followed by attention-based fusion and weighted pooling.

Why This Matters

Research impact: The paper directly targets three bottlenecks it identifies in VEA — limited scale and diversity, weak annotation reliability, and the absence of interpretable emotion grounding. If the reported cross-dataset transfer results hold, EmoVerse suggests that annotation quality and attribution supervision, not just dataset size, drive generalization: the size-matched EmoVerse-Cut subset still performs strongly, which the authors use to argue the gains come from annotation richness rather than scale.

Real-world applications:

• Emotion-aware image editing and emotion-controllable generation, which the paper names as a target use case.
• Retrieval and classification of emotionally salient images in large collections.
• Recommender and social media systems that need to infer affective response from user-generated visual content.
• Affective computing and human-computer interaction, where systems adapt to a user's emotional state.

Industry relevance: Companies working on content moderation, advertising creative selection, generative image tooling, and user-facing multimodal agents all need to reason about why an image feels a certain way, not just what it depicts. A dataset that supplies word-level and subject-level emotional attribution — plus an open pipeline that keeps human labeling effort low — is directly aimed at those production needs. The paper also ties the work to emotion alignment between generative models and human affect.

Future Directions

• Extend to multi-emotion scenarios: The current annotation assigns emotion at the image level; the authors explicitly plan to move toward settings where multiple emotions coexist.
• Add multimodal cues: Integrating modalities beyond static images is listed as future work.
• Emotion-controllable generation: A stated goal is to use the B-A-S framework and DES space to steer image generation by emotion.
• Improve subjective-attribute verification: The Critic Agent's weak recall on corrupted emotion intensity (45.79%) remains an open problem, tied to the difficulty of defining ground-truth intensity values.
• Validate the DES space more broadly: The paper demonstrates separability via MDS visualization and a linear classification head, but how well the 1024-dimensional embeddings transfer to other datasets and tasks is not reported.

Target Audience

Researchers and graduate students in computer vision, affective computing, and multimodal learning who need a large, interpretable emotion dataset or a reference pipeline for building one. It is also useful for practitioners who build emotion-aware image systems — search, recommendation, editing, or generation — and want to understand the tradeoffs between categorical and dimensional emotion labels. Readers need familiarity with vision-language models, dataset benchmarking, and emotion models to follow the training and evaluation details, though the core ideas about attribution and annotation verification are accessible without that background.

Authors’ abstract

Visual Emotion Analysis (VEA) aims to bridge the affective gap between visual content and human emotional responses. Despite its promise, progress in this field remains limited by the lack of open-source and interpretable datasets. Most existing studies assign a single discrete emotion label to an entire image, offering limited insight into how visual elements contribute to emotion. In this work, we introduce EmoVerse, a large-scale open-source dataset that enables interpretable visual emotion analysis through multi-layered, knowledge-graph-inspired annotations. By decomposing emotions into Background-Attribute-Subject (B-A-S) triplets and grounding each element to visual regions, EmoVerse provides word-level and subject-level emotional reasoning. With over 219k images, the dataset further includes dual annotations in Categorical Emotion States (CES) and Dimensional Emotion Space (DES), facilitating unified discrete and continuous emotion representation. A novel multi-stage pipeline ensures high annotation reliability with minimal human effort. Finally, we introduce an interpretable model that maps visual cues into DES representations and provides detailed attribution explanations. Together, the dataset, pipeline, and model form a comprehensive foundation for advancing explainable high-level emotion understanding.