ArtSociety: Multi-Agent Multimodal Collaboration for Art Emotion Understanding

AI summary

Overview

• Research area: Computer Vision / affective computing, specifically multimodal large language models (MLLMs) and multi-agent systems applied to artwork emotion understanding.
• Technical level: Advanced. The paper assumes familiarity with vision transformers (DINOv2), LoRA fine-tuning, Mixture-of-Experts (MoE) backbones, chain-of-thought prompting, and ensemble/voting methods.
• Scope (1 sentence): The paper describes ArtSociety, a five-agent multimodal system with training-free coordination controllers that won the top public leaderboard position on the AffectiveArt 2026 Track-2 challenge by jointly predicting an artwork's 12-class emotion, binary valence/arousal, and five attribute-grounded descriptions.

What This Paper Is About

The AffectiveArt 2026 Grand Challenge asks a system to look at a painting and simultaneously produce three very different kinds of output: a fine-grained emotion label (one of 12 categories), binary valence and arousal judgments, and five short written analyses of formal qualities such as brushstroke, color, and composition. These sub-tasks pull in opposite directions — the emotion labels are extremely imbalanced (1549:1 head-to-tail ratio) and reward rare-class recall, while valence/arousal is already near-saturated at over 90% accuracy and description quality depends on open-ended language generation rather than classification accuracy.

The authors' goal is to show that a single monolithic model cannot optimize all three regimes at once, and that a coordinated society of heterogeneous specialist agents — with a router deciding which agents handle which sub-task — can beat any individual model.

Key Contributions

1. Agentic decomposition of a multi-output task. The paper formulates art emotion understanding as a society of multimodal experts with task-aware routing, matching each sub-task (hard long-tailed classification, saturated binary prediction, generative description) to a tailored agent configuration rather than forcing one model to do everything.

2. A rare-class-aware voting arbiter. The arbiter aggregates emotion votes across five heterogeneous agents using a lowered agreement threshold (τ_r = 2 instead of the implicit ≥3 for common classes) on a designated rare set, exploiting decorrelated error patterns to rescue tail-class recall without hurting majority accuracy.

3. A description-first reasoning agent (DESCRIBE → CLASSIFY). By forcing the model to write visual evidence about brushstroke, composition, color, line, and light before committing to an emotion label, the agent achieves a near-perfect description score of 0.9952 and near-perfect attribute specificity of 0.996.

4. An eleven-variant single-agent ablation isolating what actually matters. The study shows method-side and scale-side changes plateau around 0.76 Combined, that a 30B MoE trained on older data (0.7530) underperforms an 8B model trained on better data (0.7628), and that genuine scene-grounded chain-of-thought supervision is the only intervention that gives the rarest classes their first non-zero F1 scores.

Main Findings

• Final test-set score of 0.8870 Overall. The routed system achieves 0.7789 on Classification (driven by emotion accuracy of 0.796) and 0.9952 on Description over 1,000 official test artworks, exceeding every single agent it is built from.

• No single agent dominates all emotion classes. Per-class F1 analysis shows the vision-only DINOv2 agent is strongest on Contentment and Aroused, Qwen3-VL-8B leads on Calm, Annoyed, and Tired, and SceneCoT-30B wins mid-frequency classes (Sad, Bored, Excited) — the decorrelated errors that voting can exploit.

• Rare-class arbiter beats plain majority voting by 5.5 percentage points. A 3-way closed-source majority vote reaches 0.741 emotion accuracy; the 5-agent vote with the rare-class priority rule reaches 0.796.

• Prompt ordering matters more than fine-tuning for description. On the same GPT-5.4 backbone, switching from label-first to describe-first ordering raises the description score from 0.969 to 0.995 and attribute specificity from 0.963 to 0.996. The fine-tuned 30B open model scores only 0.769 on description with 0.656 specificity, a 0.226-point gap.

• Loss re-weighting and resampling fail under 1549:1 imbalance. Focal loss drops Combined to 0.7474; aggressive asymmetric resampling and image augmentation do not beat the Qwen3-VL-8B baseline (0.7611); high oversampling ratios cause several tail classes to collapse to F1 = 0.

• Data quality dominates model capacity. Replacing templated rationales with scene-grounded chain-of-thought targets (SceneCoT) lifts emotion macro-F1 and gives tail classes their first non-zero scores (Aroused 0→0.028, Annoyed 0→0.078, Tired 0→0.086). The 30B MoE only reaches the best single-agent score (0.7678) when paired with that supervision.

• Ensembling saturated tasks is not worth it. Replacing the routed valence/arousal prediction with a full five-agent vote changes results by ≤0.2 percentage points while adding latency, confirming that binary near-saturated axes gain no signal from voting.

• A data ceiling limits the rarest classes. Manual inspection found that for very rare classes (Glad, Tired, Annoyed) the free-text annotations are frequently semantically inconsistent with the assigned label, an artifact of automatic annotation. No method tried recovered a stable visual-to-label mapping for these classes.

Methodology in Plain English

Rather than training one large model to do everything, the authors assembled five different "experts" that make different kinds of mistakes, then wrote simple rules (no additional training) to combine them.

The five agents:

• A vision-only DINOv2 ViT-Giant model, fine-tuned with class weighting and balanced resampling — it has no language knowledge, so its errors are purely visual.
• A Qwen3-VL model fine-tuned with LoRA on scene-grounded chain-of-thought targets derived from the dataset's own human-curated fields. The final version is a 30B MoE variant called SceneCoT-30B.
• GPT-5.4 given a distribution prior and a reasoning prompt.
• Gemini 3.1, contributing predictions from a different model family.
• A second GPT-5.4 configuration specialized for descriptions.

Two training-free controllers do the coordination:

• A voting arbiter resolves the 12-way emotion label. Its key trick is asymmetry: for a small set of rare emotions (happy, aroused, glad, annoyed, tired), only two agents need to agree, whereas common classes implicitly require three. The intuition is that if two independent models both see a rare emotion, that agreement is strong evidence and should override a larger but lower-precision majority block. Ties are broken by a fixed priority order calibrated on validation macro-F1.
• A description-first reasoning agent handles valence, arousal, all five attribute descriptions, and the caption. Its prompt enforces a chain of thought where the output schema places attribute descriptions before any label, so the model must verbalize what it sees before naming what it feels. It follows Russell's circumplex quadrant-first ordering (valence → arousal → quadrant → emotion), then runs a post-hoc consistency check that auto-corrects any mismatch between the chosen emotion and its expected quadrant.

A router assigns sub-tasks based on difficulty: the hard long-tailed emotion task goes to the full five-agent ensemble; the near-saturated valence/arousal and the generative description tasks go to the single strongest reasoning agent, since voting on a saturated axis adds noise and a single coherent narrator produces better descriptions than a committee.

Post-processing repairs JSON, normalizes label variants ("contentment" → "content"), enforces the deterministic emotion-to-V/A mapping from Russell's circumplex, and fills empty fields as a fallback. Fewer than 2% of test responses required correction.

Why This Matters

Impact on research. The paper's central methodological claim is that when a benchmark bundles heterogeneous objectives, designing the society of agents and their routing can matter more than scaling any single model. This challenges the default assumption that bigger backbones win, and it reframes the challenge as a problem of collaboration and data quality. The finding that 30B parameters of MoE trained on old data lose to 8B parameters trained on better data is a concrete, reusable lesson for multimodal affective computing, where annotation quality is often the real bottleneck. The rare-class-aware arbiter also offers a training-free alternative to focal loss and class-balanced loss, which the authors show are too blunt at 1549:1 imbalance.

Real-world applications:

• Museum and gallery experiences: automated docents that describe why a painting feels calm or agitated, grounded in observable formal qualities rather than generic affect words.
• Art therapy and mental-health assessment: systematic emotion annotation of patient-created artwork at scale, supporting clinicians who currently rely on subjective interpretation.
• Content moderation and recommendation: classifying the emotional tone of visual media for platforms that need to route or filter artwork and imagery by affective content.
• Digital art archiving and cataloging: automatically enriching large museum collections with structured emotional metadata, attribute analyses, and searchable captions.
• Creative tools: giving artists or art students feedback on the emotional effect of their color, composition, and line choices.

Industry relevance. The architecture is a practical template for any company deploying multimodal AI on tasks with conflicting sub-objectives — the routing policy (ensemble for the hard task, single best agent for easy and generative tasks) directly reduces latency and inference cost without sacrificing quality. The result that prompt engineering beats supervised fine-tuning for grounded description also has direct cost implications: it suggests companies may get better affective descriptions from a carefully structured closed-source API prompt than from fine-tuning an open model, at least on this type of data. The code is publicly released at github.com/swordlidev/ArtSociety.

Future Directions

• Fixing the annotation data ceiling. The paper identifies that for the rarest classes (Glad, Tired, Annoyed) the free-text annotations are semantically inconsistent with the assigned labels, and no method recovered a stable visual-to-label mapping. Re-annotating these classes or building a principled method for detecting annotation noise is the most direct path to further gains.

• Learning the arbiter and router rather than hand-tuning them. Both controllers are training-free and use hand-calibrated thresholds (τ_r = 2) and a fixed tie-breaking priority order tuned on validation macro-F1. Whether these rules can be learned, or transfer to other long-tailed multimodal benchmarks without re-tuning, is an open question.

• Extending the framework to video, 3D, or music emotion understanding. The agentic society plus routing design is not specific to static paintings. Testing whether decorrelated error patterns across heterogeneous agents persist in temporal or cross-modal affective domains would test the generality of the diversity-over-capacity principle.

• Quantifying the latency-cost-quality trade-off more rigorously. The paper argues routing saves latency without sacrificing quality, but does not report wall-clock or token-cost figures. A systematic study of how many agents are actually needed per sub-task would clarify how far the ensemble can be pruned.

Target Audience

This paper benefits most readers working at the intersection of multimodal large language models, affective computing, and multi-agent system design. It is particularly relevant to:

• Researchers entering the AffectiveArt or broader visual emotion analysis challenges who want a strong baseline architecture and a clear map of which interventions work and which do not.
• Practitioners building production multimodal systems on tasks with conflicting sub-objectives (classification plus generation), who want a concrete example of task routing and ensemble design.
• Computer vision and NLP researchers interested in the "data quality beats model scale" argument, since the eleven-variant ablation provides unusually direct evidence for it.
• Art historians and digital humanities scholars curious about what current AI can and cannot reliably say about the emotional content of visual art.

The paper is written at an advanced level and assumes prior familiarity with transformer architectures, LoRA, and prompting techniques; readers without that background will find the ablation tables dense but the high-level argument accessible.

Authors’ abstract

The AffectiveArt Multidimensional Art Emotion Understanding task asks to jointly predict an artwork's fine-grained emotion (12 classes, 1549:1 head-to-tail ratio), binary valence/arousal, and five attribute-grounded descriptions -- sub-tasks that exhibit strong empirical trade-offs, so the single-model solutions we tried do not jointly optimize all of them well. We present ArtSociety, a multi-agent framework that assembles heterogeneous multimodal experts -- a DINOv2-Giant vision agent (A1), a scene-grounded CoT fine-tuned MLLM (A2), and three closed-source reasoning agents (A3-A5) -- and coordinates them with two training-free controllers: (i) a rare-class-aware voting arbiter that lowers the agreement threshold for tail emotions, exploiting decorrelated error patterns across agent families; and (ii) a description-first reasoning agent whose DESCRIBE-then-CLASSIFY chain of thought forces visual evidence before label commitment, yielding near-perfect grounded descriptions. A task-routing policy directs the hard emotion task to the full five-agent ensemble while assigning the near-saturated valence/arousal and generative description tasks to the single strongest reasoning agent. On the official test set (1,000 artworks), ArtSociety achieves an Overall Score of 0.8870 (Classification 0.7789, Description 0.9952). An eleven-variant ablation study reveals that, once method and scale saturate at around 0.76, the decisive gains come from agent collaboration and data-side supervision -- a 30B MoE model trained on older data does not outperform an 8B model trained on better data. Code is available at https://github.com/swordlidev/ArtSociety