AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization

AI summary

Overview

• Research area: Multimodal large language models (MLLMs) for audiovisual emotion understanding, with a focus on direct preference optimization (DPO) as an alignment technique.
• Technical level: Advanced. The paper involves multimodal DPO objectives, a text-prior debiasing regularizer, and detailed attention/log-likelihood analysis.
• Scope: The paper introduces a benchmark (EmoReAlM) for diagnosing reasoning and perception errors in audiovisual emotion models, and a preference optimization method (AVEm-DPO) that improves two reference MLLMs in zero-shot settings (note: the paper title uses "AVERE" while the method itself is named AVEm-DPO throughout the content).

What This Paper Is About

Audiovisual multimodal LLMs make two characteristic mistakes when reasoning about emotion: they ground predictions in irrelevant cues such as attire color or ambient noise (reasoning errors), and they fabricate cues that are not present in the input to justify an emotion (perception errors), often driven by text priors in the language backbone. The authors build EmoReAlM, a 4,000-question multiple-choice benchmark that isolates these failure modes, then propose AVEm-DPO, a preference optimization method that aligns model responses with both the audiovisual inputs and emotion-centric queries.

Key Contributions

1. EmoReAlM benchmark: 4,000 human-verified multiple-choice question–answer samples over 2,649 unique videos, covering Emotion Reasoning – Basic (audio and visual), Modality Agreement, and an Emotion Reasoning – Stress Test with sub-tasks for no hallucination, spurious cue–emotion associations, and emotion-relevant cue hallucination. The benchmark is used only as a test set.
2. AVEm-DPO: A multimodal DPO technique that constructs preferences over (i) responses exhibiting spurious associations or hallucinations and (ii) audiovisual input pairs guided by textual prompts.
3. Text Prior Debiasing (TPD): A regularization term that penalizes policy reward for responses explainable by text-only input, aimed at suppressing modality-specific cue hallucinations.
4. Extensive zero-shot evaluation: Across EmoReAlM and existing benchmarks (DFEW, RAVDESS, MER2023, EMER), with ablations, attention-redistribution analysis, adversarial robustness tests, and human verification of the automatically generated preference data.

Main Findings

• Relative improvement of 6–19%: AVEm-DPO improves reference baseline models by 6–19% relative performance in zero-shot settings on DFEW, RAVDESS, and EMER.
• EmoReAlM gains for the base model: On EmoReAlM, the base model improves from 69.2 to 77.9 (reasoning basic, audio accuracy), 85.3 to 92.5 (visual accuracy), 34.6 to 60.0 (modality agreement F1), 50.3 to 80.9 (audio stress test F1), and 59.9 to 94.6 (visual stress test F1).
• EmoReAlM gains for EmotionLLaMA⋆: With AVEm-DPO, EmotionLLaMA⋆ moves from 64.8 to 76.5, 84.9 to 89.9, 33.1 to 56.8, 46.7 to 75.4, and 63.2 to 91.7 on the same five columns.
• Existing benchmark numbers: The base model with AVEm-DPO reaches DFEW UAR 58.54 / WAR 64.24, RAVDESS UAR 58.66 / WAR 55.48, MER2023 F1 92.18, and EMER scores of 6.37 (clue), 7.08 (label), 7.09 (spurious), and 6.75 (hallucination).
• Baselines struggle on agreement and stress tests: Strong systems such as Qwen 2.5 Omni (76.8, 89.2) and VITA-1.5 (63.1, 84.3) perform well on basic reasoning tasks but are much weaker on modality agreement (33.3 and 30.2 F1) and stress tests.
• Hallucination is the more severe issue for top systems: For systems such as Qwen 2.5 Omni and VITA-1.5, hallucination emerges as a more severe problem than spurious cue–emotion associations; unlike findings in prior work (Sung-Bin et al., 2025), audio and visual hallucinations appear equally prevalent in emotion reasoning tasks.
• User evaluation on EMER (40 participants): The AVEm-DPO model was selected most often for emotion description (54.74%) and emotion-cue associations (43.35%), and least often for inconsistencies (4.67%), versus EmotionLLaMA at 1.89%, 11.53%, and 68.61% respectively.
• Every component matters (ablation): Removing PMP drops the hallucination stress-test score from 97.6 to 88.1; removing ERP drops spurious-association performance from 92.7 to 84.9; removing TPD drops hallucination performance from 97.6 to 77.8. Training-free contrastive decoding is substantially worse (61.7 stress test, 54.8 hallucination).
• Best negative sampling choices: Prompt-based modality preference worked best when the rejected audiovisual input was a sample with a different emotion from the chosen one, and using both emotion-relevant and video-relevant rejected responses outperformed using only one or a completely irrelevant response.
• Preference data validity: Human verification of 1,000 samples per response type by 90 participants found 912 majority-correct chosen responses, 895 for video-relevant rejections, and 856 for emotion-relevant rejections (with 967, 923, and 912 having one or more correct ratings).
• Attention and robustness effects: Attention over the relevant modality increases after AVEm-DPO, and changing an irrelevant modality's input produces negligible shifts in the log-likelihood distribution of correct responses under KDE analysis.

Methodology in Plain English

The authors first build a diagnostic test set. Starting from DFEW videos, they use an MLLM to write separate audio and visual captions, then have an LLM check whether either caption actually reflects an emotion (one of six basic emotions or neutral). Samples where neither caption yields a valid emotion are discarded. GPT-4o then generates multiple-choice questions for each task type. To avoid questions answerable from text alone, GPT-4o, Gemini-2.5, and Qwen-2.5 are asked to answer using question text only; any question all three get right is removed. Over 470 crowd participants on Prolific then verify the surviving samples.

For the training method, they collect preference pairs from MAFW and a subset of the MER2025 Track-1 train set, with Gemini-2.5 generating altered responses. Two kinds of preference are combined. The first is prompt-based modality preference: for a question about one modality, only that modality's input is corrupted in the rejected pair, so the model cannot lean on the other modality. The second is emotion-based response preference: each chosen response is contrasted against a video-relevant but emotion-irrelevant response and an emotion-relevant but hallucinated response, with weights summing to one. A third term, text prior debiasing, subtracts a reward for responses that a text-only language model would also produce, using the language backbone of the reference model with gradients stopped and LoRA modules attached so text-only ability is preserved. The final objective combines the response-level term with a weighted input-level term.

Evaluation covers EmoReAlM plus DFEW, RAVDESS, MER2023 (weighted F1) and EMER (GPT-based evaluation against ground truth for clue overlap, label overlap, spurious cue–emotion associations, and hallucinated cues). Reference models are EmotionLLaMA⋆ (retrained without subtitles) and the authors' own base model, which uses whisper-large-v3 as the audio encoder and LanguageBind as the video encoder. Baselines include VideoLLaMA 2, OLA, VITA-1.5, Qwen-2.5 Omni, MoSEAR, Naive-DPO, and a modified Vista-DPO.

Why This Matters

Impact on research. The paper separates two failure modes that are usually lumped together as "hallucination" — irrelevant-cue grounding and fabricated-cue justification — and supplies a reproducible, MCQA-based benchmark that needs no proprietary LLM at inference time. It also extends DPO from response-level preferences in vision-language models to modality-level input preferences in audiovisual models, and provides analysis of attention redistribution and robustness to adversarial modality input.

• Social AI agents: The paper motivates emotion understanding as a prerequisite for socially intelligent agents that generate tailored responses in human–machine interaction.
• Health and therapy: The introduction cites applications where responding appropriately to affective states can improve therapeutic alliance.
• Education: The paper cites education as a domain where appropriate responses to affective states can improve learning outcomes.
• Reducing unreliable emotion inferences: By targeting spurious cue–emotion associations and hallucinated cues, the method aims to make emotion-predicting systems more trustworthy in the settings above.

Industry relevance. The method can be applied to existing open-source audiovisual MLLMs (the paper trains two reference models), and the benchmark offers a fixed test set for comparing products or checkpoints without relying on GPT-based judges. The paper explicitly cautions against deployment of these models in sensitive real-world contexts and states the benchmark and models are intended strictly for academic research.

Future Directions

• Long video emotion understanding: The benchmark and training data derive from short videos of roughly 2–10 seconds; the authors identify long-video emotion understanding and reasoning as an open topic for future work.
• Reducing cultural bias: EmoReAlM inherits the emotion labels and potential cultural biases of DFEW, which the authors flag as a limitation.
• Improving weak emotion classes: The AVEm-DPO model performs poorly on recognition of disgust, an ambiguous emotion, which the authors attribute to the limited number of training samples for that class.
• Mitigating spurious audio cue–emotion associations: Sub-task analysis of the Emotion Reasoning – Stress Test shows remaining room for improvement specifically on spurious audio cue associations.

Target Audience

Researchers and engineers working on multimodal large language models, audiovisual emotion recognition, and preference optimization or alignment techniques. It is also relevant to practitioners who need transparent, LLM-judge-free evaluation protocols for emotion reasoning, and to those studying hallucinations in multimodal models beyond the object-level setting.

• Health and therapy: The introduction cites applications where responding appropriately to affective states can improve therapeutic alliance.
• Education: The paper cites education as a domain where appropriate responses to affective states can improve learning outcomes.
• Reducing unreliable emotion inferences: By targeting spurious cue–emotion associations and hallucinated cues, the method aims to make emotion-predicting systems more trustworthy in the settings above.

Authors’ abstract

Emotion understanding is essential for building socially intelligent agents. Although recent multimodal large language models have shown strong performance on this task, two key challenges remain - spurious associations between emotions and irrelevant audiovisual cues, and hallucinations of audiovisual cues driven by text priors in the language model backbone. To quantify and understand these issues, we introduce EmoReAlM, a benchmark designed to evaluate MLLMs for cue-emotion associations, hallucinations and modality agreement. We then propose AVEm-DPO, a preference optimization technique that aligns model responses with both audiovisual inputs and emotion-centric queries. Specifically, we construct preferences over responses exhibiting spurious associations or hallucinations, and audiovisual input pairs guided by textual prompts. We also include a regularization term that penalizes reliance on text priors, thereby mitigating modality-specific cue hallucinations. Experimental results on DFEW, RAVDESS and EMER demonstrate that our method significantly improves the performance of the reference baseline models with 6-19% of relative performance gains in zero-shot settings. By providing both a rigorous benchmark and a robust optimization framework, this work enables principled evaluation and improvement of MLLMs for emotion understanding and social AI. Code, models and benchmark will be released at https://avere-iclr.github.io.