An Agentic Approach with Evidence-Grounded Verification for Multimodal Emotion Reasoning in Conversation

AI summary

Overview

Research area: Multimodal emotion recognition in conversation, built on agentic pipelines of frozen multimodal large language models (MLLMs), with an emphasis on evidence traceability and prompt-space adaptation rather than weight updates.

Technical level: Intermediate. The paper assumes familiarity with MLLM prompting, retrieval-augmented generation, multi-agent decomposition, and standard classification metrics, but its central idea (blame the right component, then patch only that component's prompt) is explained in accessible terms.

Scope: The paper proposes and evaluates an agentic framework that decomposes multimodal emotion reasoning across modality-specialist agents, verifies wrong predictions against cited evidence, and converts attributed failures into validated prompt patches, tested on MER2025 Track 3 across four MLLMs from 2B to 38B parameters.

What This Paper Is About

Systems that read emotion from video must combine what a person says, how they sound, and how they look. MLLMs can do this, but improving them by instruction tuning is expensive, and pipelines assembled from frozen models are hard to debug: when the final emotion prediction is wrong, nothing in the output says which agent or stage caused the error, and correctness is usually judged by another model rather than checked against the actual signal. The authors build a pipeline where every affective claim carries a citation to a specific frame, time span, or transcript span, then add a separate verification layer that mechanically re-checks those citations on wrong predictions, assigns blame to the responsible component, and repairs that component through validated prompt edits only.

Key Contributions

1. An evidence-cited agentic pipeline for emotion reasoning. Three frozen perception agents (visual, acoustic, lexical) post observations to a shared evidence board, each claim paired with a citation—a frame identifier for visual, a time span for acoustic, a verbatim transcript span for lexical. An integration agent reconciles the board with retrieved multimodal precedent to produce a description, a label set, a confidence, and a citation list, with an inline MLLM critic allowed at most one revision request before the draft is finalized.

2. An evidence-grounded verification layer that audits only wrong predictions. Deterministic, non-LLM checks cover visual re-derivation (the cited frame exists and contains a detectable face), acoustic re-derivation (reported pitch, RMS energy, and speech rate must agree with recomputed values within a 15% relative tolerance), retrieval sanity, and citation checks (reference validity, hash consistency, citation resolution). A separate entailment check decomposes the draft into atomic claims and tests each against its cited evidence at a threshold of 0.5. Failures are attributed to the responsible stage rather than to another model's opinion.

3. Verifier-guided prompt-space reinforcement. Attributed failures are clustered by component and defect type, and a patch is synthesized only once at least 5 failures of the same type accumulate within a training round. Each candidate patch is inserted into an isolated copy of the affected agent's prompt and rerun on a held-out validation split, admitted only if validation F-score does not decrease, and otherwise discarded and retained as feedback for the next synthesis attempt. No model weights are updated anywhere in the loop.

4. A controlled comparison of repair routing strategies. Under the same patch budget, the paper compares targeted (patch routed only to the blamed component), uniform (same patch content given to all agents), and global (patches pooled into a single free-text reflection, in the style of Reflexion), plus static-pipeline, Direct, and Chain-of-Thought baselines, across four MLLMs.

Main Findings

• Targeted repair beats the frozen pipeline on every backbone. F-score improves by 0.6 to 2.2 points over the static pipeline depending on the model, including 52.3 to 54.5 on Qwen2.5-VL-7B. The deltas are 0.6 for VideoLLaMA3-2B, 1.4 for Phi-3.5-Vision, 2.2 for Qwen2.5-VL-7B, and 2.1 for InternVL2.5-38B.

• Global, undifferentiated reflection hurts. Broadcasting one undifferentiated revision to all agents scores below the static pipeline on all four models: 38.2 vs 39.0 (VideoLLaMA3-2B), 47.1 vs 47.6 (Phi-3.5-Vision), 52.2 vs 52.3 (Qwen2.5-VL-7B), and 58.7 vs 59.1 (InternVL2.5-38B).

• Targeted repair is best on three of four models and ties on the fourth. It achieves the highest F-score on Phi-3.5-Vision (49.0), Qwen2.5-VL-7B (54.5), and InternVL2.5-38B (61.2), and ties uniform reflection at 39.6 on VideoLLaMA3-2B. Precision is best on three of four models (43.0, 53.0, 58.9, 64.5), but on VideoLLaMA3-2B the single-call Direct baseline has the highest precision (45.1 vs 43.0).

• Repair benefit scales with backbone size. The gain is 2.2 F for Qwen2.5-VL-7B and 2.1 F for InternVL2.5-38B, but only 0.6 F for VideoLLaMA3-2B. The authors attribute this to blame attribution and patch synthesis both being performed by the model being corrected, which a 2B model is likely too weak to do well.

• Chain-of-thought helps only the largest model. CoT prompting underperforms the Direct baseline on VideoLLaMA3-2B (31.0 vs 35.6), Phi-3.5-Vision (38.0 vs 41.1), and Qwen2.5-VL-7B (44.1 vs 46.3), and only edges past it on InternVL2.5-38B (54.5 vs 53.3).

• The validation gate is the most load-bearing component. Removing the re-evaluation step causes the largest ablation drop on InternVL2.5-38B: 61.2 to 58.2 F (−3.0), with the largest precision drop (−4.5) and a recall drop of 1.8 that ties retrieval for the largest.

• Retrieval and clustering matter nearly as much. Removing memory retrieval (RAG) drops F to 59.1 (−2.1) and lowers both precision and recall; removing clustering drops F to 59.4 (−1.8) and mainly lowers precision.

• Verification components contribute smaller but measurable shares. Removing the entailment check costs 1.2 F (60.0), removing deterministic checks costs 0.9 F (60.3), and removing the critic costs 0.5 F (60.7). The critic's removal appears entirely in precision (−3.0) while recall rises (+1.6), consistent with it filtering a narrow slice of false positives rather than performing independent verification.

• NLI and MLLM entailment scorers agree equally with humans. On 56 human-labeled claim–evidence pairs, both a DeBERTa-v3-large NLI model and a prompted MLLM reached 84% agreement; the NLI model is used because it avoids an extra MLLM call.

• Repair does not overfit and keeps improving with data. Across 20 training-data fractions from 5% to 100% in 5% steps, F-score climbs steadily with no decline at any fraction, gaining +2.0 for Qwen2.5-VL-7B, +1.9 for InternVL2.5-38B, and +1.2 for Phi-3.5-Vision from the smallest to the full set. Recall peaks around 70 to 75% of the data and gives a little back by 100%, while precision stays flat for roughly the first half of the range and then climbs.

Methodology in Plain English

The system is a chain of frozen models that never retrain, only re-prompt.

First, the clip is split into three separate views. A visual agent sees up to eight uniformly sampled frames and must tag each observation with the identifier of the frame that supports it. An offline preprocessing step runs speech recognition and computes five numbers per speech segment—pitch mean, pitch variance, RMS energy, speech rate, and the length of the pause before the segment—which the acoustic agent reasons over as text, citing the segment's time span. The lexical agent reads the transcript and quotes the exact span it is interpreting. All three post to a shared evidence board.

Second, a retrieval step supplies precedent, because describing a furrowed brow or a rising pitch does not by itself name an emotion. An offline index over roughly 114k labeled samples from MER-Caption stores a text embedding of each sample's emotion description and a visual embedding of its mean-pooled keyframes. At inference, the clip's transcript and keyframes run two independent nearest-neighbor searches, fused with equal weight, and the top 5 neighbors are returned as text. Because the query is independent of what the perception agents said, perception and retrieval act as two evidence sources the integration agent must reconcile.

Third, the integration agent reads the board and the retrieved neighbors, runs an explicit cross-modal conflict test, may ask one perception agent a clarifying question, and emits a draft description, label set, confidence, and citation list. An MLLM critic then passes or fails the draft and, on failure, gives a revision instruction; the integrator redrafts once.

Fourth comes the part that distinguishes the paper. Only wrong predictions are audited. Deterministic, rule-based checks confirm that cited frames exist and contain a face, that reported prosodic numbers match recomputed ones within 15%, that retrievals are not unanimously opposite in valence to the board, and that every citation resolves to a real board entry or retrieved neighbor. An entailment check then asks whether the draft's atomic claims actually follow from the evidence they cite. Every detected failure is charged to a specific component.

Fifth, those charges are used for repair rather than retraining. Failures of the same type for the same component are grouped once at least 5 accumulate, a patch synthesizer writes a short corrective rule for that component, and the rule is tested by rerunning the entire pipeline on a held-out set with the rule inserted into an isolated copy of that one prompt. It is kept only if validation F-score does not fall. Training uses three disjoint rounds of roughly 331 samples each, a 60-sample development split purely for patch validation, and a 480-sample test split; the test split is disjoint from the 114,395-sample retrieval pool, so no test instance can be retrieved at inference. Evaluation uses the Emotion-Wheel metric, which maps gold and predicted free-text labels onto a fixed emotion vocabulary so synonyms count as matches, averaged over five emotion wheels.

Why This Matters

Impact on research. The paper reframes the debugging problem for agentic LLM systems: instead of asking a second model whether a claim sounds plausible, it checks the claim against the raw signal it cites and routes the fix to the responsible stage. This makes process-level error analysis—not just output-level scoring—a first-class object of study, and it provides a training-free adaptation path that sits between frozen prompting and expensive instruction tuning. The comparison showing that undifferentiated global reflection hurts on all four backbones is a direct challenge to the common practice of broadcasting self-reflection across an agent stack.

Real-world applications:

• Human-computer interaction, where a system that can point to the frame or the pause that drove its read of the user's state is easier to trust and to correct.
• Affective-state monitoring, where a wrong prediction needs a traceable cause before anyone acts on it.
• Sentiment and emotion analysis of online video and social media, where free-form descriptions must be checked against source footage at scale.
• Real-time conversational agents, the setting named by the workshop this paper appears in—though the authors explicitly note they do not report latency, throughput, or per-clip cost, so the accuracy gains come with an unquantified overhead.

Industry relevance. The method requires no weight updates and no gradient access, so it applies to proprietary or API-only backbones. It works across four model families from 2B to 38B, and the finding that a small labeled set already captures most of the gain (5% to 100% of training data buys only 1.2 to 2.0 F) matters for teams with limited annotation budget. The result that repair benefit scales with backbone size is equally practical: a 2B model gains only 0.6 F, so self-critique is not a free lunch at the small end.

Future Directions

• Turn attributions into supervision for weight updates. The authors propose using the same per-agent failure attributions to fine-tune the responsible agent, rather than only editing its prompt.
• Use verification scores as a reinforcement-learning reward. Per-claim deterministic and entailment scores could become a process-level reward, extending the current training-free loop to drive weight updates directly.
• Quantify the cost side. The paper reports no latency, throughput, or per-clip inference cost for any condition, despite running multiple sequential model calls per clip (three perceivers, retrieval, the integrator, and up to one critic-triggered revision) against single-call baselines. Measuring this is a prerequisite for the real-time settings the work targets.
• Raise the ceiling on attributable errors. Verification audits only what a draft cites, so a wrong prediction whose citations pass every check is labeled irreducible and cannot be repaired. How much of the error rate is attributable in the first place bounds what the repair loop can achieve, and improving coverage of that slice is an open problem.

Target Audience

Researchers and practitioners working on multimodal emotion recognition, agentic LLM system design, and retrieval-augmented reasoning who want a concrete, training-free recipe for making multi-agent outputs auditable and self-correcting. It is also useful for engineers deploying frozen or API-only MLLMs who need a debugging story for wrong predictions, and for readers interested in process-level evaluation and failure attribution as alternatives to end-to-end fine-tuning. Readers looking for a fully benchmark-saturated comparison or for latency and cost analysis will not find it here; the paper is explicit about that gap.

Authors’ abstract

LLM-based multimodal emotion reasoning systems perform well on standard benchmarks, but improving them through instruction tuning can be computationally expensive. Agentic pipelines built from frozen models avoid this cost, but introduce a different challenge: when a prediction is wrong, the system must identify which agent or stage caused the error. We address this problem with an agentic framework for traceable multimodal emotion reasoning and prompt-space adaptation. Modality-specific perception agents analyze visual, acoustic, and lexical inputs and post evidence-cited observations to a shared board. An integration agent combines these observations with retrieved multimodal examples to produce an emotion description, label set, and supporting citations, while an MLLM critic can request one revision before the output is finalized. For incorrect predictions, a separate evidence-grounded verification stage applies deterministic checks to verify that cited evidence and references are valid, and an entailment check to determine whether the final claims are supported by that evidence. Detected failures are then grouped by the responsible component and failure type, converted into targeted prompt patches, and accepted only after validation on a held-out set. This adapts the agent prompts without updating model weights. On the MER2025 dataset, targeted prompt adaptation outperforms the frozen static pipeline on all four MLLMs, ranging from 2B to 38B parameters, and matches or outperforms undifferentiated reflection baselines on each of them: VideoLLaMA3-2B, Phi-3.5-vision-instruct, Qwen2.5-VL-7B-Instruct, and InternVL2.5-38B. It improves F-score by 0.6--2.2 points depending on the backbone, including an increase from 52.3 to 54.5 on Qwen2.5-VL-7B.