When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?

AI summary

Overview

Research area: Multimodal large language models (MLLMs) that process both video and audio, specifically their failure modes under asymmetric audio-visual input, and reinforcement-learning post-training methods for fixing them.

Technical level: Advanced. The paper assumes familiarity with Group Relative Policy Optimization (GRPO), reward modeling, KL penalties, entropy minimization, and NLL loss.

Scope: The paper introduces a benchmark (AV-ConfuseBench) for "Audio-Visual Confusion"—cases where an object is visually present but its sound is muted or replaced—and proposes RL-CoMM, a two-stage reinforcement-learning system built on Qwen2.5-Omni-3B that improves audio-visual question answering and hallucination benchmarks with limited training data.

What This Paper Is About

MLLMs that jointly process video and audio are trained on synchronized multimodal data, so they tend to assume that whatever they see is also making a sound. The authors ask whether these models can detect the opposite case: an object that is clearly visible but acoustically absent or replaced. Their experiments show that models including Qwen2.5-Omni-7B, Baichuan-Omni-7B, and Gemini 2.5 largely fail this test because their reasoning is visually dominated. The goal is both to measure this failure systematically and to correct it through targeted reinforcement learning.

Key Contributions

1. AV-ConfuseBench, a new benchmark for "Audio-Visual Confusion" with two settings: audio-muted (muting one instrument in a multi-instrument performance and asking "Is there a/an {muted-object} sound?") and audio-modified (replacing the entire soundtrack with unrelated environmental audio and asking the model to describe what it sees and hears). The benchmark contains 39 videos / 73 Q&A pairs for audio-muted and 20 videos / 100 Q&A pairs for audio-modified, with all ground truths manually labeled.

2. RL-CoMM, a Reinforcement-Learning-based Collaborative Multi-MLLM system built on the Qwen2.5-Omni-3B foundation, which uses an external Large Audio Language Model (LALM) as a reference model to supply audio-only reasoning that the Omni-LLM lacks.

3. Step-wise Reasoning Reward (Step-RR), a reward function built on GRPO that combines format rewards, an Audio Reasoning Rationality reward (ARR), and an Audio-Visual Correlation reward (AVC), using Qwen3 Embedding-0.6B for semantic alignment. The KL penalty is removed during policy gradient optimization because the reference and policy models are structurally heterogeneous.

4. Answer-centered Confidence Optimization (Ans-CO), a second stage combining Negative Log-Likelihood loss with entropy minimization over answer tokens only, to reduce model uncertainty arising from heterogeneous reasoning differences.

Main Findings

• Nearly all models fail the audio-muted test. On AV-ConfuseBench audio-muted, Video-LLaMA2-7B scores 2.73 accuracy, Baichuan-Omni-7B 5.47, and Qwen2.5-Omni-7B 9.59, with "Yes" response rates of 97.27%, 94.53%, and 90.41% respectively—meaning they almost always claim a muted instrument is audible. Random choice would score 50.00.

• Closed-source models do better but not well. Gemini 2.5 Flash reaches 28.76 accuracy with 71.24% "Yes" responses, and Gemini 2.5 Pro reaches 68.50 with 31.50% "Yes" responses.

• Even thinking mode does not fully help. The authors report that about 38.36% of Gemini 2.5 Pro's responses with thinking mode remain affected by visually guided thinking and output errors.

• Models are insensitive to audio in the audio-modified setting. On a 0-5 scale judged by GPT-4, audio accuracy (A-Acc.) ranges from 0.88 to 2.83 across tested models, while visual accuracy (V-Acc.) ranges from 3.88 to 4.78—the generated content mostly elaborates visual information.

• RL-CoMM improves AVQA and AVH by 10~30% over the base model with roughly 20% of total training samples. On Music-AVQA, RL-CoMM reaches 79.46 average accuracy versus 54.95 for Qwen2.5-Omni-3B (a Δ of 24.51), with per-subtask gains of +25.59 (Exist), +22.84 (Localis), +22.79 (Count), +12.58 (Comp), and +29.73 (Temp). On AVQA it reaches 95.87 versus 83.78 (Δ 12.09).

• Gains generalize to Music-AVQA-R without extra training data. RL-CoMM averages 79.95 versus 69.43 for the base Qwen2.5-Omni-3B and 71.43 for GRPO, on head and tail splits across Exist, Localis, Count, Comp, and Temp.

• AVHBench is also improved, with one exception. RL-CoMM raises Audio-driven Video Hallucination accuracy to 78.96 (from 65.85), Video-driven Audio Hallucination to 65.63 (from 59.65), and Audio-visual Matching to 51.85 (from 48.77). The paper reports up to a 13.11% accuracy improvement and states that audio-visual matching performance "failed to optimal."

• Progressive gains on AV-ConfuseBench audio-muted. Qwen2.5-Omni-3B scores 8.22 accuracy; SFT drops it to 5.48; GRPO raises it to 15.07; RL-CoMM raises it to 27.40. In the audio-modified setting, A-Acc./V-Acc. move from 1.14/4.10 (base) to 1.84/4.47 (GRPO) to 2.36/4.54 (RL-CoMM).

• Ablation confirms both stages matter. On Music-AVQA mean accuracy, the base model scores 54.95; adding format plus accuracy rewards gives 70.05; format plus Step-wise Reasoning gives 74.49; and adding Ans-CO gives 79.46.

• Reward curves differ between the two rewards. The AVC reward rises smoothly across global steps, while the ARR reward fluctuates considerably mid-training and its peaks fall short of expectations—evidence that Omni-LLMs still struggle to reason about audio without visual interference.

Methodology in Plain English

The researchers first build a small diagnostic benchmark by taking videos where the visual content is intact but the audio is either muted for one object or replaced entirely with unrelated sound. They ask models simple yes/no or description questions and score the responses, using GPT-4 to rate the audio and visual portions of descriptions on a 0-5 scale. Observing that models lean almost entirely on what they see, they design a training procedure to force independent audio reasoning.

The training runs in three phases. First, a warm-up supervised fine-tuning stage on 100 curated Q&A pairs teaches the policy model (Qwen2.5-Omni-3B) to produce outputs in a fixed structure with three tags: audio thinking, visual thinking, and answer. Second, online reinforcement learning uses a separate audio-only model (a LALM) as a reference that produces question-oriented audio reasoning from the ground-truth answer. The policy model generates several reasoning paths per question, and each path receives a score from three sources: a format reward, an Audio Reasoning Rationality reward measuring semantic similarity between the model's audio reasoning and the reference audio reasoning (threshold ω = 0.8, judged by Qwen3 Embedding-0.6B), and an Audio-Visual Correlation reward measuring coherence between the model's audio and visual reasoning. These scores are normalized into group advantages following GRPO, and the policy is updated by gradient ascent with the KL penalty removed because the reference and policy models have different architectures. Third, Ans-CO applies an additional objective that crops each response to only the answer tokens and combines negative log-likelihood loss with an entropy-minimization term, with λ = 0.5 by default and λ = 0 set when estimated uncertainty exceeds 0.75. All experiments run on 8 NVIDIA A800 GPUs.

Why This Matters

Impact on research. The paper identifies a failure mode that is related to but distinct from audio-visual hallucination: the model is not inventing content so much as refusing to notice that a modality is incomplete. It also provides a benchmark and a training recipe showing that an external, single-modality model can serve as a useful reasoning reference for a heterogeneous multimodal policy model.

Real-world applications:

• Video accessibility and content verification, where missing or corrupted audio tracks need to be detected rather than assumed away.
• Media forensics and tamper detection, since the benchmark is built from deliberately muted or replaced soundtracks.
• Robust video search and indexing, where audio descriptions must be trusted independently of the visual stream.
• Assistive and robotic perception systems that need to reconcile conflicting or degraded sensor channels rather than defaulting to the dominant one.

Industry relevance. The paper targets open-source Omni-LLMs that are far cheaper to run than closed-source alternatives; RL-CoMM reports large gains on Qwen2.5-Omni-3B using only around 20% of total training samples, which is a practical argument for efficient post-training over simply scaling model size.

Future Directions

• Improving the audio reasoning reward. The ARR reward fluctuates and never reaches its target, suggesting that the reference-model similarity signal is too weak or noisy for audio reasoning. Better reward design is an explicit open problem.
• Fixing the audio-visual matching task. RL-CoMM improves AVHBench accuracy, precision, and F1 on audio-driven and video-driven hallucination but leaves audio-visual matching below optimal, which the authors flag as motivation to improve the reward model for audio-visual combinations.
• Removing dependence on ground-truth answers at training time. The reference model is prompted with the ground-truth answer to produce audio reasoning, which limits how the method could be applied in unlabeled settings.
• Scaling the benchmark and evaluation. AV-ConfuseBench is described as a mini-benchmark with 73 audio-muted and 100 audio-modified Q&A pairs; audio-modified quality is judged by an AI-assisted 0-5 scoring procedure rather than exact-match metrics.

Target Audience

Researchers and engineers working on multimodal LLMs, audio-visual learning, and reinforcement learning post-training; benchmark designers interested in hallucination and modality-asymmetry evaluation; and practitioners who are fine-tuning open-source omni-models on limited compute and want an efficient post-training recipe. Readers without background in GRPO, reward modeling, or multimodal architectures will find the methodology section demanding, though the benchmark results and the central "eyes versus ears" question are accessible on their own.

Authors’ abstract

Can Multimodal Large Language Models (MLLMs) discern confused objects that are visually present but audio-absent? To study this, we introduce a new benchmark, AV-ConfuseBench, which simulates an ``Audio-Visual Confusion'' scene by modifying the corresponding sound of an object in the video, e.g., mute the sounding object and ask MLLMs Is there a/an muted-object sound''. Experimental results reveal that MLLMs, such as Qwen2.5-Omni and Gemini 2.5, struggle to discriminate non-existent audio due to visually dominated reasoning. Motivated by this observation, we introduce RL-CoMM, a Reinforcement Learning-based Collaborative Multi-MLLM that is built upon the Qwen2.5-Omni foundation. RL-CoMM includes two stages: 1) To alleviate visually dominated ambiguities, we introduce an external model, a Large Audio Language Model (LALM), as the reference model to generate audio-only reasoning. Then, we design a Step-wise Reasoning Reward function that enables MLLMs to self-improve audio-visual reasoning with the audio-only reference. 2) To ensure an accurate answer prediction, we introduce Answer-centered Confidence Optimization to reduce the uncertainty of potential heterogeneous reasoning differences. Extensive experiments on audio-visual question answering and audio-visual hallucination show that RL-CoMM improves the accuracy by 10~30\% over the baseline model with limited training data. Follow: https://github.com/rikeilong/AVConfusion.