Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?

AI summary

Overview

Research area: Computer Vision / multimodal large language models (MLLMs) for social audio-visual question answering (AV-QA), with a focus on benchmark quality and chain-of-thought reasoning.

Technical level: Intermediate. The paper is readable without deep mathematics, but assumes familiarity with supervised fine-tuning (SFT), LoRA, GRPO, chain-of-thought (CoT) reasoning, and omni-models that process video, audio and text.

One-sentence scope: The paper audits the IntentBench benchmark for social understanding, shows that a plain supervised fine-tuning baseline beats existing CoT reasoning methods at far lower cost, and demonstrates that a question-independent textual caption matches full audio-visual performance.

What This Paper Is About

HumanOmniV2 and its IntentBench benchmark are widely used reference points for evaluating how well omni-models understand social interactions from video and audio. The authors inspect that benchmark closely and find it noisy, then test whether the expensive CoT reasoning pipelines built on top of it actually help. Their goal is to establish what a fair, minimal baseline looks like for social audio-visual question answering, and to measure how much models genuinely rely on the video stream.

Key Contributions

1. A diagnosis of IntentBench. The authors identify three classes of problems: questions that are broken (~7% of questions), questions answerable from text alone (~23%), and flaws inherited from the Social-IQ 2.0 source data (nearly 6% of Social-IQ questions). They show that the option-swapping curation used to make the benchmark harder did not succeed — text-answerability was 26.1% for IntentBench Social-IQ questions versus 24.4% for the same questions with original answer options, and 24.2% for a random set of Social-IQ questions of the same size.

2. IntentBench-Prime. A curated version built purely by removing questions, released as an exclusion list so any existing method can obtain Prime scores by filtering its current per-question results without re-running evaluation. It comes in two variants: Prime (Clean), which removes only broken questions, and Prime (Hard), which removes broken and text-answerable questions.

3. Vanilla SFT. A deliberately simple baseline: standard supervised fine-tuning of Qwen2.5-Omni for direct AV-QA, using the exact same audio-video and image QA data as HumanOmniV2 but no reasoning traces. The authors argue this baseline should be mandatory in future AV-QA papers.

4. A caption and question-only analysis of modality reliance. Question SFT (trained only on question and answer text) and Caption SFT (trained on a question-independent caption produced by ASID-Captioner) are introduced as reference points to isolate how much the video modality actually contributes.

Main Findings

• IntentBench is highly noisy. Approximately 7% of questions are broken and approximately 23% are trivially answerable without the video input. Broken questions originate from both the Social-IQ 2.0 source data and the curation process.

• At least one distractor was swapped in 54% of IntentBench Social-IQ 2.0 questions. Easy distractor options were not newly created but replaced with ground-truth answers from other questions on the same video. Across the three benchmark columns (IB S-IQ full, swapped, unswapped), Qwen2.5-Omni-7B scored 64.5 / 74.8 / 84.7 and HumanOmniV2 scored 68.9 / 75.9 / 88.1, while four small LLMs agreeing on the ground truth from text alone scored 26.1 / 42.4 / 39.2. The swapped options are factually grounded in the video, which is why omni-models struggle with them even though the swap makes little difference to text-only LLMs.

• The curation did not select a harder subset. Text-answerability was similar between IntentBench Social-IQ questions with original options (24.4%) and a same-sized random set of Social-IQ questions (24.2%). The authors trace this to a granularity mismatch: filtering was done at the video level rather than the question level.

• Curation removed 192 of 2,356 Social-IQ questions as broken, of which 57 (30%) involved swapped distractors, and 616 as text-answerable, of which 536 (87%) involved swapped distractors. The two sets overlap by 18 questions. IntentBench-Prime has 2,497 questions (Clean) and 1,899 (Hard) across 633 videos, versus 2,689 questions for the original IntentBench. The appendix tables report N = 2,493 and N = 1,895 for the same two splits.

• Removing trivial questions lowers accuracy but does not eliminate priors. On IntentBench-Prime (Hard), Qwen2.5-Omni-7B with text only (QA only) reached 46.9%, above the approximately 28% random-chance level, and Granite-4.1-8B reached 41.5% — 5.4% below the Qwen text-only baseline.

• Vanilla SFT matches or beats CoT reasoning methods. On IntentBench, Vanilla SFT (LoRA) scored 73.8 average versus 71.8 for HumanOmniV2 and 67.8 for Qwen2.5-Omni-7B; on IntentBench-Prime (Hard) it scored 70.4 versus 66.9 for HumanOmniV2 and 63.2 for Qwen2.5-Omni-7B. On Daily-Omni, Vanilla SFT (LoRA) averaged 65.2 versus 58.5 for HumanOmniV2 and 62.1 for Qwen2.5-Omni-7B, and on WorldSense it scored 48.8 versus 47.1 for HumanOmniV2 and 45.4 for Qwen2.5-Omni-7B. AVATAR and AffectOmni did not outperform the baseline on any benchmark.

• The exception is deception detection. HumanOmniV2 reports 64.0% there, but the authors' own reproduction reaches only 60.2%, and all methods are close to chance on this small binary subset. Four out of 200 videos caused a loading error and were excluded.

• Reasoning is far more expensive. Training cost was 18 GPU hours for Vanilla SFT versus approximately 200 for the reasoning approach (roughly 11x), output length was 2 tokens versus 527 (263x), and total latency per question was 1.23 s versus 8.31 s (6.7x), driven by decoding: 0.02 s versus 7.12 s, a 356x gap. Accuracy on IntentBench-Prime (Hard) was 70.4 versus 66.9.

• A question-independent caption matches full audio-visual input. On IntentBench-Prime (Hard), Qwen2.5-Omni-7B scored 46.9% on questions and answers only, 62.1% on an ASID caption, and 63.2% with audio, video and text. After fine-tuning, Question SFT reached 62.0% (15.1% above its baseline and within 7.5% of Vanilla SFT and 6.6% of Caption SFT), Caption SFT reached 68.6%, and Vanilla SFT 69.5% — a difference of close to a single percentage point between caption and video.

• Priors learned from text do not explain out-of-domain gains. The authors note that prior learning cannot account for Vanilla SFT's gains on WorldSense and Daily-Omni.

Methodology in Plain English

The authors start by auditing IntentBench rather than building a new model. To quantify text-answerability, they run four small LLMs — Gemma-2-9B-Instruct, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Qwen3.5-9B — on the Social-IQ questions and answer options alone, and treat a question as trivial only when all four agree on the correct answer. This is deliberately strict, so it captures only questions that are certainly trivial. To find broken questions, they rank all Social-IQ question-answer pairs by suspicion, using programmatic checks (such as a plain yes or no as the answer, two identical answers where one is the ground truth, the ground truth being a sub- or super-string of another option, or placeholder answers such as "N/A (Wrong video clip)"), an audit with Claude Haiku 4.5, and the agreement patterns of internally fine-tuned social reasoning models. They then manually inspect questions in ranked order, watching the video only when a question cannot be judged from the QA pair alone, and stop once 300 consecutive questions are clean. By that point more than half the Social-IQ questions had been manually verified.

For the modeling experiments, they fine-tune the thinker part of Qwen2.5-Omni-7B with the video and audio encoders and aligner layers frozen, training LoRA at a learning rate of 1e-4 for 1 epoch with rank 16, 2 FPS and 32 maximum frames. Training data mirrors HumanOmniV2 exactly — OmniInstruct, Video-R1, Social-IQ 2.0 training data, and the 200 EMER entries not in IntentBench, totalling 20K videos and 10K images. Training takes under 4.5 hours on 4x H100 80 GB. A full-parameter variant needed 2 epochs to reach similar or slightly worse performance than LoRA. They also train Question SFT and Caption SFT on a reduced data setting limited to Social-IQ 2.0 and EMER, identical except for the input modality, and evaluate each in the same modality setting it was trained on.

Why This Matters

Impact on research. The paper challenges the assumption that adding CoT reasoning is automatically the right move for social AV-QA. By showing that a simple LoRA SFT baseline outperforms or matches HumanOmniV2, AffectOmni and AVATAR across IntentBench, IntentBench-Prime, WorldSense and Daily-Omni, it argues for disentangling gains that come from fine-tuning on task data from gains attributable to a superior method. The release format of IntentBench-Prime — an exclusion list rather than new annotations — makes re-scoring cheap for the community. The caption result is also a methodological warning: video-grounded benchmarks may be measuring less visual grounding than assumed.

Real-world applications.

• A two-stage pipeline that first generates a dense caption of a video and then answers questions from that textual trace, enabling very low latency at query time and allowing multiple questions to be answered from a single trace.
• Social robotics and embodied agents that need to interpret intentions, emotions and deceptive behaviour during interaction.
• Emotion recognition and deception-detection systems built on video collections, such as the EMER and MDPE subsets used in IntentBench.
• Efficient deployment of multimodal QA where long reasoning traces would make inference prohibitively slow, since the measured decode-time gap is 356x.

Industry relevance. The training and inference cost figures — 18 versus approximately 200 GPU hours, and 1.23 s versus 8.31 s per question — are directly relevant to organisations choosing between reasoning-heavy and direct-answering model designs. The finding that a caption-only model can substitute for full video processing offers a route to pre-compute video content once and amortise it across many queries.

Future Directions

• Determine how much of the caption-based parity between Caption SFT and Vanilla SFT stems from format alignment, source-specific priors, or other factors, since the authors state this remains limited to the social domain.
• Reduce the substantial learnable priors that Question SFT still exploits even on IntentBench-Prime (Hard), which the authors identify as an open limitation of the curated benchmark.
• Extend the curation beyond removal of questions, since the current pipeline only shrinks the benchmark and lightly balances its category distribution.
• Test whether the results generalise beyond the single base model used throughout, Qwen2.5-Omni, and investigate how to make MLLMs extract genuinely question-specific information from audio and video streams.

Target Audience

Researchers and practitioners working on multimodal large language models, audio-visual question answering, and social or affective computing who need a critical view of benchmark validity. The paper is especially useful for anyone benchmarking against IntentBench or HumanOmniV2, for teams designing fine-tuning pipelines who want a strong baseline before investing in reinforcement learning or chain-of-thought distillation, and for engineers weighing inference cost against reasoning quality in deployed multimodal systems.

Authors’ abstract

Training Multimodal Large Language Models for audio-visual social understanding is a crucial step toward embodied social intelligence. Chain-of-thought (CoT) reasoning has become the dominant approach, with HumanOmniV2 and its IntentBench benchmark as a prominent reference point. In this context, we report three findings. First, IntentBench is highly noisy: $\sim$7% of questions are broken and $\sim$23% are trivially answerable without the video input. We remove the affected questions and release Intentbench-Prime. Second, current reasoning approaches are expensive and surprisingly ineffective. A simple Vanilla SFT baseline matches or outperforms existing reasoning methods across three benchmarks at a fraction of the cost, establishing it as an essential baseline for evaluating novel fine-tuning techniques. Third, our analysis reveals that substantial priors can be learned solely from the text modality and that using a textual caption instead of the video yields performance on par with Vanilla SFT. These surprising findings reveal the limitations of current MLLMs when it comes to social understanding. IntentBench-Prime, Vanilla SFT model, and code are publicly available.