ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark

AI summary

Overview

Research area: Computer Vision / multimodal large language model (MLLM) evaluation, specifically audio-visual question answering and grounded chain-of-thought reasoning.

Technical level: Intermediate. The paper is readable without deep implementation knowledge, but familiarity with MLLM benchmarking, chain-of-thought prompting, bounding boxes, and standard scoring metrics (accuracy, IoU, Cohen's kappa, Pearson correlation) helps.

Scope: ANCHOR is a benchmark and evaluation framework of 2,319 videos and 8,272 question–answer pairs that measures whether an MLLM's stated reasoning actually cites the visual regions, temporal intervals, and audio events that support its answer, rather than only checking whether the final answer is correct.

What This Paper Is About

Multimodal models often answer audio-visual questions correctly while their explanations rest on linguistic shortcuts, subtitles, or question phrasing rather than on what is actually visible and audible. Existing benchmarks mostly score only the final answer, so they cannot tell whether a model genuinely reasoned over perceptual evidence or merely guessed well. ANCHOR addresses this by pairing each question with a structured, human-refined "grounded chain-of-thought" trace linked to bounding boxes, timeframes, and audio cues, and by scoring answer correctness, hallucination, logical divergence, and conciseness separately.

Key Contributions

1. A grounded audio-visual CoT benchmark. ANCHOR contains 2,319 videos and 8,272 question–answer pairs with structured reasoning traces, temporal evidence, spatial bounding boxes (generated with SAM3), and salient audio cues, plus a human-refined test set of 224 videos and 975 QA pairs.

2. A multi-facet evaluation framework. Rather than a single accuracy number, the benchmark separately scores answer correctness, hallucination/faithfulness, logical divergence, and conciseness on 1–5 Likert scales using three LLM judges (GPT-5-nano, Grok-4.1-fast, Claude-Haiku-4.5), and adds direct spatial (IoU) and temporal (precision/recall/F1) grounding measurements on a focused 100-video, 446-QA subset.

3. A human-in-the-loop annotation pipeline. Pseudo ground-truth traces from Gemini-2.5-Pro and SAM3 box proposals are refined by 26 IRB-approved study participants in an extended VidAT interface that synchronizes video, audio, bounding boxes, timeline, and a CoT editor.

4. Diagnostic evidence that answer accuracy hides grounding failures, including a modality ablation across three models and a reference-source circularity check showing rankings stay stable (mean Pearson r = 0.99) across human-refined, Gemini-generated, and GPT-generated references.

Main Findings

• Correct answers do not imply grounded reasoning. Gemini-3-Flash and GPT-5.2 reach similar answer scores (3.83 and 3.77) but differ on hallucination and divergence; on the 2,012 items where all judges rate both answers fully correct, GPT-5.2's hallucination score is at least one point lower on 250 items and at least 1.5 points lower on 81, with all judges agreeing on direction in 74% of the latter cases.

• Three recurring failure modes. Inspection of those cases identified identifying the wrong event, localizing the wrong entity or interval, and retrieving relevant evidence without connecting it correctly to the answer.

• Verbosity is not grounding. Gemini-3-Flash earns the highest conciseness score in the main table (4.47, 203 tokens) while GPT-5.2 scores 2.72 at 342 tokens; Grok-4.1-Fast-Reasoning and Claude-Sonnet-4.6 are the most verbose (644 and 571 tokens, conciseness 1.30 and 1.50). Figure 1 contrasts a verbose but uninformative Gemini-3-Flash rationale against a GPT-5.2 "early exit" that misses late audio evidence and answers incorrectly.

• Input format is associated with claimed audio absence. Models receiving frames plus ASR rather than native audio claim no usable audio is available on 25.3% of GPT-5.2, 25.0% of Claude, and 20.9% of Grok responses, versus 3.7% for Nova and 0.1% for Gemini-3-Flash; in 85% of GPT-5.2's cases the reference rationale depends on informative non-speech audio. The authors explicitly treat this as explanatory, not causal, since input format and architecture are confounded.

• A clear modality hierarchy in the ablation (975 QA pairs; Gemini-3-Flash, GPT-5.6-Luna, Claude-Sonnet-5). Full-modality accuracy is 57.7, 52.5, and 52.6. Removing video drops accuracy by 38.4–41.8 points (to 16.1, 10.7, 14.2). Removing audio costs 4.2–5.6 points, removing bounding boxes costs 3.2–7.7 points, and removing both costs 7.5–9.6 points — smaller, complementary, and model-dependent effects.

• Reference-source circularity is not driving the rankings. On the matched 100-video, 446-QA subset, model scores correlate across human-refined, Gemini-2.5-Pro-generated, and GPT-5.6-Luna-generated references at Pearson r = 0.96–1.00 (mean 0.99; Spearman 0.88–1.00, mean 0.95; n = 9 models), with mean absolute difference of 0.05 points on the 1–5 scale (max 0.08). Gemini does not score highest under its own generated reference on any judged facet.

• Direct grounding differs sharply between models. Against human-refined tracks, GPT-5.6-Luna reaches IoU 0.459, IoU ≥ 0.5 on 48.0% of boxes, and temporal F1 0.80; Claude-Haiku-4.5 reaches IoU 0.217, 10.9%, and temporal F1 0.61, with higher temporal recall (0.95 vs 0.85) but lower precision (0.45 vs 0.76).

• LLM judges agree with each other more than with humans. Pairwise weighted Cohen's kappa peaks at 0.714 (Divergence, Grok-4.1-Fast vs GPT-5-Nano) and 0.629 (Hallucination, Grok-4.1-Fast vs Claude-Haiku-4.5). Against human ratings on 100 sampled responses, the judge panel correlates only moderately (Pearson r = 0.50 pooled; per-facet 0.43–0.67; weighted kappa 0.11–0.39), and humans score systematically higher — hallucination mean 4.35 versus judge-panel 2.35.

• The benchmark is deliberately hard. Difficulty labels from 16 rollouts per question (four rollout models answering four times each) yield 27.5% easy, 18.7% medium, and 53.8% hard, with about 80% of LLM-labeled hard pairs confirmed as hard by human review.

• Scale and diversity. Videos total 745,443 seconds (approximately 207 hours), averaging 321.45s (median 183.53s), with 37.86% under one minute, 54.12% between one and ten minutes, and 8.02% over ten minutes, drawn from nine source datasets and 21,003,063 processed frames at an average of 28.24 FPS.

Methodology in Plain English

The team did not author new videos. They curated 8,272 question–answer pairs from nine existing datasets (AVQA, Ego4D, THUMOS-14, Animal-Bench, MAVERIX, COIN, Moviechat, Video-MME, LVBench), picking examples that genuinely require visual or audio evidence, and labeled them with GPT-5.2 into seven top-level categories (Shopping, Sentiment, Egocentric Agent, Information Query, Social Interaction, Sports, Gaming).

To rate difficulty, they had four models answer each question four times, then used judge models to check how often the 16 candidate answers disagreed with the reference and how similar the answers within one model's batch were. Questions with at least 10 disagreements, or with maximum intra-rollout diversity, were marked hard; the rule takes the more severe of the two signals. Human experts then reviewed hard-labeled items.

For the grounding annotations, Gemini-2.5-Pro was prompted with the question and reference answer to write a structured trace with separate "Visual Cue" and "Audio Cue" sections (each with timeframe, description, and analysis) plus a final answer. SAM3 produced bounding-box proposals for the concrete noun phrase most relevant to each question, with abstract concepts mapped to the visible entity performing the action.

Human annotators then worked in an extended version of the VidAT tool that synchronizes video, audio, bounding boxes, timeline, and an editable CoT panel. They corrected boxes, propagated them across frames, fixed wrong time spans, added missing audio evidence, and rewrote reasoning steps so each one pointed at the raw perceptual evidence. Twenty-six participants completed this at roughly 10 videos per hour.

At evaluation time, native audio-video models get the stream directly, while others get sampled frames plus a Whisper ASR transcript. Clips of 20 seconds or less are fed whole; longer clips use a Socratic sequential-chunk procedure where the model sees each chunk plus its prior intermediate reasoning, followed by a text-only synthesis pass. Frames are resized to 512×384; frame-based models use 120-second chunks at 0.5 FPS, native-video models use 1800-second chunks. Three LLM judges score answer correctness, faithfulness, and divergence, while conciseness is computed mechanically from token counts using the cl100k_base tokenizer and binned into quintiles.

Why This Matters

Impact on research. The paper argues that answer-only evaluation obscures grounding failures, so leaderboard gains may reflect shortcut learning rather than perceptual understanding. ANCHOR provides evidence-linked scoring that can distinguish the two, and its reference-circularity check offers a template for showing that LLM-generated references do not simply reward the model that produced them.

Real-world applications:

• Assistive and accessibility systems that describe scenes and sounds to blind or low-vision users need explanations that cite the actual object and sound event, not plausible-sounding filler.
• Video content moderation and compliance review, where an automated rationale must point to the specific interval, entity, and audio event that justifies a flag.
• Clinical or eldercare monitoring from egocentric and social-interaction video, where "the model was right" is insufficient if the cited evidence is fabricated.
• Robotics and embodied agents that must act on first-person audio-visual input, matching the Egocentric Agent category in the benchmark.

Industry relevance. The work is a CMU–Amazon collaboration funded by Amazon, and its ablation results give practitioners a concrete priority ordering: visual evidence dominates (removing it costs 38.4–41.8 points), while bounding boxes and audio contribute smaller but complementary gains. The finding that frame-plus-ASR pipelines claim audio is unavailable far more often than native audio-video models (25.3% for GPT-5.2 versus 0.1% for Gemini-3-Flash) is directly actionable for teams choosing input representations.

Future Directions

• Scale up human-refined annotations. The authors name the limited scale of human-refined annotations as an explicit limitation, since only 224 of 2,319 videos and 975 of 8,272 QA pairs received full human refinement.

• Reduce reliance on pseudo ground truth. Initial traces come from Gemini-2.5-Pro, which the authors list as a limitation; extending human-authored traces, or validating pseudo traces more broadly, would strengthen the gold standard.

• Improve judge calibration. The gap between judge-panel and human ratings (Pearson r = 0.50; hallucination 2.35 versus 4.35) points to a need for better rubrics, human-in-the-loop judging, or judge training rather than assuming either evaluator is more reliable.

• Broaden dataset coverage and disentangle input format from architecture. The authors flag coverage of the source datasets as a limitation, and note that the audio-absence association is confounded because input format and model architecture vary together — a controlled study could separate them.

Target Audience

Researchers and engineers building or evaluating multimodal large language models for video and audio understanding, benchmark designers who need to move beyond accuracy-only leaderboards, and teams deploying audio-visual models in settings where the stated justification for a prediction must be verifiable. It is also useful for practitioners choosing between native audio-video input and frame-plus-transcript pipelines, and for anyone studying LLM-as-judge reliability, given the reported human-judge calibration gap.

Authors’ abstract

Multimodal large language models (MLLMs) perform well on audio-visual question answering, but correct answers do not necessarily imply that their stated reasoning is grounded in relevant perceptual evidence. Existing benchmarks mainly evaluate final-answer accuracy, offering limited insight into whether model-generated rationales correctly identify visual regions, temporal intervals, and audio events supporting predictions. We introduce ANCHOR, a benchmark and evaluation framework for measuring grounded-rationale alignment in multimodal reasoning, comprising 2,319 videos and 8,272 question-answer pairs with structured reasoning traces, temporal evidence, spatial bounding boxes, and salient audio cues, including a human-refined test set of 224 videos and 975 question-answer pairs. We separately measure answer correctness, hallucination, logical divergence, and conciseness, with direct spatial and temporal grounding measurements on a focused subset. Across the full benchmark, twelve contemporary MLLMs show a substantial alignment gap: models with comparable answer performance can differ markedly in whether their explanations are supported by annotated evidence. On a matched subset of 100 videos and 446 question-answer pairs, rankings remain highly stable across human-refined, Gemini-generated, and GPT-generated references (mean Pearson r=0.99), providing evidence against model-reference circularity within this subset. Grounding-cue ablations of three MLLMs on the full human-refined test set reveal a consistent modality hierarchy: removing video reduces accuracy by 38.4-41.8 points, while removing bounding boxes (3.2-7.7 points) and audio (4.2-5.6 points) causes smaller, complementary, model-dependent drops. These findings show answer-only evaluation obscures important grounding failures and position ANCHOR as a testbed for evaluating accurate and evidence-aligned audio-visual reasoning.