EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

AI summary

Overview

Research area: Computer Vision / multimodal (audio-visual) evaluation of Omni-modal Large Language Models.

Technical level: Intermediate (readable with basic familiarity with benchmarks and LLM evaluation; the metric definitions are given in closed form).

Scope: The paper introduces EvaOmni, a human-annotated, evidence-grounded benchmark of 392 videos and 1,027 question-answer pairs that scores Omni LLMs on both answer correctness and the timestamped audio-visual evidence used to justify the answer.

What This Paper Is About

Existing audio-visual reasoning benchmarks are mostly multiple-choice tests scored only on whether the final answer is correct, so a lucky guess or a shortcut (using only text cues, only audio, or only video) earns the same credit as genuine joint reasoning. The authors first quantify this weakness across prior benchmarks, then build a new benchmark in which expert annotators mark the audio and visual time intervals that actually support each answer, and models must return both the answer and those intervals. The goal is to make evaluation reflect the full chain from evidence to conclusion rather than just the endpoint.

Key Contributions

1. An evidence-grounded benchmark (EvaOmni): 392 videos and 1,027 purely human-annotated question-answer pairs organized into 15 fine-grained capabilities across three levels—Level 1 Retrieval and Grounding (436 questions), Level 2 Basic Understanding (438 questions), and Level 3 Complex Reasoning (153 questions)—with expert-annotated audio and visual evidence intervals for every question.
2. A multi-stage quality-control pipeline: video screening (post-January 2025 uploads, no hard-coded subtitles, 1–12 minute duration, contamination probing with Gemini3.1-Pro-Preview), expert annotation by five senior PhD students, human cross-validation, and model-assisted shortcut screening under text-only, audio-only, and vision-only inputs. Roughly 78% of initially submitted pairs pass first review and roughly 82% pass human cross-validation, with the remaining 18% discarded.
3. Two new grounded metrics: the Coverage Grounded Score (C.G.), which averages the fraction of annotated audio and visual evidence segments hit by predictions and weights this by answer correctness, and the Bottleneck Grounded Score (B.G.), which multiplies answer correctness by the minimum of the audio and visual IoU so that strong localization in one modality cannot mask failure in the other.
4. A large-scale evaluation: 24 Omni LLMs (7 open-weight, 17 proprietary) plus a human expert reference, with additional cost-efficiency analysis and input-modality ablations.

Main Findings

• Best overall model: Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%) among the 24 evaluated models.
• Open-weight versus proprietary gap: The strongest open-weight model, MiMo-V2.5-310B-A15B, reached 56.8% accuracy, 48.4% C.G., and 20.5% B.G.—gaps of 22.6, 22.4, and 14.6 percentage points behind Gemini3.7-Flash.
• Large models under 100B parameters struggle most with grounding: Across the four open-weight models with fewer than 100B total parameters, overall C.G. ranged from 0.8% to 12.6% and B.G. from 0.3% to 4.0%, compared with 30.4%–70.8% and 10.0%–35.1% for proprietary models. Qwen2.5-Omni-7B, for instance, scored 16.4% accuracy but only 0.8% C.G. and 0.3% B.G.
• Humans remain far ahead in precise localization: Human experts scored 96.8% accuracy, 96.0% C.G., and 94.3% B.G., exceeding Gemini3.7-Flash by 17.4, 25.2, and 59.2 percentage points respectively. The drop from accuracy to B.G. was 2.5 points for humans versus 44.3 points for Gemini3.7-Flash.
• Accuracy and grounding diverge across levels: Accuracy decreased from Level 1 to Level 2 for 21 of 24 models but increased from Level 2 to Level 3 for 13 models, while Level 3 produced the lowest C.G. and lowest B.G. for all 24 models. Gemini3.7-Flash rose from 73.6% to 79.1% accuracy between Levels 2 and 3 while C.G. fell from 65.3% to 49.1% and B.G. from 29.2% to 12.3%.
• Existing benchmarks are shortcut-prone: Under text-only input, 14.4% of WorldSense questions were answered correctly; under audio-only, 14.2%; under video-only, 15.7%. The authors note this is not a strict shortcut ratio because it depends on the test models. They also point out that uniform guessing on four-option questions yields an expected 25% accuracy.
• Both modalities matter: For MiMo-V2.5, accuracy dropped by 26.8 points with audio alone and 30.0 points with video alone relative to Video+Audio, while B.G. fell from 20.5% to 0.0% in both cases.
• Native audio beats ASR transcripts in the tested settings: MiMo-V2.5 with audio alone reached 30.0% accuracy and 10.0% C.G., versus 23.8% and 7.1% with ASR alone; Video+Audio (56.8% accuracy, 48.4% C.G., 20.5% B.G.) exceeded Video+ASR by 8.1, 10.3, and 8.9 percentage points.
• ASR helps vision-language models but does not fix grounding: Adding ASR to video raised Qwen3VL-8B from 13.8% to 23.2% accuracy and Qwen3VL-A3B from 8.5% to 19.3%, but their B.G. remained at only 1.9% and 0.8%, versus 11.6% for MiMo-V2.5 under Video+ASR.
• Correct answers often lack correct grounding: Depending on the model, 48.1%–79.9% of correct answers failed the criterion that both modality IoUs reach 0.5. Gemini3.7-Flash answered 79.4% correctly but satisfied both IoU criteria on only 41.8%; Qwen3.8-Omni-Flash dropped from 58.2% to 11.8%.
• Grounding without correct answers is rarer: 12.2%–32.6% of grounding-passing predictions still chose the wrong option, and this category never exceeded 8.8% of all questions.
• Audio is easier to localize than video for most models: Audio IoU exceeded visual IoU by 0.7–3.7 points for six of the seven models analyzed, with Qwen3.8-Omni-Flash the exception (27.4% versus 34.3%).
• Higher scores do not imply better cost efficiency: Doubao-Seed-2.1-Lite had the best cost efficiency on all three metrics ($0.0188, $0.0208, $0.0491 per unit of normalized score, total cost $13.71). Qwen3.8-Omni-Flash had the lowest total cost ($11.54). Gemini3.7-Flash, despite the best scores, ranked fourth on cost efficiency ($0.0545, $0.0611, $0.1232, total $44.41) but outperformed Gemini2.5-Pro on all three metrics while cutting total cost from $224.48 to $44.41 (reductions of 82.3%, 82.8%, and 83.7%).

Methodology in Plain English

The authors started by testing whether prior benchmarks can be gamed: they fed three existing benchmarks only the question text, only the audio, or only the video, and measured how often models still answered correctly. Finding substantial shortcut rates, they built a benchmark designed to block them.

Video collection favored recency (uploads after January 2025) and variety, excluded videos with burned-in subtitles so models could not read answers off the screen, and kept clips between 1 and 12 minutes. To probe for training-data contamination, they sampled three frames per video and asked Gemini3.1-Pro-Preview to describe the content, excluding videos the model could reconstruct. Five senior PhD students wrote questions that require both modalities, and for each question marked the audio and visual time ranges—with content descriptions—that justify the answer; Level 3 multi-hop items also include a step-by-step reasoning chain.

Quality control was layered. Each annotator's questions were split into four parts and redistributed to the other four annotators for cross-checking on capability alignment, shortcut susceptibility, and evidence accuracy. Disputed items went to two further experts who were neither the author nor the first reviewer, and an item survived only if both approved. Finally, Gemini3.1-Pro-Preview and Qwen3-Omni-Instruct-30B-A3B were run under text-only, audio-only, and vision-only conditions; any question a model could still answer with a plausible rationale was sent back for revision.

At evaluation time, models receive the video and audio and must return an answer plus start and end timestamps for supporting audio and visual evidence. Accuracy is combined with two scores: coverage (how much of the annotated evidence was overlapped at all) and a bottleneck score (the worse of the two modalities' intersection-over-union, multiplied by whether the answer was right). Frames were sampled at 1.0 FPS and resized so the longer side is 480 pixels; open-weight models outside the MiMo family ran on a single server with eight NVIDIA H20 GPUs, while MiMo models and all proprietary models were accessed through official APIs.

Why This Matters

Impact on research. The paper shows that answer-only multiple-choice scoring can obscure large capability differences—models that look acceptable on accuracy can have near-zero evidence localization—and it provides a released dataset, annotations, and evaluation framework (GitHub, ModelScope, Hugging Face) so that grounding can be measured reproducibly. The dual metrics separate two distinct abilities: broad evidence coverage (C.G.) and precise temporal localization in both modalities (B.G.).

Real-world applications:

• Sports event understanding, where a system must tie commentary and crowd noise to specific on-screen actions.
• Film and television analysis, where plot points depend on dialogue, music, and visual action at different moments.
• Verifiable video question answering, where a user can check the cited timestamps rather than trusting a bare answer.
• Model selection under budget constraints, using cost-efficiency figures for accuracy, coverage, and localization together.

Industry relevance. The cost-efficiency analysis gives practitioners a way to weigh capability against API spend, and the finding that stronger scores do not imply better efficiency is directly actionable for deployment decisions. The localization results also identify precise temporal grounding as the capability most clearly missing from current Omni LLMs, which signals where model development effort is needed.

Future Directions

• Improving precise dual-modal temporal localization, since even the best model reaches 35.1% B.G. against 94.3% for human experts, and Level 3 B.G. falls to 12.3% for Gemini3.7-Flash.
• Extending the benchmark to longer videos; the authors explicitly list audio-visual reasoning over longer time spans as future work, since the current collection is capped at 1–12 minutes.
• Understanding why accuracy can rise from Level 2 to Level 3 while grounding scores fall—one hypothesis offered is that partial evidence lets models eliminate distractors without localizing all supporting evidence, but the authors state the aggregate results do not directly establish models' answer-selection strategies.
• Reducing reliance on transcribed speech, given that ASR improved the tested vision-language models but left their B.G. at 1.9% and 0.8%, and that non-speech sounds, prosody, and transcription or timestamp errors may explain the audio-versus-ASR gap.

Target Audience

Researchers and engineers working on Omni-modal and audio-visual LLMs who need a reliable way to evaluate joint reasoning; benchmark designers interested in grounded, evidence-based evaluation protocols; and practitioners choosing among proprietary and open-weight models who need to weigh answer quality, evidence localization, and API cost together. It is also useful for readers tracking the gap between model and human performance on multimodal tasks.

Authors’ abstract

Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.