OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

AI summary

Overview

Research area: Computer Vision / Multimodal Large Language Models — specifically evaluation benchmarks for video grounding and video question answering.

Technical level: Intermediate. The core ideas are intuitive, but the paper assumes familiarity with temporal grounding (predicting when an event happens), spatial grounding (predicting where a target is), and omni-modal models that ingest audio plus video.

Scope: The paper introduces OmniCompass, a manually annotated benchmark of 820 videos (2,460 instances) plus a 240-video audio-visual subset (720 instances) that scores 50 multimodal models on temporal grounding, spatial grounding, and question answering under two paired target-specification settings.

What This Paper Is About

Existing video benchmarks each test only part of the picture: some only use text queries, some only test visual evidence without audio, and most cover either question answering or grounding but rarely both. That makes it hard to tell why a model fails — whether it retrieved the wrong moment, localized the wrong object, or misunderstood the event.

OmniCompass is built so that the same target in the same video is queried three ways (when, where, what) and two ways per query (text alone versus text plus a reference image), with and without audio. The goal is a diagnostic benchmark: not just a leaderboard score, but a breakdown of where multimodal video models actually break.

Key Contributions

1. A holistic evaluation framework with paired specifications. OmniCompass unifies four dimensions in one benchmark: paired Text and Text+Image target specifications, two evidence tracks (visual-only OmniCompass-VL and audio-visual OmniCompass-AV), three tasks (TG, SG, QA), and short/medium/long video durations. The paper also introduces Cross-Specification Consistency Scores (CS), which require a model to be correct in both specification settings and to agree with itself, so that a model cannot score well by being consistently wrong or by agreeing on wrong answers.

2. A diverse, rigorously verified dataset. OmniCompass spans 820 videos, 80.6 hours, 7 domains, and 52 subcategories, drawn from nine public datasets plus 240 self-collected movie, TV, variety, and news videos. Construction took approximately 1,800 human-hours across annotator training, a pilot calibration phase, annotation, expert adjudication, and revision, with multi-round quality verification of every instance.

3. A diagnostic evaluation of 50 MLLMs. The study covers proprietary and open-source vision-language and omni-modal models, and reports category-level QA breakdowns, consistency gaps, and controlled ablations over target specification, video duration, frame-level timestamps, and input modality.

4. Actionable findings about what is actually hard. Grounding and fine-grained perception lag far behind semantic QA, reference images do not reliably help, and audio helps question answering more than temporal grounding — results that point at specific capability gaps rather than a single aggregate number.

Main Findings

• Proprietary models lead, and the gap is enormous on audio-visual tasks. Gemini-3.8-Flash is the top proprietary model on both tracks, at 42.8% CS_Avg on OmniCompass-VL and 57.4% on OmniCompass-AV. Kimi-K3 leads open-source models on the VL track at 39.6%; MiMo-V2.5 leads open-source on the AV track at 24.6%. The best proprietary model exceeds the best open-source model by 3.2% CS_Avg on VL and 32.8% on AV.

• QA is strong; grounding is weak. On the VL track, 29 of 36 models exceed 50% in CS_QA, peaking at 69.1%, while the best CS_TG is only 27.1% and the best CS_SG only 34.9%. On the AV track, Qwen3.8-Omni-Flash reaches 73.2% in CS_QA but only 21.8% in CS_TG, and all ten open-source models score at most 3.5% in CS_TG.

• Visual Counting is the hardest QA category on the VL track. It ranks last in CS_QA for all eight plotted models, at 26.4%–48.1%. Spatial Perception averages 61.0%, below 69.8% for Causal Reasoning.

• On the AV track, temporal reasoning is the weak spot while causal reasoning is easy. Temporal Reasoning ranks lowest for five of six models, whereas Causal Reasoning leads all six at 74.3%–94.3%. Qwen3-Omni-30B-A3B scores lowest on Event Understanding at 31.5%.

• Audio-visual event binding is where open-source models fall furthest behind. The proprietary/open-source CS_QA gap is 12.5% on Speech Understanding but reaches 38.9% on Event Understanding, which requires linking audio cues to visible entities, actions, or scene context.

• Adding a reference image does not reliably help. Text+Image improves the single-setting average for only 11 of 36 models on the VL track — 8 of 12 proprietary models but only 3 of 24 open-source models. On the AV track, Text+Image improves SG m_sIoU and QA accuracy for 12 of 17 models each, but TG m_tIoU for only five. Among sub-100B open-source VL models, only Qwen3.8-27B improves, gaining 1.9% in TG m_tIoU and 1.5% in SG m_sIoU while losing 2.2% in QA accuracy.

• Consistency across specifications lags behind single-setting accuracy. Across the 36 VL models, CS_TG, CS_SG, and CS_QA average 4.3%, 4.6%, and 6.4% below the lower corresponding single-setting score. MiMo-V2.5 on the AV track achieves 59.2% and 58.7% QA accuracy under Text and Text+Image respectively, but only 47.3% in CS_QA — similar aggregate accuracy without answering the same pairs correctly.

• Bigger models do not win every task. Within Qwen3.5, the 397B model beats the 122B model in CS_Avg (37.2% vs 33.5%) but trails in CS_TG (23.1% vs 24.3%); the 27B model trails the 9B model in CS_SG (15.3% vs 17.7%).

• Frame-level timestamps help almost everywhere. Timestamps improve CS_Avg in all five model-track configurations, with gains in 14 of 15 task-level comparisons. VL gains are 7.0% (Gemini-3.8-Flash), 10.4% (Qwen3.5-397B-A17B), and 2.3% (Qwen3.8-27B), averaging 6.5%, and concentrate in TG (+18.3% and +18.2%). AV gains are 3.6% and 6.2%, averaging 4.9%. QA improves in every comparison by 0.7%–5.9%.

• Audio reliably helps QA, less reliably helps grounding. Adding audio raises CS_QA by 18.6%–28.1% across the four tested AV models (average 22.9%) and improves all 12 model-task comparisons when video is added over text-only. But CS_TG gains vary wildly — Gemini-3.8-Flash gains 29.4% and Qwen3.8-Omni-Flash 11.1%, while MiMo-V2.5 loses 1.5% — and CS_SG changes stay within −0.4% to +1.8%.

• Text alone is nearly useless for grounding. With all media removed (Text-only setting), CS_TG is only 0.1–0.8%, versus 18.8%–30.1% for CS_QA.

Methodology in Plain English

The researchers built the benchmark in three stages.

Stage 1 — finding videos worth annotating. They started from a pool of 8,489 candidate videos drawn from nine public datasets and added 240 self-collected videos from films, TV series, variety shows, and news programs. They used Qwen3-VL-8B-Instruct to profile each public candidate for four kinds of referential cues: entity, action, attribute, and scene. Videos shorter than 30 seconds or missing any of the four cue types were dropped, leaving 6,946 eligible videos. These were ranked on reference necessity, grounding suitability, temporal-spatial usefulness, and evidence diversity, then filtered by domain, duration, source, and subcategory constraints. Manual review kept 580 high-quality videos, which combined with the 240 self-collected ones gives the 820-video VL track. The same 240 videos form the AV track, with extra audio-visual annotations.

Stage 2 — annotating tasks in pairs. Four professional annotators received two days of task-specific training, then ran a pilot on 30 VL and 30 AV videos spanning all seven domains and three duration types, producing 180 task instances. The pilot surfaced disagreements about target pairing, temporal boundaries, spatial target consistency, and QA evidence, which the lead expert used to tighten the guidelines. Every annotation comes as a matched Text/Text+Image pair that shares the same video, target, task semantics, and ground truth: in the Text+Image version, the text is deliberately rewritten so the reference image is necessary to identify the target. Reference images are either video crops (preferably from a view outside the answer interval) or semantically matched external images, chosen so they never leak the event or the answer. TG intervals must tightly cover the complete target event; SG boxes localize the same target at both interval boundaries. QA follows CG-Bench's format — a clue interval plus five mutually exclusive options, one correct and four plausible distractors.

Stage 3 — checking the work. Four annotators cross-verified each other's cases against the stated principles, checking event completeness, box alignment at boundary frames, and the uniqueness and evidential support of QA answers; for AV cases they additionally checked native-audio dependence and audio-visual alignment. The lead expert then adjudicated remaining inconsistencies and performed final acceptance.

Scoring. For each grounded task, the CS score takes the minimum of three quantities: how well the Text prediction matches ground truth, how well the Text+Image prediction matches ground truth, and how well the two predictions match each other. For QA, a pair scores only if both predictions equal the ground truth. CS_Avg averages the three CS scores.

Running the models. Models receive uniformly sampled, timestamped frames — 64 for proprietary and >100B open-source models, 32 for smaller ones, 49 for GPT-series models (to fit the reference image under a 50-image limit), and 16 for VITA-1.5-7B per its official limit. The AV track adds the synchronized audio. The reference image, when present, appears after the video inputs and immediately before the task prompt.

Why This Matters

Impact on research. The paper reframes video benchmark reporting: instead of a single aggregate score, it separates when a model fails (temporal retrieval vs. spatial localization vs. semantic understanding) and whether it fails consistently across target specifications. The CS metric is the notable methodological contribution — it closes the loophole where a model can look accurate on average while answering different questions correctly in each setting. The finding that visual references help only 11 of 36 models, and help proprietary models far more than open-source ones, is a concrete challenge to the assumption that more reference modalities always help.

Real-world applications (these follow from the domains the benchmark covers — sports, traffic, surveillance, documentary, movie, TV, variety, and news):

• Media archive search and editing. Finding the exact moment a specific player, anchor, or character appears, and the region of frame they occupy, in hours of footage — the benchmark's news and sports content directly mirrors this.
• Broadcast monitoring and compliance. Checking what was said on air against what was shown on screen, which requires exactly the audio-visual event binding the open-source models struggle with.
• Surveillance and traffic analysis. Temporal and spatial grounding over surveillance and traffic footage is one of the benchmark's domains, and it demands both precise intervals and boxes.
• Accessibility tooling. Generating audio descriptions or captions that reference the right entity at the right timestamp depends on the same target identification and grounding skills.

Industry relevance. The 32.8% CS_Avg gap between proprietary and open-source models on the audio-visual track is large enough to matter for anyone choosing a model to deploy. The specific weakness in Event Understanding — 38.9% behind proprietary models — identifies the capability that needs work before open audio-visual models are usable in production. The benchmark was supported by the PolyU-OPPO Joint Innovative Research Center, indicating direct industrial interest in evaluating these capabilities.

Future Directions

• Audio-aware grounding. Audio consistently boosts QA (average +22.9%) but helps temporal grounding inconsistently — improving by 29.4% for one model and hurting another by 1.5%. Understanding when audio carries localizable temporal signal, and building models that exploit it reliably, is the clearest open problem.
• Training for cross-specification consistency. The 4.3%–6.4% consistency deficit across 36 VL models shows that single-setting accuracy does not transfer. Whether consistency can be trained for directly — rather than emerging as a byproduct of accuracy — is untested here.
• Closing the reference-image gap. Visual references help 8 of 12 proprietary models but only 3 of 24 open-source models, and can cost QA accuracy by omitting useful textual detail. How to fuse a reference image with a text query without discarding either signal remains unresolved.
• Scaling and broadening the audio-visual track. The AV track has 240 videos and 720 instances against the VL track's 820 and 2,460, and covers three perception plus two reasoning QA categories rather than five plus three. Extending audio-visual coverage, and testing whether the observed gaps persist at larger scale, is a natural next step.

Target Audience

Researchers building or evaluating multimodal large language models, particularly those working on video understanding, temporal and spatial grounding, and audio-visual reasoning. It is also directly useful for benchmark designers interested in consistency-aware metrics, and for practitioners choosing between proprietary and open-source models for video-heavy applications — the per-task, per-category breakdown is far more actionable for a deployment decision than a single leaderboard number. Readers new to the area will find the four-dimension comparison table (target specification, video evidence, task coverage, video duration) a compact map of the benchmark landscape.

Authors’ abstract

With the rapid development of multimodal large language models (MLLMs), evaluating their video understanding capabilities has become increasingly important. Existing video understanding benchmarks cover only part of the evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding. We introduce OmniCompass, a holistic benchmark for omni-modal video grounding and understanding. OmniCompass defines two evaluation tracks: OmniCompass-VL assesses vision-language models on tasks requiring visual evidence, while OmniCompass-AV assesses omni-modal models on tasks requiring joint audio-visual evidence. Each track includes three complementary tasks: temporal grounding (TG), spatial grounding (SG), and question answering (QA), assessing when a referenced target appears, where it is, and what happens. QA covers five perception and three reasoning categories in OmniCompass-VL, and three perception and two reasoning categories in OmniCompass-AV. OmniCompass-VL contains 820 videos and 2,460 instances across domains and three duration types: short, medium, and long. A narrative- and audio-rich subset forms OmniCompass-AV with 240 videos and 720 instances. Each instance is evaluated under two paired settings: text-only and text+image, specifying the same target through text and visual-language guidance. All instances are manually annotated with multi-round quality verification. Evaluations of 50 MLLMs show that open-source models still lag behind proprietary counterparts, particularly in audio-visual understanding. Precise temporal and spatial grounding remains challenging even for leading models, and their predictions still exhibit inconsistencies across the paired target specifications.