SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

AI summary

Overview

Research area: Multimodal AI evaluation — benchmarking Multimodal Large Language Models (MLLMs) on joint audio-video understanding, with a fairness lens.

Technical level: Intermediate. The paper is readable for someone familiar with basic machine-learning evaluation concepts (accuracy, IoU, ROUGE), but the task design and fairness analysis assume some benchmark-literacy.

Scope: The paper introduces SONIC-O1, an open, human-verified benchmark of ≈60 hours of real-world audio-video conversation (231 clips, 4,958 annotations across 13 topics) used to evaluate MLLMs on summarization, multiple-choice QA, and temporal localization, including group-wise analysis by race, gender, and age.

What This Paper Is About

Most multimodal benchmarks focus on static images or treat audio as optional (or replace it with text transcripts), and they rarely report how model performance varies across demographic groups. SONIC-O1 addresses both gaps by building a fully human-verified audio-video benchmark from real conversational recordings in high-stakes domains, where models must jointly reason over audio, video, and text, and where results can be sliced by perceived demographic metadata. The goal is to reveal where current MLLMs succeed, fail, and diverge across demographic groups.

Key Contributions

1. A new open benchmark. SONIC-O1 is an open-source, human-verified benchmark with domain-expert annotations and demographic metadata, built from ≈60 hours of real-world audio-video interaction (231 clips, 4,958 QA instances) spanning 13 topics across five domains.
2. Three complementary tasks. The benchmark evaluates (i) open-ended video summarization, (ii) evidence-grounded multiple-choice QA with rationales, and (iii) IoU-grounded temporal localization of event start/end times with supporting reasoning.
3. Systematic model benchmarking. The authors evaluate both closed-source and open-source omnimodal MLLMs, quantifying performance gaps between model families and across task types.
4. Group-wise fairness analysis. The benchmark enables demographic slice analysis (6 racial groups, genders, and age bands), exposing disparities that aggregate scores hide, plus a released evaluation suite (dataset, scripts, leaderboard).

Main Findings

• Closed-source models lead across the board. Gemini 3.0 Pro posts the highest results on all three tasks: 81.4% MCQ accuracy, a 7.07 LLM-judge summarization score, and 25.4% R@0.5 on temporal localization.
• Temporal localization is the hardest task, and the family gap is largest there. The best closed-source model outperforms the best open-source model by 22.6% on temporal localization (the results section phrases this as a gap of about 23%); MCQ accuracy shows the smallest gap between model families.
• Qwen3-Omni is the strongest open-source model, reaching 65.7% MCQ accuracy and a 5.72 summarization judge score.
• Scale does not strictly predict closed-ended reasoning. Baichuan-Omni 1.5 and OLA (both 7B) reach 56.2% and 53.2% MCQ accuracy, above the larger UniMoE-2.0 (33B, 51.6%). Scale still matters for generation and grounding: smaller models show up to 41.6% lower summarization judge scores than Qwen3-Omni and 6.7× lower R@0.5 on temporal localization.
• Demographic disparities are most visible in temporal localization. Accuracy gaps of up to 21.4% on temporal localization exist across demographic groups; as an example, Gemini's R@0.5 by race ranges from 40.9% (Indigenous) to 19.5% (Black). MCQ accuracy stays relatively stable across groups. Mann-Whitney U tests confirm the disparities are statistically significant.
• Audio genuinely helps. Adding audio to video consistently improves performance; Qwen3-Omni gains +10.3% MCQ accuracy, +2.38 summarization score, and +2.7% R@0.5. GPT-4o gains 8.8% MCQ accuracy when text transcripts are added. VideoLLaMA2 shows slight summarization degradation (−0.13) even with no MCQ change.
• More frames is not a universal win. Increasing input frames (16, 32, 64, 128) leaves summarization and MCQ largely unchanged, but improves temporal localization, mainly for stronger models such as Qwen3-Omni; weaker models like VideoLLaMA2 show inconsistent trends.
• Longer videos hurt, especially temporal grounding. Gemini's temporal R@0.5 falls from 50.6 (short) to 26.0 (medium) to 18.6 (long), while its MCQ accuracy stays comparatively flat (81.9 / 82.8 / 79.5).
• High-stakes topics remain hardest. Emergency response and mental health are the most challenging domains, while some professional topics (consultations, interviews) are comparatively easier.
• Models have distinct emotional-language defaults. Using LIWC-22, Baichuan-Omni 1.5 shows the highest tone (60.09), followed by OLA (60.00), while Gemini 3.0 Pro adopts a neutral, clinical style (tone 46.16, total emotion 4.35%).

Methodology in Plain English

The researchers collected real-world conversation videos from YouTube restricted to CC BY 4.0 licenses, using topic-specific queries expanded with demographic descriptors. From an initial pool of 2,237 candidates, 1,794 met licensing constraints, and manual quality filtering produced 231 final videos totaling roughly 60 hours. Videos were stratified into short (<5 minutes), medium (5–20 minutes), and long (20–60 minutes) bins.

Three tasks were built on the same video pool. For summarization, long videos were chunked into 10-minute segments, summarized, and merged into a reference summary (300-word limit for model outputs). For MCQ, videos were split into 3-minute windows with 30-second overlap, and each segment got a five-choice question (four candidates plus a "Not enough evidence" option) paired with a required rationale. For temporal localization, an anchor-target formulation asks models to predict a target event's start and end times plus supporting reasoning.

Draft annotations and perceived demographic metadata were bootstrapped with Gemini 2.5 Flash and then verified, corrected, or removed by domain experts using an internal review interface; ambiguous or low-evidence items were dropped and disagreements resolved by consensus. Evaluation compares closed-source (Gemini 3.0 Pro, plus GPT-4o for ablation) and open-source models (Qwen3-Omni-30B-A3B, Uni-MoE-2.0-Omni 33B, MiniCPM-o-2.6 9B, Baichuan-Omni 1.5 7B, OLA 7B, VITA-1.5 8B, VideoLLaMA2 7B), with open-source models running at their optimal configurations up to 256 frames and Gemini processing at 1 FPS. Metrics include ROUGE-L, cosine similarity, and a GPT-5-mini LLM-judge score (0–10) for summarization; accuracy and rationale-judge scores for MCQ; and mIoU plus R@0.5 for temporal localization, all macro-averaged across topics.

Why This Matters

Impact on research. SONIC-O1 pushes multimodal evaluation past static images and transcript-only setups toward genuine audio-video reasoning, and it adds a demographic axis that most prior audio-video benchmarks omit. Its public leaderboard and modular evaluation suite give the community a reproducible target for measuring temporal grounding and fairness together, rather than one at a time.

Real-world applications:

• Healthcare and clinical communication — patient-doctor consultation understanding, where affect and timing of a diagnosis relative to symptoms matter.
• Education and training — summarizing and indexing recorded instruction, interviews, and panel discussions.
• Legal and civic processes — courtroom proceedings and hearings, where accurately timestamping who said what, and when, is essential.
• Public safety and service sectors — emergency response and service-oriented interactions, which the paper identifies as the most challenging domains for current models.

Industry relevance. The finding that audio input improves performance, that more video frames mainly help temporal localization, and that performance degrades on longer videos gives concrete guidance for how practitioners should configure multimodal pipelines, budget frames, and segment long recordings. The reported demographic gaps are directly relevant to teams deploying these models in high-stakes settings, and the modality ablation with GPT-4o illustrates the cost of substituting transcripts for native audio.

Future Directions

1. Reducing temporal reference-frame hallucination. The paper notes open-source models often report timestamps relative to a segment's start rather than absolute time (e.g., 50–80s instead of 550–580s), indicating weak absolute temporal representation — a concrete target for future work.
2. Closing the temporal localization gap. Temporal localization is the weakest task for open-source models and the most affected by video length, so better long-horizon event tracking is an open problem.
3. Extending beyond English and beyond current scale. The authors cite English-only coverage, limited scale and task scope, and unequal demographic sample sizes as limitations that future versions should address.
4. Task-specific fairness evaluation. Because MCQ stays stable across groups while temporal localization does not, the paper argues fairness must be measured per task rather than assumed to transfer.

Target Audience

Researchers and engineers working on multimodal LLMs and video/audio understanding; benchmark designers interested in human-verified annotation pipelines and fairness-aware evaluation; practitioners deploying multimodal systems in healthcare, education, legal, or public-safety contexts who need to know where these models are reliable and where they are not.

Authors’ abstract

Multimodal Large Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, while their ability to process sequential audio-video data remains underexplored. This gap highlights the need for a high-quality benchmark to systematically evaluate MLLM performance in a real-world setting. We introduce SONIC-O1, a comprehensive, fully human-verified benchmark of 60 hours (231 clips) spanning 13 real-world conversational domains with 4,958 annotations and demographic metadata. SONIC-O1 evaluates three capabilities: open-ended summarization, multiple-choice question (MCQ) answering, and temporal localization with supporting rationales (reasoning). Across closed- and open-source models, we find that the MCQ accuracy shows the smallest gap between model families, but the best closed-source model outperforms the best open-source model by 22.6% on temporal localization. We further observe accuracy gaps of up to 21.4% on temporal localization across demographic groups, indicating persistent disparities in model behaviour. SONIC-O1 provides an open evaluation suite for temporally grounded and demographically robust multimodal understanding. SONIC-O1 is publicly available for research: Project page (https://vectorinstitute.github.io/sonic-o1/), Dataset (https://huggingface.co/datasets/vector-institute/sonic-o1), GitHub (https://github.com/vectorinstitute/sonic-o1), Leaderboard (https://huggingface.co/spaces/vector-institute/sonic-o1-leaderboard).