SleuthVLM: Do We Really Need Audio-in-Video for Omni Video Understanding?

AI summary

Overview

Research area: Video–language understanding, specifically "omni" (audio-visual) video question answering and LLM-based agent systems.

Technical level: Intermediate. The paper is a systems-and-benchmark contribution rather than a new neural architecture; it requires familiarity with vision-language models (VLMs), automatic speech recognition (ASR), and tool-using agents, but the core idea is easy to grasp.

Scope (one sentence): The paper asks whether an audio-visual video question-answering system must actually process audio inside the model, and shows that a VLM agent working over a timestamped speech transcript plus a small set of video-inspection tools outperforms omni models that hear the audio directly.

What This Paper Is About

Omni models answer questions about a video in a single turn by jointly processing sampled frames and the audio track, and the field has mainly progressed by scaling model size and collecting more paired audio-visual training data. The authors challenge both assumptions: most of the audio that questions depend on is speech (they report that 75.2% of LVOmniBench questions involve speech), which can be converted to timestamped text cheaply, and a single fixed-rate pass over a long video cannot go back to the few seconds that actually contain the answer.

Their goal is therefore to build a system that never hears the audio directly, but instead "sleuths" the video — searching a word-timestamped transcript as an index, then zooming into the relevant moments with tools — and to measure how far this approach gets against state-of-the-art omni models and against other agents.

Key Contributions

1. An agent harness that removes the audio encoder entirely. SleuthVLM pairs any VLM with a once-per-video ASR transcript carrying word-level timestamps (plus recorded gaps where nobody speaks), and a small tool set (bash, view, watch, answer, with vsheet, vframes, vcrop and asr inside bash). The model is given only its resources, a brief operating guide, and a budget — no predefined workflow — and decides where to look, how closely to inspect, and when to stop.

2. A new state of the art on omni video benchmarks without direct audio access. With Claude Opus 5.5 as the reasoning model, SleuthVLM reaches 75.0% average accuracy over Daily-Omni, WorldSense, OmniVideoBench and LVOmniBench, 2.6 points above Gemini 3.8 Flash, and leads the Video-MME-v2 leaderboard at 77.8% accuracy with a non-linear score of 63.5.

3. Evidence that a small open model plus agency is competitive. Using Qwen3.8-27B on a single GPU, SleuthVLM averages 66.8% across the four benchmarks, 13.8 points above the strongest open-source omni model and 16.8 points above the similarly sized Qwen3-Omni-30B-A3B.

4. A controlled study of what actually drives the gains, including ablations that remove the transcript (−17.4 points on Daily-Omni, −16.6 on LVOmniBench) or only its word times (−1.3 on both), and a comparison in which omni models are given the same harness and even an audio-listening tool yet still trail the VLM agent.

Main Findings

• Searching beats passive observation. Running the same omni models as agents in the harness improves their accuracy by up to 10.3 points; SleuthVLM adapts its search per question, using an average of 5.8 turns on Daily-Omni and 17.0 turns on LVOmniBench (Qwen3.8-27B, medium effort). A simple general-purpose harness outperforms agents that delegate perception to task-specific tools backed by other models; OmniAgent, for example, uses seven such tools.

• The transcript does most of the heavy lifting, and doubles as a temporal index. Removing it drops accuracy by 17.4 and 16.6 points on the two ablation benchmarks. Even LVOmniBench sound questions lose 10.5 points without it, because the transcript's silence gaps and word boundaries tell the model where to look as well as what was said.

• Word-level timestamps add a small, free gain. Dropping them costs 1.3 points on both benchmarks at no extra turns, mostly on music and sound questions (6.6 and 6.1 points), whose timing is inferred from the boundaries of surrounding words.

• Agentic ability matters more than direct audio access. When SleuthVLM is run with omni models as the reasoning model and given a listen_audio tool, it still trails the Opus 5.5 VLM version. On LVOmniBench, SleuthVLM (Opus 5.5) scores 75.5, SleuthVLM (Gemini 3.8 Flash, with audio) 74.4, and SleuthVLM (Qwen3.8-Omni-Flash, with audio) 70.6, against 63.3 and 70.7 for those omni models run without the harness. The largest gap appears on logical reasoning (82.6 for Opus 5.5 versus 67.9 for Qwen3.8-Omni-Flash).

• A single case study illustrates the agentic gap. Given the same question — what the host was saying when a viewer first commented, at 05:19 of an 83-minute stream — Opus 5.5 binds the comment to the sentence "same category of business" in five turns. Gemini 3.8 Flash finds both the comment and the word "business" at 05:18.8, then rescans the whole video, listens to a window that ends before the comment, and answers after 27 turns using 21 times the tokens.

• A small agentic model is competitive with closed-source omni models. SleuthVLM with Qwen3.8-27B outperforms closed-source Qwen3.5-Omni-Plus on average, and swapping in Opus 5.5 adds a further 8.2 points, showing the harness scales with the reasoning model.

• An optional audio tool covers non-speech content. An audio_caption tool backed by Ming-3.0-audio-tiny (9.3B parameters, 2.7B active) adds 9.8 points on the 448 LVOmniBench questions about music, while the reasoning model remains a VLM.

• Concurrent agent designs trail. SleuthVLM exceeds Omni-Decision (a GPT-5.2 planner over Gemini-3.1-Pro perception) by 11.8 points on WorldSense (70.1 versus 58.3) and the best sandboxed coding agent by 15.5 points on LVOmniBench.

Methodology in Plain English

The design rests on three moves.

Turn audio into timed text once. For every video, an ASR model (Qwen3-ASR-1.7B) transcribes the speech and a forced aligner (Qwen3-ForcedAligner-0.6B) assigns each word a start and end time. Stretches of at least 3 seconds with no words are kept as gaps. The result is stored as two plain files in the agent's sandbox: asr.txt, listing sentences and gaps in time order, and asr_words.tsv, listing individual words with their times. This transcription happens independently of the question, once per video; the authors report that transcribing the 3,249 videos of the four benchmarks — about 302 hours of audio — takes roughly 113 minutes on a single H200.

Give a VLM tools instead of audio. The model sees a muted copy of the video and a small command set. It can render a contact sheet of time-stamped frames over an interval (vsheet), extract frames at a chosen rate (vframes), crop a native-resolution frame (vcrop), search the transcript or list word times and gaps (asr), and use ordinary grep and python3. Rules keep it inside the sandbox: relative paths only, no network, no package or process commands, no audio extraction. A budget of 50 turns, 30 images, 10 clips and 30 minutes per question caps the effort, and the model must eventually call answer.

Let the model plan. The system prompt contains only a scope definition and a short suggested operating sequence — read the question and transcript, get an overview from one contact sheet, localize a window from transcript hits and gaps, zoom in, verify, answer. Nothing forces those steps. In practice the model uses transcript hits to jump to a moment, then narrows from coarse to fine intervals, viewing far fewer frames than a uniform one-frame-per-second sweep would require. When the reasoning model is itself an omni model, the harness adds one extra tool, listen_audio, capped at 4 calls on windows of at most 30 seconds.

Everything else is measurement: comparisons against reported scores of closed- and open-source omni models and agents on four benchmarks, plus Video-MME-v2, followed by ablations that strip the transcript or the word times to see what each part contributes.

Why This Matters

The paper reframes a problem the field has been treating as a modality-alignment problem. If a VLM with a transcript and tools beats models trained on large paired audio-visual corpora, then the bottleneck in omni video understanding is not the audio encoder but the agent's ability to plan, search, and bind evidence across time. That has direct consequences for how research effort is allocated: it argues that agentic capability should be developed alongside, or instead of, more omni training data, and it shows that progress can come from a harness that plugs into any existing VLM.

It also matters practically, because the approach is cheap and portable. A 27B open model on a single GPU lands within range of closed-source omni models, and the entire audio pipeline is a one-off transcription pass rather than a retrained multimodal backbone.

Real-world applications implied by the approach:

• Long-form media and broadcast analysis — searching hours of talk shows, streams, or podcasts for a specific remark and the moment it was said, as in the paper's 83-minute stream case study.
• Meeting, lecture, and interview review — locating where a topic was raised and then visually checking what was on screen at that moment.
• Video archive and compliance search — using a transcript as an index into footage that would be far too long to scan frame by frame.
• Assistive and accessible video navigation — timestamped transcripts plus targeted frame inspection as a way to answer questions about content without processing the full audio stream.

Industry relevance: The result challenges the current scaling recipe for omni models (add an audio encoder, train on paired data, increase size). For teams building video products, it suggests a deployment path that is cheaper than training an omni model and can be upgraded simply by swapping the reasoning model — the paper notes the same harness with Opus 5.5 adds 8.2 points over Qwen3.8-27B without changing anything else.

Future Directions

• Better agentic training for omni models. The paper's second finding is that omni models lose planning and evidence-binding ability relative to strong VLMs, and that giving them audio does not close the gap. The obvious next question is whether training for agency rather than modality alignment could let omni models use their audio encoder productively.

• Closing the non-speech gap. Roughly a quarter of LVOmniBench questions are about music or sound, and the optional audio_caption tool added 9.8 points on music questions. How far a general-purpose audio tool can be pushed, and whether it should be part of the default harness rather than an add-on, is left open.

• Extending the harness beyond multiple-choice question answering. Every benchmark in the paper is multiple choice and bounded by a fixed budget of 50 turns, 30 images, 10 clips and 30 minutes per question. Whether the same free-form sleuthing generalizes to open-ended description, grounding, or summarization tasks is untested here.

• Improving efficiency and search policy. The medium-effort Qwen3.8-27B averages 5.8 turns on Daily-Omni and 17.0 on LVOmniBench, and the Gemini 3.8 Flash case study burned 21 times the tokens of Opus 5.5 on one question. Learning when to stop, and how to search more sample-efficiently, is an open engineering question.

Target Audience

Researchers and engineers working on video-language models, omni/multimodal models, and tool-using agents will get the most from this paper, particularly those deciding whether to invest in audio encoders and paired audio-visual training data. It is also useful for practitioners building long-video retrieval or question-answering systems who want a strong, low-cost baseline that runs on a single GPU with an open 27B model. Readers need some familiarity with vision-language models and agent tooling, but the paper's argument is readable without a background in audio modeling.

Authors’ abstract

Omni models answer questions about videos by jointly processing sampled frames and audio in one turn, with progress driven largely by scaling model size and omni training data. We ask whether a model needs to hear the audio directly at all, and present SleuthVLM, an agent harness built on a VLM that never hears the audio directly: speech will be transcribed into text, and the VLM will sleuth the video on its own without any predefined workflow. On four omni benchmarks with videos up to 90 minutes, SleuthVLM achieves the best average accuracy: \textbf{75.0%}, 2.6 points above Gemini 3.8 Flash, and it also tops the Video-MME-v2 leaderboard with \textbf{77.8%}; with an open-source 27B VLM, it beats all open-source omni models. We draw three findings: SleuthVLM beats a model that glances; hearing is not the bottleneck, agentic ability is; omni understanding needs better agentic ability, not just more omni data for modality alignment. So we provide a simple but effective starting point for doing omni video understanding. Omni understanding needs better agents, not just bigger models that hear.