Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams

AI summary

Overview

Research area: Multimodal AI evaluation — specifically benchmarks for omnimodal large language models (Omni-LLMs) that observe continuous audio-video streams and must decide when and how to intervene.

Technical level: Advanced. The paper assumes familiarity with streaming/duplex model architectures, tool-calling interfaces, causal prefix evaluation, and temporal scoring protocols.

Scope (one sentence): The paper introduces Omni-InteractScope, a human-annotated benchmark of 1,479 interactions and 1,790 temporal targets across eight task groups, and uses it to evaluate 12 model configurations on whether they intervene at the right moment, with the right content, and across entire response sequences.

What This Paper Is About

Assistants built on omnimodal models are increasingly expected to watch a live audio-video stream and act on the user's behalf — call a tool when a specific event happens, warn about a hazard, or keep giving guidance as an activity unfolds. Existing benchmarks mostly test whether a model can answer a question about a video, not whether it connects stream evidence to a user's need at a useful time, and then keeps responding while that need persists.

Omni-InteractScope addresses this gap by recording, for each interaction, the required response content and the time or valid interval in which it is due, including multiple ordered targets when one standing request demands successive responses. The goal is to measure three things separately: whether the model produces the right content, whether it produces it on time, and whether it covers every obligation in an interaction rather than just the first one.

Key Contributions

1. A benchmark for timely and sustained intervention. 1,479 human-annotated, cross-reviewed interactions across eight task groups, containing 1,790 targets that each specify what to produce and when — including successive targets retained in temporal order when a user need continues (163 interactions have multiple targets).

2. Focused evaluation of two proactive demands with paired controls. 200 event-timed tool-invocation targets (T1) and 122 query-free reminders (T8-A). Diagnostics separate tool choice from timing, test whether invocation scores depend on scene evidence, and examine how explicit event cues, response scheduling, history, and response frequency change coverage on the same reminder episodes.

3. Explicit modeling of continuing obligations. The benchmark distinguishes individual-target accuracy from complete-sequence coverage, reporting both step-level recall and how many of the 26 multi-step guidance sequences (T8-C) a model covers in full.

4. A 12-model empirical study spanning proprietary, open-source prompted, and streaming configurations, exposing gaps between tool selection and timely action, between question answering and query-free reminders, and between step coverage and complete guidance.

Main Findings

• Timely tool invocation is unreliable across the board. All 12 evaluated models produce an on-time first response for at most 40% of tool-invocation targets, and joint timing-and-tool accuracy reaches at most 32.0% (Qwen3.5-Omni-Plus, which leads T1).

• Query-free reminders are poorly covered. Every configuration scores lower timely reference-target recall on query-free reminders (T8-A) than on explicit audiovisual questions (T8-B), measured on their respective target sets. Reminder recall never exceeds 23.0%. Even Gemini-3.1-Pro, the leader on both, reaches 22.95% on reminders versus 62.25% on questions.

• High step recall does not imply complete guidance. Gemini-3.1-Pro covers 51.72% of T8-C steps but completes only 3 of 26 guidance sequences; Gemini-3.5-Flash completes 2 and Qwen3.5-Omni-Plus 1. Nine models complete none, and no open-source prompted or streaming configuration completes a single sequence.

• Aggregate rankings hide severe task-level failures. The best scores on cumulative-state tracking (T4) and continuous responses (T7) are only 11.2% and 11.4%. Gemini-3.5-Flash responds prematurely on 172/206 T4 targets, and its T7 failures include 233/457 premature responses and 152/457 timely content failures.

• Tool-invocation scores alone do not establish event grounding. Removing scene input did not lower Gemini-3.5-Flash's T1 joint accuracy on the same 200 interactions: it was 29.50% with scene input and 33.00% without, a paired difference of −3.50 percentage points. A request-only control selects 96.5% of tools from request text and public definitions alone, and emitting those choices after a fixed three-second delay yields 68.0% joint success — the highest of four tested delays.

• Reminder coverage depends on response policy. Giving Gemini-3.5-Flash access to the five most recent responses reduces output volume by 58.1% while both conditions cover 22 of 122 T8-A targets; the matched targets differ, with 12 gained and 12 lost. Adding a ten-second cooldown reduces outputs further but costs 9.84 percentage points of recall relative to history alone.

• Event cues help mainly when the response time is supplied. Under scheduled responding at an annotated valid time, Gemini-3.5-Flash and Qwen2.5-Omni benefit from event-specific instructions; under autonomous responding the effects are small and uncertain.

• Low sequence completion is consistent with independent step failures. On the same 20 three-step T8-C episodes, Gemini-3.1-Pro covers 30 of 60 steps and completes three sequences, against an expected 2.4 completions if successes at each step position are independent — so the result does not establish a separate memory or planning failure.

Methodology in Plain English

The researchers built each benchmark item as an interaction: a video-plus-audio stream, a user context, and one or more reference interventions. Annotators recorded the request or general assistance need, the evidence that makes a response due, the required response content, and the time or valid interval in which that response counts. A separate group of reviewers cross-checked both the content and its temporal relation to the event before release.

The data come from two panels. Explicit-request tasks draw on 905 online video records (888 from Douyin, 11 from Xiaohongshu, six from Bilibili; 495 first-person and 410 third-person). An impairment-assistance panel of 299 interactions uses 207 self-recorded first-person videos, split into 261 visual-impairment and 38 hearing-impairment interactions. Clips span indoor (37.4%), outdoor (34.3%), and digital-screen (28.1%) settings, with a median duration of 2.16 minutes and a 90th percentile of 6.15 minutes.

Models were given causal inputs — only what would be available at that moment — at query time, at each subsequent integer second of media time, and at clip end, on a schedule that does not depend on where the targets are. Prompted models made independent prefix calls with no access to their own earlier answers (Offline Probing); streaming models kept a native session state across observations (Online Rolling), with initialization outputs excluded. Scoring combines a content judgment with a timing check: T1 requires an exact match of the tool-name set, T2–T8 use a binary GPT-4o judge that accepts equivalent wording, and timing is checked separately, with a two-second tolerance for T1–T7 and T8-B/C and the annotated valid interval for T8-A. For T1–T7 only the first response assigned to a target counts, so a premature or wrong answer cannot be rescued later; T8 uses ordered matching that maximizes content-time matches while using each response at most once.

Beyond the main leaderboard, the authors ran targeted controls: a scene-input ablation (the model sees the request but not the video), a request-only tool-selection control, history and cooldown manipulations on the same reminder episodes, event-specific versus general instruction runs, and a comparison of observed sequence completion against an independence-based reference.

Why This Matters

Impact on research. The paper makes a methodological point that reaches beyond its own leaderboard: a tool-and-time accuracy number can be sustained by request priors and a fixed response delay without the model using the scene at all. That argues for pairing proactive-interaction scores with evidence controls, and for reporting how much response history and how many generation opportunities a model was given, since both change coverage. The distinction between step-level recall and complete-sequence coverage gives the field a second axis for judging assistants that carry continuing obligations.

Real-world applications:

• Assistive technology for blind and low-vision users — the T8-A and T8-C panels cover visual-assistance reminders and multi-step guidance delivered without an event-specific question.
• Hearing assistance and sound awareness — a sub-panel of 38 hearing-impairment interactions tests relaying relevant environmental audio under a general instruction.
• Safety and error monitoring — T3 requires a warning when an unsafe or incorrect action occurs, not merely an announcement that the event happened.
• Streaming tool-using agents — T1 measures whether an assistant calls the right tool at the right moment during a live stream, the core loop of an always-on agent.

Industry relevance. The 12-model panel spans proprietary systems (Gemini-3.1-Pro, Gemini-3.5-Flash, Qwen3.5-Omni-Plus), open-source prompted models (Qwen3-Omni, Qwen2.5-Omni, Qwen3-VL, InternVL3.5), and purpose-built streaming models (LiveCC, MiniCPM-o 4.5, MMDuet2, MOSS-VL, AURA). The results show that streaming architectures do not automatically win: the best T1 score among streaming models in the reported table is 1.5%, and top overall assistance recall comes from a proprietary prompted model. For teams shipping always-on assistants, this is a direct signal that current proactivity is far from deployable.

Future Directions

• Stronger evidence controls for tool use. The authors note that the scene ablation is a weak test and suggest varying event occurrence and timing while holding requests fixed — a stronger requirement than removing the video entirely.

• Evaluating selectivity, not just recall. Because unmatched T8 outputs incur no penalty, the current reminder metric does not measure interruption cost. The paper calls for evaluating the value and cost of interventions beyond the annotated targets.

• Explicit records of delivered guidance and pending obligations. The authors frame tracking of what has been said and what remains owed as a design hypothesis to be tested through interventions on history and state, rather than an established mechanism.

• Closed-loop user-task success. Complete T8-C measures coverage of an annotated trajectory, not whether the user actually completed their task; extending evaluation to downstream task outcomes is left open.

• Isolating planning from scheduling. The current scheduled-versus-autonomous comparison changes temporal information and response opportunities together, so it cannot attribute differences to a planning mechanism.

Target Audience

Researchers and engineers working on streaming or duplex multimodal models, proactive agents, and tool-calling assistants who need a rigorous way to measure when a model acts, not just what it says. It is also valuable for practitioners building assistive technology for blind, low-vision, and hard-of-hearing users, and for benchmark designers interested in how evaluation protocol choices — observation schedule, response history, timing tolerance — shape the conclusions drawn from a leaderboard. Readers without background in streaming evaluation will find the setup sections accessible but the results tables dense.

Authors’ abstract

Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams. However, existing evaluations provide limited insight into how reliably these models connect stream evidence and user needs to timely, sustained interventions. We introduce \textbf{Omni-InteractScope}, a benchmark of 1,479 human-annotated, cross-reviewed interactions with 1,790 temporal targets across eight task groups. It links input modalities, event triggers, intervention types, and user needs, covering event-timed tool invocation and query-free assistance alongside notifications, warnings, state tracking, and continuous responses. Our experimental results on the proposed benchmark reveal three critical limitations of current Omni-LLMs: (1) \textbf{Timely tool invocation remains unreliable}: all 12 evaluated models produce an on-time first response for at most 40\% of tool-invocation targets, and joint timing-and-tool accuracy reaches at most 32.0\%. (2) \textbf{Query-free reminders remain poorly covered}: all evaluated models achieve lower timely reference-target recall on query-free reminders than on explicit questions, measured on their respective target sets. Reminder recall never exceeds 23.0\%. (3) \textbf{End-to-end guidance coverage remains limited}: nine models cover none of the 26 guidance sequences in full, and the remaining three cover only one to three sequences each. Targeted controls further show that tool-invocation scores alone do not establish event grounding and that reminder coverage depends on response history and frequency. These findings motivate developing and evaluating streaming assistants for timely action, initiative under general user needs, and coverage of entire response sequences.