AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding

AI summary

Overview

Research area: Computer Vision / multimodal large language models (MLLMs) — specifically streaming video understanding and proactive human–model interaction.

Technical level: Advanced. The paper formalizes interaction as a partially observable Markov decision process and introduces a custom timing-aware metric (Ask-F1), so some familiarity with benchmark design and video-language models helps, though the core idea is intuitive.

Scope (one sentence): The paper introduces AskStream, the first multi-turn benchmark that adds a third action — ask — to the existing wait and answer actions of proactive video models, and measures whether models can ask for missing information at the right moment and with the right content when the video or the user leaves a query under-specified.

What This Paper Is About

Existing proactive video benchmarks assume the information needed to answer a query is fully present in the stream and will eventually appear, so models only have to decide whether to wait or answer. In reality information is often incomplete — the key entity is blurred, the viewpoint is wrong, the user's question is ambiguous or based on a false premise, or the user's preferences are unknown — which can make a model wait forever or hallucinate. AskStream tests whether models can instead proactively ask for exactly the missing information (from the video or from the user) and then answer correctly once it arrives.

Key Contributions

1. A new interaction paradigm. Quantitative and qualitative analysis (Section A.3) shows that acquiring the missing evidence lowers model uncertainty and shifts probability toward the correct answer, motivating an action space of three actions — WAIT, ASK, ANSWER — rather than the previous two.
2. The AskStream benchmark. A multi-turn benchmark spanning video uncertainty (explicit: Key Visual Evidence Missing, Viewpoint Information Missing; implicit: Individual Concept Missing, Personalized Action Concept Missing) and user uncertainty (intent: Ambiguous User Question, Wrong User Question; preference: User Preference Missing, User Preference Conflict) — four task categories and eight subtasks, 438 videos and 1,401 questions, with 2,714 manually annotated frame-level time windows.
3. Ask-F1 and StreamHarness. A new metric that jointly scores the timing and content correctness of both the ASK and the ANSWER via a harmonic mean, plus a hierarchical memory framework (StreamHarness) for evaluating offline models under streaming conditions.
4. Extensive experiments. Evaluation of offline closed-source, offline open-source, and streaming proactive models under both multiple-choice and open-ended settings, exposing a large gap between the two families.

Main Findings

• Proactive asking is largely unsolved. Across multiple-choice and open-ended settings, the strongest model (Gemini-3.8 Flash, offline) reaches only 52.18% Ask-F1, while the best streaming proactive model (AURA, on a Qwen3-VL-8B backbone) reaches only 7.22%.
• ASK is the bottleneck, not ANSWER. For stronger models the asking score S_ask is markedly lower than the answering score S_ans; Gemini-3.8 Flash scores 44.63 S_ask against 62.80 S_ans.
• Subtask performance is very uneven. Gemini-3.8 Flash reaches 81.36% on User Preference Conflict and 64.03% on User Preference Missing, but only 17.46% on Viewpoint Information Missing and 17.73% on Personalized Action Concept Missing.
• Early ASK is the dominant error type. Figure 5 shows early ASK as the main error category across offline models: models often sense that more information is needed but ask too soon, so when to ask is a major bottleneck.
• Oracle timing information helps most. Giving oracle information about when to ASK raises Ask-F1 from 37.17% to 58.30% (+21.13 points) on Key Visual Evidence Missing with Gemini-3.8 Flash in the multiple-choice setting; performance generally rises as more decision-stage uncertainty is removed, and oracle information about both when and what to ask improves Ask-F1 by 43.83%.
• Contextual memory helps only for user uncertainty. Keeping cross-turn history produces gains of 4.40 and 5.38 points on Ambiguous User Question and User Preference Missing, but has little effect on video-uncertainty tasks.
• More ASK attempts saturate quickly. Increasing the ASK budget from 1 to 16 improves performance but plateaus around 52.35% (Gemini-3.8 Flash, Key Visual Evidence Missing, open-ended setting).
• Missing evidence measurably hurts accuracy. In the uncertainty analysis with Qwen3.5 9B over 130 five-choice events from 78 videos, accuracy is 42.3% with blurred input, rises to 62.3% after asking for and receiving key evidence, and approaches 63.1% with fully clear input; mean gold-answer probability rises from 25.8% to 36 (value truncated in the provided text).
• Metric rankings are stable. Replacing the linear within-window temporal decay with logarithmic or exponential weighting changes absolute scores but leaves model ordering identical (Spearman ρ = 1.00 against the default linear metric).

Methodology in Plain English

The authors treat a streaming interaction as a sequence of events, each starting when a user query arrives and ending when the next one does. Inside each event they manually mark frame-level time windows for three actions: when the model should WAIT, when it may ASK, and when it should ANSWER. Two window patterns are defined — "wait before asking," where the uncertainty only becomes visible as the video unfolds (annotated with WAIT and ANSWER windows), and "ask before waiting," where the uncertainty is apparent immediately from the query (annotated with ASK and ANSWER windows).

To build the data, they collected videos from YouTube, existing datasets, and Blender-synthesized clips, then annotated queries, time windows, key entities, and required evidence on a custom webpage. They used Grounded SAM 2 to segment and track entities so they could selectively Gaussian-blur them, and PySceneDetect to assist with splitting videos; they also swapped viewpoints, rewrote queries, and withheld or altered personalized concepts and preferences. GPT-5.6 Sol generated the action and answer options, and humans verified that the answer is indeterminate without the missing information but determinate once it is supplied, and that each candidate set has exactly one correct option.

Evaluation uses Ask-F1, a harmonic mean of two timing-aware scores: S_ask (did the model ask the right thing inside the valid ASK window) and S_ans (conditional on a correct ASK, did it answer the right thing inside the valid ANSWER window). Actions made earlier within a valid window receive more credit than later ones. Models act at most once per second with a ±1 second tolerance, and at most one ASK is allowed per turn in the main setting. Offline models are run through StreamHarness, which keeps the 10 most recent frames (1 fps) as short-term memory, summarizes every 10 frames into one mid-term memory, consolidates every four mid-term memories into a long-term block, and keeps up to eight such blocks; native streaming models use their own memory mechanisms. Open-ended answers are graded by Gemini-3.8 Flash as judge.

Why This Matters

Impact on research. AskStream reframes proactive video understanding from "when to answer" to "when and what to ask," and supplies the annotation scheme, metric, and evaluation harness needed to study that question. The large gap between offline models (52.18%) and streaming proactive models (7.22%) gives the field a concrete target, and the error analysis (early ASK dominating) points to a specific capability gap rather than a generic "models are weak" conclusion.

Real-world applications (the paper names everyday visual assistance and remote operational guidance as the motivating settings):

• Everyday visual assistance, where a camera view is blurry, occluded, or pointed the wrong way and the assistant must request a better view.
• Remote operational guidance, where a helper model must ask for a clearer angle or a specific action demonstration before advising.
• Anomaly alerts and continuous task monitoring, where the model must recognize that what it is seeing is insufficient to decide.
• Personalized assistants that must ask about the user's current preferences rather than relying on stale ones.

Industry relevance. The results suggest that deployment of streaming proactive models for customer-facing or safety-adjacent monitoring is premature: most streaming proactive baselines score zero on most subtasks, and even strong offline APIs top out around half of Ask-F1. The finding that extra ASK budget saturates near 52% implies that simply allowing more questions will not close the gap; better decisions about timing are needed.

Future Directions

• Multi-ASK turns. The current setting permits at most one ASK per turn; the authors note that real scenarios may need repeated clarification, especially with multiple ambiguous entities or when one ASK reveals further missing information.
• Longer and more complex uncertainty horizons. Real uncertainty may unfold over longer horizons with overlapping, discontinuous, or dynamically changing ASK and ANSWER opportunities, beyond the current frame-level windows.
• Autonomous insufficiency detection. Every interaction here begins with an explicit user query; a more autonomous agent should detect insufficient information on its own and decide whether clarification is needed.
• Embodied and action-taking agents. The benchmark does not cover vision-language-action models that must jointly coordinate perception, information seeking, and physical action.

Target Audience

Researchers and practitioners working on multimodal large language models, streaming/online video understanding, and proactive or interactive agents — particularly those designing benchmarks, interaction protocols, or memory mechanisms for real-time video systems. It is also useful for product teams building camera-based assistants, remote guidance tools, or monitoring systems who need to know how far current models are from reliable proactive clarification. Readers without a background in benchmark metrics will need to work through the Ask-F1 definition, but the task taxonomy and error analysis are accessible on their own.

Authors’ abstract

Recent advances in multimodal large language models have increased interest in interaction paradigms for video understanding. The evolution from offline to streaming and proactive interaction has gradually shifted models from passive responders toward more proactive agents that can decide when to answer. However, existing proactive paradigms typically assume that the information required to answer a query is complete, leaving models to only wait or answer. In the real world, however, information is often incomplete due to uncertainty from either the video or the user. In proactive paradigms, such uncertainty can cause the model to wait forever or generate highly hallucinatory responses. We therefore argue that proactive interaction should be redefined with a third action: ask, allowing models to actively ask for missing information. To study when and what models should ask before answering, we introduce AskStream, the first multi-turn benchmark spanning video and user uncertainty for streaming video understanding, with four task categories and eight subtasks. It contains 438 videos with 2,714 manually annotated frame-level time windows across three types: wait, ask, and answer. Across both multiple-choice and open-ended settings, even the strongest model attains only 52.18% on our proposed Ask-F1 metric, highlighting substantial limitations in proactive information seeking under uncertainty.