StreamVibe: Proactive Video Agents for Evolving User Intent

AI summary

Overview

Research area: Computer vision and streaming vision-language models, specifically proactive agents that watch continuous video and decide when and how to help a user.

Technical level: Advanced. The paper combines a new benchmark, supervised fine-tuning, reinforcement learning, and an agentic memory framework, and assumes familiarity with streaming video-language models and RL for LLMs.

Scope: The paper introduces a benchmark, a model, and an agent harness for sustained proactive video assistance in which user intents may be unstated, may change over time, and must be tracked against an evolving environment (arXiv:2610.34662v1 [cs.CV], 26 Oct 2026).

What This Paper Is About

Existing proactive video agents mostly wait for an explicit request ("notify me when a goal is scored") and treat each request as an isolated single turn. Real assistance is messier: users often convey only a high-level goal ("help me finish this task whenever I get stuck"), the state of the task changes as earlier help is delivered, and the user's intent itself can be revised, extended, or interrupted mid-task. The paper's goal is to make agents infer such intents from continuous observation, decide the right moment to speak, and keep their understanding of the user's intent current throughout a long interaction.

Key Contributions

1. StreamVibe-Bench, described as the first benchmark to study user intent in proactive streaming interaction. It unifies explicit, implicit, and dynamic intents across physical (egocentric daily activity) and digital (GUI/computer-use) environments, and supports both pointwise and trajectory evaluation. It contains 4,100 evaluation samples, 20,546 help points, and 183.4 hours of video (average sample duration 8.3 minutes).
2. StreamVibe-VL, a streaming proactive video model trained in two stages: adaptive streaming supervised fine-tuning that jointly learns when to speak and what to say, and localized trajectory reinforcement learning (GRPO) that concentrates optimization on interaction-critical moments using point-level and trajectory-level rewards.
3. StreamVibe-Harness, an agent framework that stores active intents, task progress, and delivered responses as persistent, updatable interaction state, gives each intent an independent lifecycle with flexible termination, and uses hierarchical memory to retrieve relevant history before each decision.
4. A combined empirical study across proactive-interaction benchmarks (StreamVibe-Bench, General Timing, EgoProactive, IPIBench) and general video-understanding benchmarks (MVBench, Video-MME, OVO-Bench, OVBench), with diagnostic analyses of where current agents fail across intent categories, intent subtypes, and interaction length.

Main Findings

• Headline improvement: StreamVibe-VL preserves strong general video understanding while improving response timing and answer quality by 10.7% over other baselines.
• Harness adds further gains: Adding StreamVibe-Harness to a fixed policy improves response consistency and adaptation over sustained interaction, producing a 12.3% gain in proactive interaction.
• Proactive models are weak on intent: In the paper's own summary of results, implicit and dynamic intents are markedly harder than explicit intents for existing proactive models, which struggle both to understand what help is needed and to decide when to respond.
• Selectivity is a systemic weakness: Higher Recall tends to come with more false triggers, so current methods cannot balance coverage against unnecessary responses. Precision/Recall/F1 and False Positive Rate are reported separately to expose this trade-off.
• Errors compound over time: Timing F1 declines steadily as interaction length grows; Revision shows a sharper late-stage Recall drop, and Composition shows weaker content quality, indicating difficulty updating intents while maintaining multiple simultaneous constraints.
• Policy learning vs. state tracking do different jobs: SFT makes the model more sensitive to assistance opportunities (higher Recall, more unnecessary triggers); RL improves selectivity, substantially reducing FPR while improving answer quality; the Harness, with the policy fixed, further suppresses unnecessary responses and improves content quality, with the clearest gains on Dynamic Intents.
• Generalization transfers: StreamVibe-VL shows competitive results on EgoProactive and consistent gains over other models on IPIBench, and outperforms its base model on all reported general video benchmarks.
• Streaming baselines fail in practice: Qualitative case studies show streaming baselines responding before sufficient evidence appears, or failing to update later responses as the task evolves, producing premature, missing, inconsistent, or redundant assistance.

Methodology in Plain English

Benchmark construction. The authors collected task videos from physical (egocentric daily activities) and digital (GUI, computer-use) settings. For each video–request pair, Gemini-3.8-Flash was used to clean the data and propose candidate help points with response windows, preferred timings, and reference answers; for dynamic intents, timestamped multi-turn interactions were constructed in which intents are revised, extended, or interrupted. Four human reviewers independently verified requests and timing, and only samples approved by all reviewers were kept. Validated annotations were aligned to 1 FPS streams and converted into causal interaction sequences.

Task formulation. Proactive assistance is cast as a causal streaming decision process: at each timestep the agent sees only the video so far, the intents revealed so far, and its own past interaction history, and must output either silence or a response. An intent is considered covered only if the agent responds inside the valid window of one of its evidence points with a semantically correct answer; intents stay active until all evidence points are covered.

Difficulty ladder. Three levels are defined: Level 1 explicit intents (a specified condition to monitor), Level 2 implicit intents (only a high-level goal, so the agent must infer when and how to help), and Level 3 dynamic intents in three forms—Revision, Composition, and Interruption.

Evaluation. Pointwise evaluation samples positive and negative timestamps around annotated help points and scores each decision independently, both with gold history (isolating local decision ability) and self-generated history. Trajectory evaluation runs the model continuously over the full stream, exposing error propagation, repeated responses, and missed responses. Metrics split into timing (Precision, Recall, F1, False Positive Rate) and quality (Answer Score, Content F1, judged by Claude Sonnet 4.6 on a 0–4 scale).

Model training. Stage one adjusts decision-token weights according to the current training distribution to balance silence versus response, while separately supervising response text. Stage two builds short causal rollouts around annotated help points and optimizes them with GRPO, using a point-level reward that multiplies a timing score by an LLM-judged content score under one-to-one matching (penalizing missed evidence and over-response), plus a trajectory-level reward for overall coherence; the two are combined with weight 0.7 on the point-level term.

Agent harness. The harness maintains an explicit interaction state of active intents, task progress, and previously delivered responses, updating it only when supported by sufficient evidence. Each intent runs its own lifecycle (persistent intents stay open across events; one-time intents close once satisfied). Recent history is kept fine-grained while older content is progressively compressed, and task-relevant history is retrieved before each decision. The harness decides which intents remain valid; the model still decides when and what to respond.

Why This Matters

Impact on research. The paper reframes proactive video assistance from real-time long-context modeling into intent inference plus state tracking, and supplies a benchmark with a difficulty ladder (explicit, implicit, dynamic) and two complementary evaluation protocols. Its diagnostic results—particularly the finding that per-decision accuracy does not translate into stable behavior over long interactions—give the field a concrete failure profile to optimize against rather than a single leaderboard number.

Real-world applications:

• Digital task assistance: inferring "where should I click next?" while a user navigates a website or completes a GUI workflow.
• Egocentric everyday help: monitoring an ongoing activity such as cooking and intervening at the right moment without being asked.
• Guidance and learning: step-by-step support where the agent must judge from task progress whether the user is stuck.
• Tool-use and monitoring: watching a task run and flagging problems only when they actually occur, while adapting if the user changes the goal mid-task.

Industry relevance. The explicit separation between a trainable streaming policy and a stateful harness maps directly onto deployed assistant architectures, where a model is wrapped by orchestration and memory layers. The paper's evidence that harness-level state tracking yields gains independent of the underlying policy—clearest on dynamic intents—is actionable for teams that cannot retrain the base model but can change the surrounding system.

Future Directions

• Co-evolving intent and environment: the authors note that real interaction may involve tighter coupling, where new observations, task outcomes, and prior assistance jointly reshape both user needs and the environment, creating new goals and constraints.
• Richer intent evolution over longer horizons: current dynamic intents are limited to Revision, Composition, and Interruption; studying more complex and combined forms of intent change is left open.
• Better implicit-intent inference: implicit and dynamic intents remain markedly harder than explicit ones, so improving intent understanding from unstated goals is a direct next target.
• Timing precision: the appendix rescores the same predictions at ±2, ±1, and ±0 second tolerances, leaving room to study finer-grained response timing and how tolerant real applications need to be.

Target Audience

Researchers and engineers working on streaming and online video-language models, proactive or always-on assistants, and agentic memory/orchestration systems. It is also relevant to practitioners building GUI agents, egocentric assistant products, and evaluation suites for interactive AI, and to readers who want a concrete taxonomy of explicit, implicit, and dynamic user intent. Beginners can follow the problem framing and findings, but the training and reward design sections presume familiarity with SFT, GRPO-style RL, and video-language model evaluation.

Authors’ abstract

Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, prevailing methods largely rely on explicit requests while miss the implicit user intent (e.g., "where should I click next?" while a user navigate a website), meanwhile fail to update the state simultaneously as user requests might update continuously. These limitations make them less suited to intent-aware interaction in practical. To address these challenges, we introduce StreamVibe-Bench, a new benchmark serving as testbed for user intent interaction, with 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents. Building on this benchmark, we develop StreamVibe-VL, a streaming proactive model that infers implicit user intents from continuous observations and dynamically tailors its interactions as these intents evolve. We further introduce StreamVibe-Harness, which maintains user intents as persistent states with a fleixble lifecycles via a hierarchical memory to track the latest user intent and environmental state. Our experiments reveal three key findings: (i) StreamVibe-VL preserves strong general video understanding while improving response timing and answer quality by 10.7% over other baselines; (ii) existing proactive models remain limited in understanding user intent and deciding when to respond, especially under implicit and dynamic intents; and (iii) StreamVibe-Harness further improves response consistency and adaptation over sustained interaction, resulting in a 12.3% gain in proactive interaction.