StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos

AI summary

Overview

Research area: Multimodal large language models (MLLMs) for streaming video understanding, specifically the integration of human eye-gaze signals into temporal and proactive reasoning.

Technical level: Intermediate — readers should be comfortable with multimodal LLMs, video QA benchmarks, and basic eye-tracking concepts (fixations, scanpaths, fields of view).

Scope: The paper introduces StreamGaze, the first benchmark and data-construction pipeline for evaluating whether MLLMs can interpret and exploit real-time gaze trajectories to reason about the past, present, and future in egocentric streaming video.

What This Paper Is About

Streaming video agents (e.g., AR glasses, robots) must interpret a continuously arriving video feed and anticipate what a user will do next, but existing benchmarks ignore the human perceptual signal that actually drives real-world attention: eye gaze. StreamGaze closes this gap by building a large gaze-video QA dataset that pairs egocentric footage with real gaze trajectories, then testing whether state-of-the-art MLLMs can use that gaze for temporal reasoning and proactive prediction. The goal is to expose and quantify how far current models are from human-level gaze-aware streaming understanding.

Key Contributions

1. A gaze-guided data construction pipeline that aligns egocentric video with raw gaze trajectories via fixation extraction, FOV/out-of-FOV regioning, and full scanpath modeling — producing spatio-temporally grounded QA pairs that static per-frame gaze pipelines cannot generate.
2. The StreamGaze benchmark: 8,521 QA pairs from 285 videos across 10 tasks, spanning past, present, and proactive timestamps, making it larger and more task-diverse than prior streaming or gaze QA benchmarks.
3. A comprehensive zero-shot evaluation of closed-source, open-source, gaze-specialized, and streaming MLLMs, revealing substantial and consistent gaps versus human performance (0.827 average human accuracy vs. 0.535 for the best closed-source model).
4. Diagnostic analyses of gaze prompting strategies (textual, visual, salience-map), reasoning strategies (text, gaze, visual), proactive error types, and task-specific failure modes, offering concrete design guidance for future gaze-aware streaming models.

Main Findings

• Large human-model gap persists across all tasks. Human annotators average 0.827 accuracy; the best model (GPT-4o) reaches only 0.535, and most open-source and streaming models fall below 0.48.
• General-purpose MLLMs cannot accumulate gaze evidence over time. Limited model capacity plus sparse keyframe sampling cause models to process frames independently rather than as a continuous stream, hurting long-horizon past tasks like Object Transition Prediction (OTP) and Gaze Sequence Matching (GSM).
• Streaming MLLMs struggle specifically with proactive tasks. Dialogue-trained models such as VideoLLM-online collapse (0.080 average) and emit generic conversational output, while the non-dialogue streaming model ViSpeak performs best among streaming baselines (0.467).
• Gaze-specialized models do not transfer to streaming. AssistGaze, designed explicitly for gaze QA, achieves only 0.223 average accuracy, showing that static or short-range gaze understanding does not generalize to long-horizon streaming scenarios.
• No gaze-prompting strategy reliably beats a gaze-free baseline. The salience-map prompt is best on average (0.454 vs. 0.446 gaze-free for Qwen2.5-VL), suggesting current MLLMs lack intrinsic machinery for reasoning over raw gaze signals.
• Combined reasoning helps but unevenly. GPT-4o improves to 0.565 when text, gaze, and visual reasoning are all enabled, but visual grounding helps Scene Recall while hurting Non-Fixated Object Identification — evidence that a single uniform strategy is insufficient.
• Proactive behavior is highly model-specific. InternVL3.5-8B over-triggers with high false-positive rates, GPT-4o degrades as tasks harden with both error types rising, and Qwen2.5-VL-7B shows the most balanced proactive error profile.

Methodology in Plain English

The researchers started with three public egocentric video datasets covering cooking, lab work, and assembly (EGTEA+, EgoExoLearn, HoloAssist) that already include raw eye-tracking.

1. Project gaze into the video frames. For 3D gaze rays, they use camera intrinsics and extrinsics to convert world-coordinate gaze into pixel coordinates on each frame.
2. Detect fixations. They locate stable periods where the gaze lingers in a localized region, enforcing spatial dispersion limits (normalized by frame width), a minimum duration, and a scene-consistency check via Hue–Saturation histogram correlation to discard abrupt cuts.
3. Split each frame into FOV and out-of-FOV. A circular region around the fixation centroid defines what the user is attending to; everything else is background context. For FOV extraction, they crop a circular patch and overlay a red dot at the gaze point. For out-of-FOV, they black out the FOV circle so only peripheral content remains.
4. Extract objects with an MLLM. InternVL3.5-38B with region-specific visual prompts identifies and captions objects in each region.
5. Build scanpaths. Ordered sequences of fixated and non-fixated objects over time capture how attention shifts — the core signal that distinguishes this pipeline from static gaze QA.
6. Generate QA pairs across three temporal regimes. Past tasks (NFI, OTP, SR, GSM) test memory and gaze-dynamics reasoning; present tasks (OI Easy/Hard, OAR, FAP) test current perception and intention inference; proactive tasks (GTA, OAA) require the model to decide when to alert rather than just answer.
7. Verify with humans. Annotators check and correct extracted objects and scanpaths, achieving roughly 83% correctness before inclusion.

Evaluation was zero-shot: non-streaming models were fed the corresponding video clip offline, gaze was injected via visual overlays (green dot for gaze point, red circle for FOV), and proactive tasks used a multi-triggering "Yes/No" protocol at each timestamp with accuracy as the metric.

Why This Matters

Impact on research. StreamGaze shifts streaming video evaluation from "does the model see the frames?" to "does the model understand what the human is actually looking at and about to do?" — a more realistic framing for attention-aware assistants. By releasing the pipeline, benchmark, and code publicly, it provides a reproducible testbed for a research direction (gaze-conditioned streaming reasoning) that previously had no standardized evaluation.

Real-world applications.

• AR and smart glasses that follow a wearer's gaze to offer timely, context-aware assistance (e.g., surfacing a recipe step when the user looks at an ingredient).
• Robotics and embodied agents that must infer operator intent from where the operator is looking to coordinate handoffs or avoid collisions.
• Assistive technology for users with motor or cognitive impairments, where gaze is already a primary interaction channel and proactive alerts could replace manual commands.
• Training and remote guidance in cooking, surgery, or industrial assembly, where a system could detect when a trainee fixates on the wrong tool or misses a step and intervene before an error.

Industry relevance. The benchmark directly targets the interaction loop of AR-glasses products and embodied AI platforms. The finding that no current MLLM can reliably interpret raw gaze signals — and that salience maps outperform raw overlays — gives product and research teams a concrete starting point for designing gaze-encoding layers rather than relying on visual prompting.

Future Directions

• Gaze-native model architectures. Since visual and salience-map prompting barely help over a gaze-free baseline, dedicated encoders or memory modules that ingest scanpath dynamics appear necessary rather than optional.
• Task-adaptive prompting and reasoning. The uneven effects of textual, gaze, and visual reasoning across tasks suggest that dynamic, task-aware strategy selection would outperform any single uniform prompt.
• Better proactive calibration. The divergent error profiles of InternVL3.5, GPT-4o, and Qwen2.5-VL indicate that proactive alerting needs explicit false-positive/false-negative trade-off control, not just accuracy optimization.
• Broader domains and hardware. Extending the pipeline beyond cooking, lab, and assembly datasets to real AR-glasses capture, more diverse cultural and task settings, and longer time horizons would test generalization.

Target Audience

Researchers and engineers working on multimodal LLMs, streaming/online video understanding, AR and wearable computing, human-computer interaction, and eye-tracking-based interfaces. It is also valuable for benchmark designers and product teams evaluating gaze-aware assistants, and for graduate students looking for a well-instrumented entry point into attention-aware streaming video reasoning.

Authors’ abstract

Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realistic applications such as Augmented Reality (AR) glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether Multimodal Large Language Models (MLLMs) can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs utilize gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively assess streaming video understanding. These tasks evaluate whether models can use real-time gaze signals to follow shifting attention and infer user intentions based only on past and currently observed frames. To build StreamGaze, we develop a gaze-video Question Answering (QA) generation pipeline that aligns egocentric videos with raw gaze trajectories through fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, highlighting key limitations in gaze-based temporal reasoning, intention modeling, and proactive prediction. We further provide detailed analyses of gaze prompting strategies, reasoning behaviors, and task-specific failure modes, offering insights into current limitations and directions for future research. All data and code are publicly available to support continued research in gaze-guided streaming video understanding.