StreamReady: Learning What to Answer and When in Long Streaming Videos

AI summary

Overview

Research area: Computer vision / multimodal large language models for streaming video understanding, with a focus on answer timing rather than only answer correctness.

Technical level: Advanced. The paper combines streaming memory architectures, a dual-branch Q-Former retrieval design, a learnable readiness token, and a new timing-aware evaluation metric.

Scope: The paper formalizes "readiness-aware" streaming video understanding, proposes the Answer Readiness Score (ARS) and the StreamReady framework, and introduces the ProReady-QA benchmark of 5k proactive QA pairs with annotated evidence windows.

What This Paper Is About

Existing video models usually answer questions about video content, but in live streaming settings a question can arrive before the visual evidence needed to answer it. If a model answers too early it is speculating; if it answers too late the answer loses real-time value. This paper defines a formulation in which models must decide both what to answer and when, introduces a scoring method that penalizes early and late answers asymmetrically, and builds a framework (StreamReady) plus a long-video benchmark (ProReady-QA) to measure and improve that behavior.

Key Contributions

1. Readiness-aware formulation and Answer Readiness Score (ARS). The paper defines streaming understanding as needing the correct answer at the moment the supporting evidence appears, and introduces ARS — a timing-aware objective with an asymmetric early penalty (harsher, to discourage unsupported guessing) and a milder late penalty. Combined with accuracy, ARS yields an "effective accuracy" (Acc_e) that reflects both correctness and timing.
2. StreamReady framework. A readiness-aware model that unifies temporal reasoning with a lightweight learnable <RDY> token and a Readiness Head, which decides from internal memory whether sufficient evidence has been observed before triggering the LLM to answer.
3. ProReady-QA benchmark. A benchmark of 5k proactive QA pairs over 10 one-hour Ego-4D and 22 half-hour MovieNet videos, with annotated answer evidence windows, proactive multi-turn questions spanning local and global contexts, and five task types (Sequential Steps Recognition, Repetitive Event Count, Clues Reveal Responding, Causal Trigger Detection, Goal-State Detection).
4. Broad empirical validation. StreamReady is evaluated on ProReady-QA plus eight additional streaming and offline long-video benchmarks.

Main Findings

• Best results on ProReady-QA. StreamReady (7B, Qwen-2-VL backbone) achieves the highest accuracy and ARS on all five ProReady-QA tasks, surpassing the best prior model by approximately 3% in accuracy and approximately 9% in ARS on average, with the largest ARS gains on REC, GSD, and CTD. Its average is 56.4 accuracy, 0.69 ARS, and 0.53 effective accuracy, versus 53.1 / 0.60 / 0.42 for the strongest prior model reported (StreamBridge).
• Timing, not just accuracy, improves. The paper reports that readiness reduces mistimed responses, tightening temporal alignment and directly lifting ARS, and that the gap between raw accuracy and effective accuracy shrinks.
• Strong on other streaming benchmarks. On StreamingBench and OVOBench, StreamReady reaches 48.2 / 58.8 on proactive splits and 78.3 / 63.7 on non-proactive splits (averages 63.4 and 68.2), with up to approximately 5% gains on proactive tasks of streaming benchmarks.
• Improvements are not only from timing. Most accuracy gains on those benchmarks are attributed to stronger evidence retrieval and long-horizon reasoning from the visual and contextual memories, since timing is not evaluated there.
• Retains offline long-video strength. On VStream-QA, OVBench, VideoMME, MLVU, MVBench, and EgoSchema, StreamReady scores 64.8 / 57.2 (VStream-QA RE/RM), 63.9 (OVBench), 65.8 (VideoMME), 71.3 (MLVU), 71.8 (MVBench), and 70.4 (EgoSchema), despite readiness and contextual reasoning being disabled for offline benchmarks.
• Component ablations show complementarity. Basic reasoning alone improves accuracy with little timing benefit; adding the readiness mechanism helps ARS modestly; the largest joint gains appear when readiness is combined with the memory and query-aware reasoning modules (e.g., CTD rises from 30.3 Acc / 0.28 ARS at baseline to 43.5 / 0.59).
• Readiness design matters. An MLP-only readiness head is reported as insufficient, LLM or MLLM-based readiness gives only modest improvement, and embedding <RDY> inside the reasoning module yields the best and most stable ARS, with a lightweight MLP head matching Transformer performance at lower cost.
• Placement of <RDY> matters. Placing it in the short-term branch yields noisy timing signals; as input to the long-term branch it gives moderate but unstable results; attaching it to the learned long-term representation gives the best ARS.
• Stable scaling on long videos. StreamReady maintains stable latency and memory as video length increases via a fixed-size compact memory, whereas the paper reports the Qwen-2-VL baseline suffers rapid latency growth and out-of-memory failures, ReKV incurs expanding KV caches, and StreamBridge incurs repeated token-compression costs.
• Penalty parameters were selected, not arbitrary. A broad region of stable performance is reported in Figure 5, with γ_e = 6 and γ_ℓ = 1 giving a balanced trade-off.

Methodology in Plain English

StreamReady processes a video as it arrives rather than all at once. Incoming frames go into a Visual Memory Tree with three levels: a short-term buffer of the most recent frame embeddings, a mid-level set of cluster centroids built with K-means and updated incrementally with an exponential moving average (with a threshold that tightens in stable scenes and relaxes when novelty rises), and a top level of coarse prototypes for long-range abstraction. A separate Contextual Memory Bank stores each question embedding along with the representation used to generate its answer, so multi-turn conversations can reuse past context.

When a question arrives, the model switches from passive encoding to query-aware reasoning using a dual-branch Q-Former. The short-term branch looks at the recent frame buffer; the long-term branch performs coarse-to-fine retrieval — first scoring prototypes to find the top-K likely regions, then refining down to the top-m fine-grained centroids — before combining everything with the short-term representation and the question.

The timing decision comes from a learnable <RDY> token that lives inside the long-term reasoning module, so it sees the same evolving evidence as the answering pathway. A small Readiness Head reads this token and outputs a readiness score between 0 and 1; the LLM is only triggered when the score crosses a threshold of 0.35. Because ground-truth evidence timestamps are not available during training, the authors construct weak supervision: high-similarity matches between the learned representation and mid-level memory form a pseudo-positive region, low-similarity matches form a pseudo-negative region, and a pairwise contrastive loss plus a mild temporal coherence regularizer trains the readiness signal. Gradients are stopped so that the readiness loss updates only the Readiness Head and <RDY> token, keeping "what to answer" and "when to answer" learned separately.

ARN evaluation works as follows: given an annotated evidence window from first-valid to last-valid time and the moment the model actually answers, an Early Penalty and a Late Penalty are computed, ARS is their product averaged over questions, and effective accuracy is accuracy multiplied by ARS. Offline models are evaluated on ProReady-QA by feeding progressively truncated video prefixes and recording when the correct answer first appears, while streaming models process frames sequentially with the prompt "Answer whenever you are ready."

Why This Matters

Impact on research: The paper separates answer timing from answer correctness as a first-class evaluation target. Prior proactive benchmarks include scenarios that encourage waiting but do not annotate answer evidence durations, so they cannot verify whether a chosen answer time was actually supported by evidence. ARS and ProReady-QA supply that missing criterion, and the demonstrated method shows timing can be learned with a single token and a lightweight head rather than auxiliary MLLMs or prompt heuristics.

Real-world applications:

• Surveillance, where alerts must fire when the triggering event occurs, not before (false speculation) or after (missed response window).
• Sports analytics, where commentary or analysis must align with the unfolding play.
• Robotics, where an agent must act once enough visual evidence has accumulated.
• Assistive systems, where timely, context-aware responses to a user's question depend on what is currently visible.

Industry relevance: The paper emphasizes deployment constraints — stable latency and memory as video length grows, no additional inference overhead from the readiness mechanism, and performance retention on offline long-video benchmarks. Systems built on this would run continuously on live streams rather than reprocessing full videos per query.

Future Directions

• Pushing readiness learning beyond weak supervision. The current readiness signal is derived from pseudo-positive and pseudo-negative regions rather than ground-truth evidence timestamps; how much further accuracy and ARS could improve with stronger supervision is an open question.
• Extending the benchmark's scope. ProReady-QA is built from 10 Ego-4D and 22 MovieNet videos and covers five proactive tasks; whether readiness behavior transfers to other domains, video lengths, or task families is not established by the reported results.
• Task-specific penalty design. Figure 5 shows ARS behavior is sensitive to γ_e and γ_ℓ; the paper selects γ_e = 6 and γ_ℓ = 1, leaving open whether task-adaptive penalty shapes would better capture different notions of timeliness.
• Unifying readiness with offline capability more tightly. The paper reports that readiness and contextual reasoning are disabled for offline long-video benchmarks, so how much these mechanisms could further help when timing is irrelevant remains unexplored.

Target Audience

Researchers and engineers working on streaming video understanding, multimodal LLM memory and retrieval architectures, and time-sensitive video QA. It is also relevant to benchmark designers interested in temporal evaluation metrics, and to practitioners building live video pipelines for surveillance, sports, robotics, or assistive applications who need models to respond at the right moment rather than merely respond correctly. Readers need familiarity with MLLMs and video-language retrieval to follow the memory tree and Q-Former details.

Authors’ abstract

Streaming video understanding often involves time-sensitive scenarios where models need to answer exactly when the supporting visual evidence appears: answering before the evidence reflects speculation, answering after it has passed reduces real-time utility. To capture this behavior, we introduce a readiness-aware formulation of streaming video understanding with the Answer Readiness Score (ARS), a timing-aware objective with asymmetric early and late penalties. When combined with correctness, ARS defines an effective accuracy that measures not just whether a model is right, but whether it answers at the appropriate moment. Building on this formulation, we introduce StreamReady, a framework to unify temporal reasoning with on-time answering through a lightweight readiness mechanism that decides if sufficient evidence has been observed before responding. To evaluate this capability, we further introduce ProReady-QA, a benchmark with annotated answer evidence windows and proactive multi-turn questions across local and global contexts. StreamReady achieves superior performance on ProReady-QA, and consistently outperforms prior methods across eight additional streaming and offline long-video benchmarks, demonstrating robust and broadly generalizable video understanding capability.