WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs

AI summary

Overview

Research area: Streaming video understanding with Multimodal Large Language Models (Video-LLMs), specifically streaming video question answering (Streaming VQA) under online, time-causal constraints.

Technical level: Advanced. The paper assumes familiarity with the encoder–projector–LLM Video-LLM stack, key–value caches, sliding-window attention, retrieval-based memory, LoRA finetuning, and late-interaction multi-vector retrieval.

Scope: The paper diagnoses "Time-Agnosticism" in existing Video-LLMs and proposes WeaveTime, a plug-and-play framework combining a training-time temporal reconstruction task with an inference-time uncertainty-gated memory cache, evaluated on OVO-Bench and Streaming-Bench among other benchmarks.

What This Paper Is About

Streaming video systems receive frames one at a time, in order, and can never look at the future. The authors argue that current Video-LLMs fail in this setting because they treat a video as an unordered "bag of evidence" rather than a causally ordered sequence — a property they call Time-Agnosticism. WeaveTime aims to fix this in two moves: teach the model to perceive temporal order during training, then use that order to decide at inference whether to answer from the current frame or recall from memory.

Key Contributions

1. Diagnosis of Time-Agnosticism. The authors show empirically that shuffling video frames barely degrades Video-LLM accuracy and can even improve it on some tasks, including explicitly temporal ones, revealing reliance on spatio-temporal shortcuts and positional biases rather than causal order. They trace two downstream failures: Temporal Order Ambiguity and Past–Current Focus Blindness.

2. WeaveTime, a plug-and-play framework. A Video-LLM-agnostic streaming VQA framework that requires no architectural changes and no specialized streaming data, following a "teach order, then use order" design.

3. Streaming Order Perception Enhancement (SOPE). A lightweight Temporal Reconstruction auxiliary task that appends an ordering sub-question before the original QA query in a single conversation, instilling order-aware representations with minimal finetuning and no separate optimization stage or custom loss.

4. Past–Current Dynamic Focus Cache (PCDF-Cache). An uncertainty-triggered, coarse-to-fine retrieval mechanism that answers from the current window when prediction entropy is low and expands into history only when entropy is high.

Main Findings

• Frames can be shuffled with little harm to current models. In the paper's shuffled-frame comparison across eight task categories (Information Synopsis, Attribute Perception, Action Reasoning, Spatial Reasoning, Object Recognition, Counting Problem, Temporal Perception, Action Recognition, Temporal Reasoning), the model's accuracy stays strong under shuffling and barely benefits from timestamps. Humans, in contrast, collapse on temporal and action tasks when timestamps are removed and recover fully when timestamps are provided. Shuffling improved the model's accuracy on some cells, which the paper highlights in red.

• The model exhibits a temporal positional bias. A heatmap of accuracy under ground-truth answer-window shifts (0–100% along the video, grouped by binned video length such as 345s and 480s) shows systematic accuracy changes across shift positions and lengths, indicating the model prefers specific time locations rather than locating evidence by query.

• Gains on streaming benchmarks. Integrated with LLaVA-OV-7B, WeaveTime achieves up to +7.10% on OVO-Bench Real-Time and +3.74% on Streaming-Bench Real-Time over StreamBridge and ReKV. Its OVO-Bench Real-Time average is 68.82 and Streaming-Bench Real-Time average is 72.13, versus 61.72 and 66.15 for ReKV and 61.64 and 68.39 for StreamBridge. With Qwen2-VL-7B it reaches 66.28 and 75.39.

• Largest gains are on temporal-aware tasks. WeaveTime surpasses baselines by +7.56% in Action Perception (ACP), +9.04% in Event Understanding (EU), and +11.09% in Action Recognition (ACR).

• Ablation decomposes the two components. Starting from LLaVA-OV-7B + ReKV at 53.56 on OVO-Bench and 66.15 on Streaming-Bench, finetuning with timestamp prompts alone degrades performance (-3.68% and -0.54%) due to distribution mismatch. Adding Temporal Reconstruction (SOPE) raises results to 55.70 (+5.82%) and 68.49 (+2.58%). Adding PCDF-Cache yields a further 57.57 (+1.87%) and 72.13 (+3.64%).

• Coarse-to-fine recall beats coarse-only retrieval and avoids out-of-memory. On QAEgo4D, MLVU, and EventHALL, C2F reaches Recall 25.2, QAEgo4D accuracy 55.2, MLVU 68.9, and EventHALL 61.4, compared with ReKV's 23.9, 54.3, 68.5, and 60.6, and base LLaVA-OV's 14.0, 52.8, 64.7, and 60.1. The fine-only variant is marked "OOM" (out-of-memory).

• The entropy threshold matters for accuracy and latency. In the PCDF-Cache ablation on the full OVO-Bench, accuracy peaks at 57.57% when the entropy threshold is set to 0.6, while response latency decreases monotonically as the threshold grows. The paper describes this as the best accuracy–efficiency trade-off.

• The model does learn to re-order. Sampling 100 cases and evaluating next-token predictions against ground truth, the authors report that the trained model reliably reconstructs correct order of shuffled videos when guided by contextual cues, with notably high reconstruction accuracy for clips preserving key temporal anchors.

• Data and compute efficiency. LLaVA-OV + ReKV with SOPE reaches 55.7 on OVO-Bench using 30K offline video samples and 8 GPUs with zero stream-tailored data, a 2.1-point improvement. StreamForest Base reports 53.9 with 32 GPUs, and with Online IT reaches 55.6 using 32 GPUs and 121K stream-tailored data — roughly four times the GPU resources.

Methodology in Plain English

The authors start from a retrieval-based Video-LLM baseline: a vision encoder, a connector into the language space, and an LLM that maintains a growing memory of key–value pairs from incoming frames. When a question arrives, the system retrieves the top-K most similar frames from memory and answers from them.

Two changes are made on top of this. First, during training, the model is fed video segments whose frame content has been shuffled while timestamp tokens are kept explicit, plus a short instruction asking it to list each segment's true time range. Because this ordering sub-question is placed before the original QA query in one conversation, the LLM generates the order as ordinary next-token prediction — no extra prediction head, no custom loss, no dedicated streaming dataset. This is SOPE, the Streaming Order Perception Enhancement, built around a Temporal Reconstruction objective.

Second, at inference, the model first tries to answer using only the recent local window. It measures the entropy of that answer and compares it to a threshold. If entropy is low, the answer is kept. If entropy is high, a coarse-to-fine recall is triggered: a fast frame-level cosine-similarity pass contracts the search space to a set of coarse candidates, and a late-interaction max-similarity score over token-level key vectors selects the final top-K frames within those candidates. This avoids exhaustive token-level retrieval over full history while retaining token-level precision.

Training details: 30K offline video instruction-tuning samples randomly drawn from LLaVA-Video-178K (which totals 1.3M IT data), one epoch of LoRA with a learning rate of 1×10⁻⁵ on 8 GPUs. Inference uses the ReKV codebase with a maximum of 64 recalled frames and an entropy threshold δ of 0.6. Evaluation follows a multi-turn protocol adapted from StreamBridge for streaming scenarios.

Why This Matters

Impact on research. The paper reframes a practical limitation of Video-LLMs — the inability to handle ordered streams — as a diagnosable property, Time-Agnosticism, with a measurable symptom (invariance to frame shuffling). It shows that a small amount of auxiliary supervision over ordering, rather than large volumes of stream-specific data, can improve streaming performance, and it pairs that with an inference-time controller. Both components are model-agnostic and require no architectural modification, which lowers the barrier for reuse.

Real-world applications (contexts named in the paper):

• Autonomous driving, where the order of events on the road is safety-critical.
• Human-robot interaction, where a robot must act on what it has already seen.
• Real-time surveillance, where the present observation must be distinguished from accumulated history.
• Online conferencing, where questions arise at arbitrary moments during an ongoing stream.

Industry relevance. The efficiency claims matter for deployment: the method is trained on 30K offline samples with 8 GPUs and no stream-tailored data, and the PCDF-Cache deliberately reduces redundant long-range memory reloads, which the paper identifies as a latency cost. WeaveTime is reported to improve accuracy while reducing latency, making it relevant for latency-sensitive online video products rather than offline batch analysis.

Future Directions

• Generalization beyond the tested backbones and tasks. The paper evaluates WeaveTime as a plug-in on LLaVA-OV-7B and Qwen2-VL-7B, alongside comparisons against other open and proprietary models. Extending the framework to a wider range of Video-LLM backbones and the future-cue subset of OVO-Bench is a natural open question; the paper does not report results there.

• Robustness of the entropy gate. Accuracy peaks at a single threshold value of 0.6 in the paper's ablation, and latency falls monotonically as the threshold rises. Whether this threshold transfers across domains, backbones, or question types without re-tuning is not established.

• Scaling fine-grained retrieval. Fine-only retrieval is reported as out-of-memory in the retrieval strategy comparison, so the boundary between coarse-to-fine and full token-level retrieval remains a practical constraint worth investigating.

• Broader validation of temporal re-ordering. The re-order feasibility study is based on a sample of 100 cases. How re-ordering accuracy scales with clip length, number of clips, or weaker temporal anchors is not reported, and the paper notes that performance is notably high only for clips preserving several key temporal anchors.

Target Audience

Researchers and engineers working on Video-LLMs, streaming or online video understanding, efficient multimodal inference, and memory mechanisms for long-context models. The paper is also relevant to practitioners deploying video question answering in latency-sensitive or continuously streaming products, and to readers interested in evaluation methodology for temporal reasoning — particularly the shuffled-frame diagnostic, which is a reusable way to test whether a model genuinely uses temporal order.

Authors’ abstract

Recent advances in Multimodal Large Language Models have greatly improved visual understanding and reasoning, yet their quadratic attention and offline training protocols make them ill-suited for streaming settings where frames arrive sequentially and future observations are inaccessible. We diagnose a core limitation of current Video-LLMs, namely Time-Agnosticism, in which videos are treated as an unordered bag of evidence rather than a causally ordered sequence, yielding two failures in streams: temporal order ambiguity, in which the model cannot follow or reason over the correct chronological order, and past-current focus blindness where it fails to distinguish present observations from accumulated history. We present WeaveTime, a simple, efficient, and model agnostic framework that first teaches order and then uses order. We introduce a lightweight Temporal Reconstruction objective-our Streaming Order Perception enhancement-that instills order aware representations with minimal finetuning and no specialized streaming data. At inference, a Past-Current Dynamic Focus Cache performs uncertainty triggered, coarse-to-fine retrieval, expanding history only when needed. Plugged into exsiting Video-LLM without architectural changes, WeaveTime delivers consistent gains on representative streaming benchmarks, improving accuracy while reducing latency. These results establish WeaveTime as a practical path toward time aware stream Video-LLMs under strict online, time causal constraints. Code and weights will be made publicly available. Project Page: https://zhangyl4.github.io/publications/weavetime/