Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding

AI summary

Overview

• Research area: Computer vision / multimodal large language models (MLLMs), specifically streaming (online) and long-form video understanding.
• Technical level: Advanced. The paper assumes familiarity with MLLM latent spaces, KV-cache and token-budget management, entropy-based confidence measures, and REINFORCE-style policy-gradient optimization.
• Scope: The paper proposes LatentStream, a training-free framework that replaces the usual "store-and-retrieve" streaming memory pipeline with a "retrieve-and-internalize" one, in which retrieved historical evidence is absorbed into a compact, evolving set of latent memory tokens inside a frozen Video-LLM.

What This Paper Is About

Streaming video understanding requires a model to watch a continuous visual stream and answer questions under strict causality and a bounded memory budget. Existing methods save history into an external memory bank and, when a query arrives, pull relevant evidence back out as extra visual context — the evidence stays external and is never absorbed into the model's own reasoning state. LatentStream's goal is to make that retrieved evidence internal: a fixed-length latent memory that progressively absorbs task-relevant history and keeps guiding reasoning as the video stream continues.

Key Contributions

1. LatentStream framework and the retrieve-and-internalize paradigm. The paper reframes streaming memory from "store-and-retrieve" to "retrieve-and-internalize," where task-relevant historical evidence is progressively internalized into a compact, evolving latent memory that continuously guides streaming video reasoning.
2. Hierarchical Latent Memory Evolution (HME). This couples a Jenks-guided hierarchical streaming memory with three groups of Latent Memory Tokens (LMTs) that have progressively expanding memory receptive fields — group s sees only short-term memory, group m sees short- plus mid-term, group l sees short-, mid-, and long-term. Each group iteratively retrieves evidence from its own scope and internalizes it into a compact, fixed-length latent memory.
3. Progressive Confidence-guided Latent Memory Optimization (PMO). A hierarchical progression reward built from group-wise latent token predictive entropy (normalized top-δ entropy) that rewards both absolute confidence and a consistent ordering of confidence across groups, refining the latent memory token embeddings at test time.
4. Query-agnostic Hierarchical Streaming Memory (HSM). Short-, mid-, and long-term memory levels maintained online under a fixed memory budget, using Jenks Natural Breaks to derive data-dependent drop/compress/preserve partitions rather than fixed thresholds.

Main Findings

• OVO-Bench state of the art among open-source methods. LatentStream (7B) reaches 64.2% overall, a +10.2 improvement over the Qwen2.5-VL-7B baseline at 54.0%. The 3B variant reaches 59.0% (+6.8 from 52.2%).
• Large gains on the two OVO-Bench sub-areas. Real-Time Visual Perception rises from 63.3% to 68.5%, and Backward Tracing from 44.7% to 60.0%.
• StreamingBench. LatentStream reaches 76.9% (+3.0%), outperforming all compared training-free methods.
• Generalizes to offline long video. VideoMME 66.6% (+3.3), MLVU 74.0% (+6.1), and LongVideoBench 62.1% (+1.4) versus the Qwen2.5-VL-7B baseline.
• Gains hold across video durations. On VideoMME, short/medium/long scores move from 73.8%, 62.4%, 53.8% to 77.2%, 67.4%, 55.1%.
• Component ablation (OVO-Bench / VideoMME). Baseline 54.0/63.3; +HSM 58.1/65.1; +HSM+PMO 62.4/66.1; full model with HME 64.2/66.6. Full model beats baseline by 10.2% and 3.3% respectively.
• Internalization beats plain context augmentation (OVO-Bench / VideoMME). Baseline (hierarchical memory + query, i.e. FluxMem) 56.9/65.4; directly appending retrieved visual evidence 59.7/65.6; unoptimized LMTs 56.5/65.2; evolved LMTs 64.2/66.6. Evolved LMTs beat direct evidence injection by 4.5% / 1.0% and initial LMTs by 7.7% / 1.4% points. Retrieved visual tokens are removed before final decoding.
• The progression reward matters more than absolute confidence (OVO-Bench / VideoMME). Without latent memory optimization 58.1/65.1; absolute-confidence-only optimization 62.5/65.5; the hierarchical progression reward 64.2/66.6.
• Training-free. The Video-LLM backbone parameters are never modified; all adaptation happens through test-time optimization of latent memory tokens and retrieved evidence.

Methodology in Plain English

The pipeline has three stages.

First, while the video plays, incoming visual tokens are stored densely in a short-term memory to keep recent detail. As the stream moves on, a Jenks Natural Breaks analysis of temporal importance scores creates two breakpoints that split older representations into drop, compress, and preserve groups for the short-to-mid transition. For the mid-to-long transition, a two-class Jenks partition on feature distance between neighboring preserved tokens merges low-distance (redundant) pairs and keeps high-distance ones. This all happens before any question is asked, so the memory is query-agnostic, and it fits inside a global budget of 2048 visual tokens.

Second, latent memory tokens are created in three groups. Each group is given a progressively larger slice of history it may look at, and the group's tokens retrieve the top-B most similar visual patches from their own slice (similarity measured by the maximum cosine similarity between a candidate patch and the K tokens of that group). The retrieved evidence is inserted right after the corresponding token group, so each group's evidence stays associated with it. After each round of updating, relevance scores are recomputed, so the next retrieval adapts to what has already been absorbed.

Third, the tokens and the retrieved evidence are optimized at test time. The method computes the entropy of the model's predictive distribution at each group's terminal position (over the top-δ highest-probability tokens, normalized). A reward combines (a) how confident each group is on its own, and (b) whether confidence increases in the expected order — short-term group more uncertain than mid-term, mid-term more uncertain than long-term. Continuous noise is added to the latent state to form a Gaussian policy, and a REINFORCE-based direct policy gradient updates the latent embeddings. Retrieved evidence is only added to the retained pool if the candidate reward improves over the previous iteration. After R iterations, all injected evidence copies are removed and the answer is decoded from the memory, the query, and the final optimized latent tokens.

Configuration reported: backbones Qwen2.5-VL-3B and 7B; online videos at 1 fps with at most 256 frames; short- and mid-term capacities of 8 and 64 frames; long-term global budget of 2048 visual tokens; K = 2 latent memory tokens per group; B = 8 candidate patches per iteration; R = 4 evolution iterations; learning rate 1e-3; Gaussian perturbation σ = 10%. Offline: same frame rate, 64 visual tokens per frame, at most 1,024 frames. Experiments run on eight NVIDIA H20 GPUs.

Why This Matters

• Research impact. The paper argues that query-agnostic memory and query-conditioned reasoning have been loosely coupled, and shows that closing that gap with a latent working memory produces consistent gains on both online and offline benchmarks. It offers a concrete alternative to retrieval-as-extra-context, and the ablation shows the gains are not explainable by simply adding tokens.
• Real-world applications (the paper names these settings as motivation for streaming video understanding):
  • Live monitoring.
  • Autonomous driving.
  • Smart glasses.
  • Embodied agents and robotic systems.
• Industry relevance. Because LatentStream is training-free and only optimizes latent states at inference, it can be layered onto off-the-shelf Video-LLMs without retraining — the paper applies it to Qwen2.5-VL-3B and 7B. Working under a fixed memory budget (2048 visual tokens) is directly relevant to deployment on continuous streams where GPU memory and context length cannot grow without bound. The author affiliations include Ant Group, indicating industrial interest in this setting.
• Caveat on cost. The paper reports memory budgets, iteration counts, and hardware, but does not report wall-clock latency or per-iteration runtime in the provided content.

Future Directions

• Quantifying inference cost. The framework runs R = 4 evolution iterations with test-time gradient updates and multiple forward passes per query; the paper does not report latency or compute overhead, which is the natural next measurement for real-time deployment.
• Scaling to larger backbones and longer budgets. Only Qwen2.5-VL-3B and 7B are evaluated, with 256 frames online and 1,024 frames offline; whether the gains and the 2048-token budget scale further is untested.
• Reconciling streaming memory with streaming thinking. The related work distinguishes training-based streaming-thinking approaches that evolve intermediate reasoning states; the paper does not combine its latent working memory with those methods.
• Sensitivity of the design choices. K = 2 tokens per group, B = 8 retrieved patches, R = 4 iterations, and σ = 10% are fixed in the reported setup; the provided content does not report sweeps over these values beyond the component ablations.

Target Audience

Researchers and engineers working on multimodal LLMs, streaming/online video understanding, and memory-efficient long-video inference. It is most useful for readers already comfortable with transformer internals, KV-cache and token-budget management, and policy-gradient optimization, and for practitioners who want a training-free memory mechanism they can attach to an existing Video-LLM. Readers looking for a beginner-level introduction to streaming video understanding would need background reading first.

Authors’ abstract

Streaming video understanding requires multimodal large language models (MLLMs) to process continuous visual inputs and respond to user queries under strict causality and bounded memory. Existing approaches typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. Though effective, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning. To bridge this gap, we introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize. Specifically, LatentStream comprises three coordinated components. First, Query-agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels under a fixed memory budget through Jenks-guided adaptive consolidation. Once a query arrives, Hierarchical Latent Memory Evolution equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling them to iteratively retrieve historical evidence from their corresponding scopes and internalize it into a compact, fixed-length latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization constructs a hierarchical progression reward from group-wise predictive entropy and jointly refines the latent memory tokens and retrieved evidence, encouraging increasingly confident streaming reasoning. Extensive experiments demonstrate that LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.