Memory as Weights: Internalizing Long-Term History for Streaming Videos

AI summary

Overview

Research area: Computer vision and multimodal learning, specifically streaming vision-language models (VLMs) and how such models should maintain long-term memory over a continuously arriving video stream.

Technical level: Advanced. The paper assumes familiarity with LoRA, HyperNetworks, state-space models, and streaming video benchmarks.

Scope: The paper proposes MaLoW, a memory framework that stores a video stream's history inside evolving low-rank (LoRA) weight updates rather than inside the input context, and evaluates it on streaming video QA and robotic action prediction.

What This Paper Is About

Streaming VLMs must answer questions about a video while it plays, so they need to remember what already happened without losing track of what is happening now. Existing memory methods compress, retrieve, or cache old observations and paste them back into the model's input, which means history and the current frame must compete for the same limited context window. MaLoW instead converts the accumulating history into small, continually updated parameter changes, so the model's input stays clean and only the weights carry the past.

Key Contributions

1. A new memory design ("history as weights"): MaLoW internalizes historical video context into time-evolving LoRA weights applied to a frozen base VLM, while the current observation and user query remain the only inputs. This separates memory from context rather than making them compete.

2. A recurrent HyperNetwork for incremental memory updates: A single-step update function repeatedly folds each incoming clip and its textual context into a hidden state, which is then decoded into LoRA weights. The state is split into a DeltaNet-based state-space branch for local temporal dynamics and a gated-summation branch for global semantics.

3. MaLoW-Instruct, a large-scale streaming training dataset: 381K question-answer pairs over 56K videos, built from public video temporal-grounding and procedural action datasets with a question generator, answer generator, and verifier pipeline, with OVO-Bench samples filtered out to prevent leakage.

4. Evidence that the approach transfers beyond video QA: Consistent gains across base VLMs from 0.8B to 11B parameters and model families (Qwen3-VL, Qwen3.5, Gemma4, VideoChat3, MOSS-VL), plus an extension to vision-language-action (VLA) models on the Simpler WidowX robot.

Main Findings

• State-of-the-art streaming scores: MaLoW (MOSS-VL-Real-11B) reaches 74.5 on OVO-Bench and MaLoW (Qwen3-VL-4B) reaches 84.8 on StreamingBench, while MaLoW (Qwen3.5-9B) attains the highest EPM score of 63.6 among compared methods.

• Large gains on backward tracing: On the backward split of OVO-Bench, MaLoW outperforms prior memory approaches by 3.9%–11.8% across different video backbones, with only modest latency overhead.

• Wins at matched model scales: At 4B, MaLoW (Qwen3-VL-4B) surpasses StreamTTT-4B by 3.9% on the OVO-Bench backward split and 3.5% on StreamingBench. At 7–9B, MaLoW (Qwen3.5-9B) beats OASIS-8B by 9.2% and AURA-8B by 6.0% on the backward split. At 11B, MaLoW (MOSS-VL-Real-11B) improves over its base model by 2.2% on the OVO-Bench Real-Time split and 1.4% on EPM.

• Acknowledged weak spot: On the OVO-Bench real-time split, MaLoW (Qwen3-VL-4B) and MaLoW (Qwen3.5-9B) underperform StreamTTT-4B and AURA-8B respectively; the authors hypothesize those baselines may have fine-tuned their base VLMs, whereas MaLoW keeps the backbone intact.

• Lightweight memory patch: MaLoW adds only 1.4%–4.0% trainable parameters for the main backbones (197.6M for VideoChat3-4B and Qwen3-VL-4B; 308.9M for Qwen3.5-9B; 321.1M for MOSS-VL-Real-11B), leaving the pretrained backbone architecture and weights unchanged.

• Small latency cost: Profiling on 100 deterministically sampled OVO-Bench questions shows the memory mechanism adds roughly 23 ms, less than 15% of total inference time, and LoRA generation can be parallelized with backbone inference.

• Strong on long-horizon memory: On the Historical Retrospection split of StreamArena, where the median query-evidence gap is 12.1 minutes, MaLoW (Qwen3.5-9B) scores 22.8 on average, beating VST-7B by 2.2% and MOSS-VL-Realtime-11B by 4.5%.

• Consistent cross-scale, cross-family gains: Across Gemma4-E4B and Qwen3.5-0.8B/2B/4B, MaLoW improves over the full-context baseline on real-time perception splits and over the local-window baseline on memory-intensive splits (for example MaLoW (Qwen3.5-4B) reaches 62.2 on OVO-Bench average versus 59.4 full-context and 55.1 local-window).

• Transfers to robotics: On SimplerEnv-Bridge with WidowX, MaLoW (Q3.5-2B-MLP) raises average success from 59.4% to 68.8% over the Q3.5-2B-MLP baseline. The paper states this gain as 9.4% in Section 4.2 and Table 2, while the abstract reports 9.3%.

• Qualitative memory reach: Case studies show MaLoW recalling details from 16, 19, and 29 minutes earlier, updating its memory to avoid obsolete cues, and recovering relevant information when current visual evidence is occluded.

Methodology in Plain English

The base VLM is frozen. A separate, small "memory writer" network watches the stream alongside it. At each time step, the newest clip and its timestamp text are compressed into a short summary vector using a handful of learned queries and a few self-attention layers. That summary updates an internal state with two parts: one branch (a DeltaNet state-space model) tracks short-range temporal patterns, and a second branch simply accumulates summaries over time through a gated sum, where a gate decides whether the current clip is sufficiently different from the previous one to be worth writing into long-term memory. A linear readout turns this state into rank-8 LoRA weights, which are attached to selected layers of the VLM at the next step.

Crucially, the VLM never sees the past video. It sees only the current observation plus the user query, and the past influences its answer solely through the generated weights. Training uses the standard next-token prediction loss on the answer, plus a regularizer keeping the generated LoRA matrices small. Because the past is not in the input, the prediction loss must be backpropagated through the generated weights to teach the memory writer what to keep.

Training data was produced with an LLM pipeline: a question generator proposes questions and timestamps from event annotations, an answer generator produces a model answer from video up to that timestamp, and a verifier keeps only question-answer pairs where the expected answer, generated answer, and surrounding video agree. Training runs 2,800 steps with batch size 64, Adam at peak learning rate 8×10⁻⁵ with a cosine schedule, 1 fps sampling, and 2-second clips.

Why This Matters

Impact on research: The paper reframes streaming memory as a parameter-space problem rather than a token-budget problem. It shows that a frozen backbone can carry long-horizon history in a few million evolving parameters, which opens a different axis for streaming-model design than summarization, retrieval, or token merging. Releasing the 381K-sample dataset also gives the streaming-video community a training resource that is currently scarce.

Real-world applications:

• Always-on wearable or smart-home assistants that must answer "what did I just do?" or "where did I leave it?" hours later.
• Robot manipulation, where the policy must remember objects it saw earlier that are now occluded or out of frame.
• Industrial or healthcare monitoring streams that need retrospective questions about events that occurred long before the query.
• Interactive video agents and tutoring systems that must track a user's actions across a long session without unbounded context growth.

Industry relevance: Because the method is a LoRA patch that leaves the backbone untouched, it plugs into existing LoRA-enabled inference stacks and preserves pretrained capabilities. The measured overhead of about 23 ms and 1.4%–4.0% extra parameters makes it far cheaper to serve than ever-growing context windows, which matters for latency- and memory-constrained deployment.

Future Directions

• Closing the real-time gap: MaLoW trails StreamTTT-4B and AURA-8B on the OVO-Bench real-time split. Whether this stems from those baselines fine-tuning their backbones, as the authors hypothesize, is an open question about how to gain memory without ceding current-frame perception.

• Improving long-horizon absolute accuracy: An average of 22.8 on the StreamArena HR split with a 12.1-minute median gap is still low in absolute terms, leaving substantial headroom for better retention over very long streams.

• Reducing memory-write overhead further: The authors note that LoRA generation can be parallelized with backbone inference; realizing and measuring that speedup is a concrete engineering next step.

• Broadening the memory substrate and tasks: The current design targets rank-8 LoRA on selected layers and is validated on VLM and one VLA setting. Extending to other modalities, larger action spaces, or deeper layer coverage is untested.

Target Audience

Researchers and engineers working on streaming or long-video multimodal models, efficient adaptation methods (LoRA, HyperNetworks), and embodied/VLA systems. It is most useful to readers already comfortable with VLM architectures and parameter-efficient fine-tuning; newcomers can follow the motivation and results but will need background to engage with the DeltaNet state updates and LoRA decoding details.

Authors’ abstract

Streaming vision-language models (VLMs) must continuously integrate information from the past while remaining responsive to current observations and user queries. Existing memory mechanisms typically compress, retrieve, or cache historical observations and reintroduce them into the input context, forcing historical memory and current observations to compete for limited context. We propose MaLoW, a streaming memory framework that instead internalizes historical context into evolving lightweight parameter updates. MaLoW uses a recurrent HyperNetwork to incrementally consolidate incoming video observations into fixed-size LoRA weights, allowing historical information to be carried forward without revisiting the full video stream. In doing so, MaLoW represents history as weights, while keeping the current observation as input. To support training, we further construct MaLoW-Instruct, a large-scale streaming video dataset containing 381K question-answer pairs over 56K videos. We conduct extensive experiments to demonstrate that MaLoW achieves strong performance on OVO-Bench, StreamingBench, and StreamArena. On the backward split of OVO-Bench, MaLoW outperforms prior memory approaches by 3.9% to 11.8% across different video backbones while incurring only modest latency overhead. Beyond streaming video understanding, MaLoW also generalizes to vision-language-action models, improving the action success rate on Simpler WidowX by 9.3%.