WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning

AI summary

Overview

Research area: Computer Vision / video large language models — specifically memory-augmented agents for reasoning over long (hour- to week-long) video.

Technical level: Advanced. The paper assumes familiarity with video LLMs, retrieval-augmented generation, knowledge graphs, and Personalized PageRank-based retrieval.

Scope: The paper introduces WorldMM, a multimodal memory agent that builds separate textual (episodic and semantic) and visual memories from long video streams, then uses an adaptive retrieval agent to select the right memory and temporal granularity per query.

What This Paper Is About

Video LLMs handle short clips well, but scaling them to hours- or days-long videos is hard because context limits are exceeded and abstraction loses visual detail. Existing memory-based methods mostly compress video into text and retrieve a fixed number of fixed-length clips, which discards visual evidence and cannot match queries that span very different time ranges (a few seconds versus hours). WorldMM addresses both problems by constructing complementary textual and visual memories at multiple temporal scales and letting a retrieval agent decide, iteratively, which memory and which time scale to consult.

Key Contributions

1. Three complementary memory types. WorldMM constructs episodic memory (multi-scale knowledge graphs of factual events), semantic memory (a continuously consolidated knowledge graph of high-level relationships and habits), and visual memory (feature embeddings for text-query search plus frame-timestamp indices for exact moments).
2. Multi-scale temporal encoding. Instead of a single fixed temporal scale, episodic memory is a set of knowledge graphs built at resolutions t0 < t1 < ... < tN, allowing retrieval at second, minute, and hour granularities.
3. Adaptive multimodal retrieval agent. A retrieval agent iteratively selects a memory source (episodic, semantic, or visual) and formulates a modality-specific query, stopping when it judges the collected information sufficient, so the model is not forced to condition on unnecessary modalities.
4. Demonstrated gains and analysis. Evaluation on five long-video QA benchmarks, with ablations over memory composition, module design, retrieval steps, temporal intersection-over-union (tIoU), and end-to-end latency.

Main Findings

• Average accuracy gain: WorldMM-GPT reaches an average score of 69.5%, which the paper states exceeds the strongest baseline by 8.4% and represents an average 8.4% gain over previous state-of-the-art methods.
• Per-benchmark scores (WorldMM-GPT): EgoLifeQA 65.6, Ego-R1 Bench 65.3, HippoVlog 78.3, LVBench 61.9, Video-MME (L) 76.6.
• Smaller variant also competes: WorldMM-8B (Qwen3-VL-8B backbone) scores 56.4 / 52.0 / 69.7 / 55.4 / 66.0 for an average of 59.9, versus 51.6 for the base Qwen3-VL-8B model. The paper states both variants surpass their corresponding base models by more than 8% on average.
• Long video LLMs lag on days-long video: VideoChat-Flash, Time-R1, and Video-RTS all fall below 50% on EgoLifeQA and Ego-R1 Bench, while retrieval- and memory-based approaches score mostly above 52% on those same benchmarks.
• Episodic memory is the strongest single memory: Using episodic memory alone gives an average of 64.9 versus 44.9 for visual memory alone — the paper reports about 20% higher performance.
• Visual memory helps perceptual categories: The full configuration (E+S+V) surpasses the non-visual configuration (E+S) by an average margin of 4.2% as reported, notably on EntityLog and EventRecall in EgoLifeQA/Ego-R1 Bench and on Visual and Audio+Visual in HippoVlog (E+S+V reaches 81.6 on Visual and 78.3 average on HippoVlog, versus 75.2 and 73.8 for E+S).
• Semantic memory helps long-term reasoning: The full memory achieves 76.9% on the HabitInsight category, a 23% improvement over the setting without semantic memory (E+V, which scores 53.9 there).
• Temporal grounding: WorldMM achieves average tIoU of 10.09 (EgoLifeQA), 9.17 (Ego-R1 Bench), and 9.57 (LVBench), far above baselines such as HippoRAG (4.00 / 3.28 / 4.30), Qwen3 Emb. (4.35 / 2.87 / 4.54), and Time-R1 (0.58 / 0.59 / 2.70).
• Multi-turn retrieval helps: On EgoLifeQA, allowing a maximum of five retrieval steps yields a 9.3% improvement over single-step retrieval.
• Module ablations (WorldMM-8B): Replacing multi-scale graphs with a fixed timescale drops average accuracy by 6.1%; replacing graphs with embedding retrieval drops it by 4.4%; removing semantic consolidation causes roughly a 7% drop for the category requiring long-term reasoning; disabling dual-mode visual retrieval drops accuracy by about 3%.
• Efficiency: Latency was measured end-to-end on 100 randomly sampled queries from EgoLifeQA; the paper reports a superior latency–accuracy trade-off, with long-video LLMs incurring much higher latency at lower performance.
• Adaptive memory usage: Analysis of retrieval iterations per EgoLifeQA category shows HabitInsight and RelationMap rely primarily on semantic memory, while EntityLog and EventRecall benefit more from visual memory.

Methodology in Plain English

WorldMM runs in three stages.

1. Building the memories. A long video is cut into short segments at a base scale t0 and each segment is captioned. From those captions the system builds episodic memory: for each temporal scale in a set {t0, t1, ..., tN}, the video is partitioned into non-overlapping segments of that length and converted into entity–action–entity triplets forming a knowledge graph, producing one graph per time scale. Separately, semantic memory is built by splitting the video into coarse fixed-timescale segments, extracting semantic (rather than event-specific) triplets, and merging them into a single evolving graph; an embedding-similarity step finds overlapping or conflicting triplets, and an LLM decides which to remove and which to add or revise. Visual memory has two parts: embeddings of fixed-length visual segments for natural-language search, and a frame-to-timestamp index for directly fetching frames at known moments.

2. Retrieval. A retrieval agent takes the user question plus the history of past retrievals and, each turn, either outputs a (memory, query) pair or a STOP signal, up to a maximum of N iterations. Episodic retrieval runs across all time-scale graphs using a graph-based top-k search with Personalized PageRank scoring, then an LLM cross-scale reranker picks the most relevant temporal range and refines the top-m captions. Semantic retrieval also uses PageRank but scores edges (by summing the PageRank of the two endpoint nodes) to return the top-k relational triplets. Visual retrieval either matches a text query embedding against segment embeddings by cosine similarity or directly fetches frames from the timestamp index.

3. Response generation. The collected retrieval history, the selected memories, and the original question are passed to a separate response agent, which produces the final answer grounded in the retrieved evidence. Splitting retriever and responder keeps each component focused on one job.

Implementation. VLM2Vec-V2 is the multimodal encoder for visual retrieval; GPT-5-mini builds the episodic and semantic memories. The agents run on either GPT-5 (WorldMM-GPT) or Qwen3-VL-8B-Instruct (WorldMM-8B). Temporal scales are dataset-specific — for EgoLifeQA, for example, 30-second, 3-minute, 10-minute, and 1-hour intervals.

Why This Matters

The paper targets a practical bottleneck: real deployments (AI glasses, household robots) need reasoning over recordings spanning hours to days, where feeding all frames is infeasible and text-only summaries lose the visual detail many questions depend on. WorldMM shows that modality-aware, scale-aware retrieval can beat both full-context long-video LLMs and earlier memory agents.

Real-world applications:

• Egocentric assistants and wearable cameras that must answer questions about past events, object locations, and habits across days of footage.
• Household and service robots that need long-term memory of people, routines, and where things were left.
• Video surveillance and monitoring, where relevant moments are sparse but grounding in exact frames matters.
• Long-form media and sports analysis, such as questions about what happened in a specific half of a match or a summary of a vlog's audio and visual content.

Industry relevance: The results suggest a path to serving ultra-long video without proportionally scaling context windows, using lightweight structured memory plus selective visual retrieval; the reported latency-accuracy trade-off and the use of off-the-shelf models (GPT-5, Qwen3-VL-8B, VLM2Vec-V2, GPT-5-mini) make the design relatively reproducible.

Future Directions

• Improving visual indexing. The paper notes that indexing visual frames into a structured representation remains challenging, which is why visual-only memory scores far below episodic-only memory (average 44.9 versus 64.9).
• Better grounding of the retriever. The relatively low absolute tIoU values (about 9–10) leave room for more precise temporal localization.
• Reducing preprocessing cost. The paper acknowledges that RAG- or memory-based approaches "often require substantial preprocessing," an open cost to address.
• Extending the memory design. Scaling to even longer streams, other modalities, and determining how many retrieval iterations and which temporal scales are optimal per domain remain open questions.

Target Audience

Researchers and engineers working on long-video understanding, video LLM agents, memory-augmented retrieval, or graph-based retrieval systems. It is also relevant to practitioners building egocentric assistants, robotics memory, or long-form video analytics who need to reason over hours-to-weeks of video without feeding every frame into a model. Readers without background in video LLMs, knowledge graph retrieval, or PageRank-based ranking will find the methods sections dense.

Authors’ abstract

Recent advances in video large language models have demonstrated strong capabilities in understanding short clips. However, scaling them to hours- or days-long videos remains highly challenging due to limited context capacity and the loss of critical visual details during abstraction. Existing memory-augmented methods mitigate this by leveraging textual summaries of video segments, yet they heavily rely on text and fail to utilize visual evidence when reasoning over complex scenes. Moreover, retrieving from fixed temporal scales further limits their flexibility in capturing events that span variable durations. To address this, we introduce WorldMM, a novel multimodal memory agent that constructs and retrieves from multiple complementary memories, encompassing both textual and visual representations. WorldMM comprises three types of memory: episodic memory indexes factual events across multiple temporal scales, semantic memory continuously updates high-level conceptual knowledge, and visual memory preserves detailed information about scenes. During inference, an adaptive retrieval agent iteratively selects the most relevant memory source and leverages multiple temporal granularities based on the query, continuing until it determines that sufficient information has been gathered. WorldMM significantly outperforms existing baselines across five long video question-answering benchmarks, achieving an average 8.4% performance gain over previous state-of-the-art methods, showing its effectiveness on long video reasoning.