EM^2Mem: Event-Centric Multimodal Memory for Large Language Models

AI summary

Overview

• Research area: Natural Language Processing / multimodal memory and retrieval-augmented generation for long-video question answering with large language models.
• Technical level: Intermediate to Advanced. The paper assumes familiarity with multimodal LLMs, retrieval-augmented generation, and memory architectures, but its central idea (organizing evidence around events) is explained in accessible terms.
• Scope: The paper proposes EM²Mem, a framework that binds captions, transcripts, keyframes, structured metadata, temporal context, and provenance to shared "event anchors" during memory construction, and evaluates it on three multiple-choice long-video QA benchmarks (EgoLifeQA, Ego-R1 Bench, Video-MME (L)).

What This Paper Is About

Long-video QA requires models to answer questions whose supporting evidence may be scattered sparsely across minutes or hours of video, speech transcripts, OCR, scenes, and objects. Existing systems store this evidence as isolated fragments (captions, frames, transcripts, summaries, graph triples) that are searchable but not "generation-ready," forcing the language model to reconstruct cross-modal and temporal alignments at inference time under a limited context budget. EM²Mem addresses this by organizing multimodal evidence around events before retrieval, so the model reads out grounded event-level evidence rather than stitching together loose fragments afterward.

Key Contributions

1. An event-centric multimodal memory schema. The video is divided into short base segments (e.g., 30-second clips), each associated with an event anchor e_i = (i, τ_i^s, τ_i^e) that acts as a shared, language-addressable indexing key — a temporal address rather than memory content.
2. Event-indexed memory cells with multi-scale context. Each cell μ_i = (e_i, R_i, C_i) stores a local multimodal record (visual keyframe caption, transcript/dialogue context, representative keyframes, structured metadata over actions, objects, dialogue topics, visual scenes, and visual entities) plus temporal context views at multiple scales such as 3 minutes, 10 minutes, and 1 hour.
3. Event-linked graph memory. Two lightweight graphs are built over shared event anchors: an episodic graph G_E for concrete cross-event relations (shared entities, objects, scenes, topics, and temporal transitions between adjacent events) and a semantic graph G_S for recurring patterns such as habits, preferences, routines, and stable relationships — each grounded back to supporting event anchors.
4. An align-then-retrieve inference procedure. Given a question, EM²Mem performs lightweight event-level readout, expands candidates through graph-linked evidence, applies an LLM-based selector, and compiles a compact query-specific evidence view E_q for answer generation.

Main Findings

• Accuracy gains over the strongest memory baseline. EM²Mem improves average accuracy by 2.0, 2.4, and 3.7 points on EgoLifeQA, Ego-R1 Bench, and Video-MME (L), respectively. Under the controlled same-setting comparison, it reaches 66.0 vs. WorldMM† 64.0 on EgoLifeQA, and 76.8 vs. WorldMM† 73.1 on Video-MME (L). On Ego-R1 Bench it achieves 67.7 vs. published WorldMM 65.3.
• Where the gains concentrate. Improvements are especially visible on RelationMap and TaskMaster in EgoLifeQA, EntityLog in Ego-R1 Bench, and AREC, CNT, OCR, and ORES in Video-MME (L). Against the originally reported WorldMM numbers, EM²Mem is competitive and slightly higher on average, though WorldMM is stronger on several habit, temporal, and synthetic reasoning categories.
• Large inference efficiency improvements. Compared with WorldMM on EgoLifeQA, average per-query latency drops from 459.00s to 98.21s (a 4.67× speedup), wall-clock evaluation time drops from 229,502s to 6,138s (a 37.39× throughput improvement), and total inference tokens fall from 42.03M to 15.27M (a 63.66% reduction). Input tokens fall 58.42% (31.95M to 13.29M) and output tokens fall 80.29% (10.08M to 1.99M). EM²Mem processes independent questions with 8 workers, but the paper states the gain is not only due to parallel execution — the main reason is reading from pre-constructed event-indexed memory cells instead of aligning evidence during inference.
• Better evidence localization. EM²Mem achieves 30.8% strict 30-second event-level Top-5 recall, beating WorldMM after five iterative retrieval rounds by 7.0 points. Top-1 recall reaches 23.0%, close to WorldMM's 5-round 23.8%, indicating single-pass localization is viable. The largest gains are on HabitInsight (+16.3) and TaskMaster (+15.9), while EventRecall is slightly lower (-0.8).
• Ablation results on EgoLifeQA. Starting from 66.0%, removing temporal context views causes the largest drop (60.4%, -5.6), followed by removing semantic memory (61.4%, -4.6) and removing the episodic graph (61.6%, -4.4). A local 30-second event record retrieval baseline remains competitive at 64.0% (-2.0), suggesting fine-grained local records already provide strong evidence for short-range factual questions.
• Structured fields beat raw frames and flat captions. On the first 250 EgoLifeQA questions, structured event fields achieve 71.2% accuracy under construction-time unification, versus 68.0% for raw frames and 65.6% for flattened captions. Under retrieval-time fusion the numbers are 68.0%, 66.8%, and 65.2% respectively. Construction-time unification consistently beats retrieval-time fusion, with the largest gain for structured evidence (+3.2%).
• Keyframes as lightweight verification. Overall accuracy rises from 63.2% without keyframes to 66.0% with three keyframes, with larger gains on TaskMaster and RelationMap; HabitInsight benefits less consistently.
• Amortized construction cost. Although EM²Mem has a higher upfront construction cost, its lower inference-time wall-clock cost reaches break-even after roughly 23–24 queries; token usage requires a larger reuse scale to break even.

Methodology in Plain English

The researchers started from a claim about how questions are actually asked: people ask about events, situations, recurring behaviors, and temporally grounded relations — not about isolated frames or sentences. So instead of storing video evidence in separate modality-specific silos (one store for captions, another for transcripts, another for keyframes), they cut the video into short 30-second segments and give each segment an "event anchor" — essentially a timestamped label that everything else can point to.

During memory construction, they use multimodal parsing modules (captioning, transcript alignment, keyframe selection, structured field extraction) to gather all the evidence for a segment and bind it to that anchor: a visual caption, the nearby dialogue, a few representative keyframes, and structured metadata about what action is happening, which objects appear, what the conversation is about, what the scene looks like, and which entities are present. They also build summaries of surrounding time spans at coarser scales (3 minutes, 10 minutes, 1 hour) so an event can be understood in context.

On top of these cells they build two small graphs: one linking events through shared entities, objects, locations, scenes, topics, and adjacency in time; another summarizing long-term patterns like habits and stable relationships, with links back to the events that support them.

At query time, the system does something deliberately light. It finds relevant event cells using the local records, context views, and graph links; expands through temporal neighbors and shared entities or topics; runs an LLM-based selector to keep only what is needed; and compiles a small evidence bundle containing captions, transcripts, structured visual fields, temporal summaries, semantic facts, and selected keyframes. The final answer comes from an LLM conditioned on the question and that bundle. The whole point is to move the hard cross-modal alignment work to construction time, before anyone asks a question.

Why This Matters

The paper reframes the memory design question for long-video agents: not "how much should we store," but "what should the retrieval unit be." It argues that fragment-centric memories create a retrieve-then-align bottleneck, and that aligning evidence at construction time produces retrieval units that are directly usable for grounded, attributable generation. It also reports that this shift improves accuracy, evidence recall, and inference cost at the same time — which is unusual, since accuracy and efficiency often trade off.

Real-world applications implied by the paper's framing:

• Personalized life assistants. EgoLifeQA is explicitly described as a benchmark for whether a model can serve as a personalized life assistant by retrieving and reasoning over sparse evidence in week-long first-person recordings.
• Ultra-long egocentric reasoning over personal histories, including tracking objects and entities, recalling past events, identifying recurring habits, mapping relationships, and tracking ongoing plans or pending tasks.
• Open-domain long-form video understanding, evaluated through Video-MME (L), covering videos longer than 30 minutes across diverse visual domains.
• Attributable evidence retrieval, since every graph relation and semantic fact is linked back to supporting event anchors, allowing graph-derived clues to be traced to concrete video evidence.

Industry relevance: the paper is co-authored with researchers affiliated with Lenovo Group Limited, and the closest baseline (WorldMM) is a memory-based long-video reasoning framework. The efficiency results matter for deployment — reducing total inference tokens by 63.66% and per-query latency to 98.21s changes the cost profile of serving long-video QA. The paper also notes the code will be integrated into https://github.com/zjunlp/LightMem.

Future Directions

• Coarse-to-fine multimodal memory. The authors note that converting visual evidence into textual fields trades visual fidelity for searchability and may lose fine-grained pixel details such as small objects, colors, layouts, and subtle visual states. They propose preserving finer visual evidence for verification while structured fields handle retrieval.
• Completing the align-then-retrieve paradigm. The final answer stage still relies on selected keyframes for visually detailed reasoning, which leaves part of cross-modal alignment to inference time and may introduce modality bias.
• Robustness to upstream errors. EM²Mem depends on upstream MLLMs and visual tools, so errors in captioning, object extraction, or metadata normalization may propagate into event cells and graphs — how to detect or mitigate this is left open.
• Deployment constraints and governance. The framework shifts computation from inference to memory construction, making it better suited to videos processed once and queried repeatedly than to real-time scenarios. The ethics statement adds that practical deployment should require informed consent, access control, data minimization, memory deletion or expiration mechanisms, and safeguards against using inferred semantic facts for high-stakes decisions.

Target Audience

This paper is most useful to researchers and engineers building memory or retrieval-augmented systems for long-video or long-horizon multimodal reasoning — particularly those working on video agents, egocentric assistants, and knowledge-graph-backed RAG. It also suits practitioners who care about the cost side of long-context inference, since it reports latency, wall-clock, and token metrics alongside accuracy. Readers interested in evidence attribution and privacy governance for memory systems over personal video will find the limitations and ethics discussion relevant. It is less suited to readers seeking a new foundation model or purely visual representation learning, as it builds on existing MLLMs and visual parsing tools rather than replacing them.

Authors’ abstract

Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM^2Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM^2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by 4.67 times and total inference tokens by 63.66% (The code will be integrated into https://github.com/zjunlp/LightMem).