Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory

AI summary

Overview

Research area: Multimodal AI agents and agent memory mechanisms (cs.AI), specifically how agents that see, hear, and act in the world form, store, maintain, and evolve memory over long horizons.

Technical level: Advanced. This is a taxonomy and synthesis paper that assumes familiarity with large foundation models, multimodal LLMs, retrieval-augmented generation, and agent architectures.

Scope in one sentence: The paper proposes a two-axis framework — Axis I, Formation and Representation, and Axis II, Memory Management (T1 Static, T2 Dynamic, T3 Self-Evolving) — to organize and analyze 297 memory architectures and 82 benchmark records for multimodal agents, and derives design principles and open challenges from that synthesis.

What This Paper Is About

Multimodal agents increasingly operate over images, video, audio, graphical interfaces, and 3D or embodied environments across long-horizon and lifelong tasks, but their observations cannot be compactly or losslessly compressed into a single text record the way text-only agent memory can. The paper's goal is to bring order to this fragmented area by separating two questions: what experience gets retained and in what representational form (Axis I), and where persistent change occurs as experience accumulates (Axis II). The authors also use this framework to synthesize recurring design patterns, derive principles, and map out unresolved challenges.

Key Contributions

1. A two-axis analytical framework for multimodal agent memory. Axis I, Formation and Representation, classifies the representation ultimately supplied to downstream reasoning or action. Axis II, Memory Management, classifies systems by the scope of persistent deployment-time change: T1 Static Memory (fixed memory reused without modification), T2 Dynamic Memory (memory state updated through a stable policy), and T3 Self-Evolving Memory (the policy governing memory writing, maintenance, or retrieval itself is updated through feedback).

2. Comprehensive corpus-level coverage. The paper reports a corpus of 297 memory architectures and 82 benchmark records in the current version of its taxonomy, spanning images and documents, video and audiovisual streams, audio and speech, GUI interaction, and embodied or 3D environments.

3. Cross-cutting patterns observable only when both axes are analyzed jointly. The paper identifies the prevalence of dynamic memory updates, the widespread use of hybrid representations, and the relative scarcity of deployment-time memory-policy evolution.

4. Derived design principles and a future-direction agenda. The principles concern balancing source fidelity with abstraction, maintaining temporal validity and provenance, grounding memory use in recoverable evidence, and evolving memory policies over time. Future directions include recoverable and provenance-aware memory, temporal validity and cross-representation consistency, self-improving memory management, bounded lifelong memory, governed forgetting, and lifecycle-aware evaluation.

Main Findings

• Memory is defined functionally, and the definition excludes most of what passes for context. Agent memory is "experience-derived state that persists beyond the step in which it is formed and later affects reasoning, generation, planning, or action," with at least one non-text modality contributing to its formation or use. Transient activations, one-pass reasoning traces, and incidental prompt context are excluded unless explicitly persisted and reused. Context is what the model currently sees; memory determines what can survive, how it is organized, and when it can re-enter the context or policy.

• Three representation families structure Axis I. (1) Source- and modality-specific representations, where evidence is preserved close to the source modality; (2) cross-represented explicit memory, where experience is translated into textual, symbolic, or structured records; and (3) internalized latent and parametric memory, where experience lives in latent state or model parameters. The paper explicitly treats hybrid and evidence-linked architectures as system-level compositions combining multiple paths rather than as a fourth family.

• The first family splits into two distinct subforms. Retained perceptual source evidence keeps observations addressable in their native modality — complete images, regions or crops, document screenshots, sampled video frames, audiovisual segments, RGB-D observations, or observations paired with timestamps and agent poses. Modality-specific encodings and geometric state instead persist learned features, modality tokens, or world-centered geometric state that the reasoner consumes directly, preserving structure such as spatial layout, temporal order, acoustic dynamics, or metric geometry.

• Granularity of indexing is as important as the decision to retain source media. The paper contrasts frame-level organization (ToolMerge, which retains sampled video frames and uses an LLM planner to compose visual retrieval tools including global image-text matching, region-level entity matching, and OCR), event-level organization (HippoMM, which segments audiovisual experience into event records with semantic summaries, temporal boundaries, and pointers to source spans), and entity-level organization (M3-Agent, which links text, image, and audio evidence to people, objects, and events over time). PixelRAG is cited as a case where visual embeddings serve only as a retrieval index while retrieved screenshot tiles are supplied to a VLM as pixel inputs.

• Video memory is where modality-native encodings are most developed, and systems differ mainly in granularity and consolidation. MovieChat accumulates frame-level visual tokens in a fixed-capacity short-term memory and merges similar representations from temporally adjacent frames before transferring consolidated features to long-term memory. MA-LMM stores patch-level features in a visual memory bank, where they serve directly as keys and values for cross-attention, merging temporally adjacent features at corresponding spatial locations when capacity is exceeded. Flash-VStream uses a coarse-to-fine design in which low-resolution feature centroids give broad temporal coverage and high-resolution feature maps preserve detail for representative key frames.

• Image memory behaves differently from video memory. Representative image agent systems more commonly retain decodable source images or pointers to them, using visual embeddings only for indexing and retrieval, while some systems transform visual observations into textual or structured records and discard the source images.

• Audio memory is underdeveloped. Persistent encoded audio memory remains underexplored among the analyzed systems. Speaker embeddings are typically used only as retrieval keys rather than as stored memory values exposed to the downstream model; OMAC, introduced in O-MARC, is cited as a relevant encoding mechanism that retains high-importance audio tokens as anchors.

• Structured spatial memory and source evidence play complementary roles in embodied settings. GraphEQA combines a hierarchical 3D scene graph with a task-relevant memory of images captured during exploration, so retrieved images restore local visual detail abstracted away in the global graph. DovSG associates RGB-D observations and camera poses with a dynamic object-level map so historical observations support relocalization and local map refinement when the environment changes.

• Control and recoverability trade off against compactness. Native preservation keeps a wider range of perceptual detail but increases storage and retrieval cost. Translation produces compact, editable, searchable memory units but can remove information that was not anticipated when the memory was constructed.

• Management has distinct failure points. In the coupled management loop, an agent may fail to retrieve relevant evidence, retrieve it but apply it incorrectly, or produce a correct output from model priors without using memory at all. Management is constrained by formation: no access mechanism can recover information discarded during formation or later corrupted, invalidated, or forgotten.

• Growth is uneven across modalities. In the cumulative architecture counts plotted by publication year from 2023 to 2026, embodied and video memory architectures show the steepest growth by 2026, followed by image and GUI systems, while audio/speech memory remains comparatively sparse.

• Multimodal memory inherits functions from text-agent memory but faces a harder representation problem. MemGPT (working, recall, and archival memory), Generative Agents (observation streams transformed into higher-level reflections), MemoryBank (dialogue history, user modeling, and forgetting), Reflexion (feedback stored as reusable linguistic lessons), and Voyager (executable programs retrieved and composed across tasks) established these functions. Multimodal settings add spatial detail that captions may miss, continuous streams that must be segmented, interface observations bound to actionable screen coordinates, and embodied experience that must be associated across viewpoints, object identities, and changing locations.

• The paper distinguishes three often-conflated concepts. Long-context processing is a model's capacity to use information across extended input sequences; context management decides what enters the active context at each inference step; memory is experience-derived state persisting beyond the current inference step.

• Later sections are outlined but not reproduced in the available content. Sections 5 through 8 are described in the roadmap as covering how representation choices constrain memory management and corpus-level patterns, systems mechanisms under resource and coordination constraints, evaluation benchmarks and applications, and open challenges and future directions, but their detailed findings are not present in the provided text.

Methodology in Plain English

The authors did not run new experiments. They built an analytical taxonomy by surveying a large body of existing multimodal agent memory systems and organizing them along two independent questions. The first question asks what a system actually hands to its reasoner or action policy at the moment of use — the paper deliberately classifies by that downstream representation rather than by whatever format happens to be used for storage or as a retrieval index. The second question asks a behavioral question about change: does the system reuse memory as-is, does it update memory content under a fixed policy, or does it use feedback to change the policy itself? Systems that span multiple tiers or multiple representations are placed in multiple categories rather than forced into one. The authors then read across the two axes to spot patterns that are invisible when either axis is considered alone, derive design principles from those patterns, and use the gaps in the corpus to motivate future work. The framework is illustrated with a running household-robot example that traces a single experience into alternative representations and management tiers before being accessed and used for action.

Why This Matters

Impact on research. Memory is described as a core component of AI agents, and multimodal memory has lacked a shared vocabulary that separates representation choice from management behavior. By making the unit of analysis the operational lifecycle rather than the memory component, the framework lets researchers compare systems that look superficially different — a scene graph, a token bank, and a parameter update — on the same terms, and it makes explicit which failures are formation failures and which are management failures. The reported scarcity of deployment-time memory-policy evolution is a directly actionable signal about where the field is thin.

Real-world applications (drawn from the modalities and settings the paper covers):

• Personal assistants that accumulate user-specific experience across sessions and keep consistent behavior as users, tasks, and environments change.
• GUI and computer-use agents that need appearance bound to actionable screen coordinates, with relevant interface regions or screenshots retrieved at the right granularity.
• Household and embodied robots that must reconcile partial, viewpoint-dependent observations into a consistent dynamic object-level map and refine it when the environment changes.
• Video and audiovisual assistants that must localize relevant evidence inside continuous streams before it can be retrieved, which is why event- and entity-level indexing appears in the surveyed systems.
• Retrieval over heterogeneous image and document corpora, where routing queries to modality- and granularity-specific retrievers is one reported strategy.

Industry relevance. Anyone building persistent agent products faces the trade-offs this paper names: source fidelity versus abstraction, storage and retrieval cost versus recoverability, editability versus perceptual richness. The taxonomy gives engineering teams a checklist — decide your representation family, decide your management tier, and check whether your retrieval path can still reach the evidence you discarded — and the finding that most deployment sits at T2 with limited policy evolution suggests where near-term differentiation is available.

Future Directions

1. Recoverable and provenance-aware memory. Memory designs should keep connections to the supporting observations so that claims, decisions, and actions can be grounded and checked against specific evidence, rather than only against derived abstractions.

2. Temporal validity and cross-representation consistency. When the same entity or event is represented simultaneously in source media, modality-specific features, textual summaries, and structured world states, systems need mechanisms to reconcile new evidence across these linked representations and prevent stale or conflicting state from guiding later actions.

3. Self-improving memory management (T3) and bounded lifelong memory. The paper identifies the relative scarcity of deployment-time memory-policy evolution; using feedback to improve memory writing, maintenance, and retrieval is presented as an open direction, alongside the question of how memory stays bounded as an agent runs indefinitely.

4. Governed forgetting and lifecycle-aware evaluation. Managing what to discard, how to invalidate, and how to evaluate memory across an entire lifecycle — formation, management, access, and use — remains unresolved, and the paper argues current benchmarks do not cover it.

Target Audience

Researchers and graduate students working on agent architectures, multimodal LLMs, and long-context systems; practitioners designing persistent memory for deployed agents in robotics, GUI automation, and video or audio understanding; and benchmark designers who need a vocabulary for separating representation failures from management failures. The paper rewards readers who already know the standard text-agent memory lineage (MemGPT, Generative Agents, MemoryBank, Reflexion, Voyager) and retrieval-augmented generation, since it positions multimodal memory as an extension and complication of that lineage rather than starting from first principles.

Authors’ abstract

Multimodal models and agents have emerged to enable AI systems to learn from and interact with the real world more like humans by integrating information across multiple modalities. However, enabling multimodal agents to continually acquire useful information from complex, heterogeneous multimodal data remains a significant challenge as they move toward long-horizon tasks and lifelong settings. Addressing this challenge requires effective memory mechanisms that can maintain continuity across sessions, adapt to changing users and environments, and refine memory-management policies through accumulated experience. We introduce a two-axis analytical framework for multimodal agent memory structured around two key questions: how multimodal information is formed and represented (Axis I, Formation and Representation), and how its management adapts with accumulated experience (Axis II, Memory Management). Axis I, Formation and Representation, characterizes what information is retained and how it is encoded (e.g., modality-specific embeddings, textual and structured records, latent and parametric representations) to support subsequent reasoning and action. Axis II, Memory Management, distinguishes three tiers: T1 Static Memory reuses fixed memory; T2 Dynamic Memory updates memory through a stable policy; and T3 Self-Evolving Memory updates memory through an evolving policy. Based on this framework, we synthesize recurring design patterns and derive principles for balancing source fidelity with abstraction, maintaining temporal validity and provenance, grounding memory use in recoverable evidence, and evolving memory policies over time. Furthermore, we discuss the challenges and potential future directions for the memory mechanism of multimodal agents.