EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks

AI summary

Overview

Research area: Embodied AI and multimodal large language models (MLLMs) — specifically, how well agents retain, update, and reuse memory during long-horizon interaction in 3D physical environments.

Technical level: Intermediate. The paper is readable without deep reinforcement learning background, but assumes familiarity with multimodal LLMs, embodied agents, and benchmark design.

Scope in one sentence: The paper diagnoses four failure modes of embodied memory, introduces a 2,554-episode interactive benchmark (EMem-Bench) targeting those four failures, and proposes a three-part external memory system (EMem) plus a trained 8B policy (EMem-8B) that outperforms current memory baselines.

What This Paper Is About

LLM agents handle long digital tasks well partly because they can keep long-term memory, but agents operating in the physical 3D world must remember something harder: the causal chain linking what they saw, what they did, and what happened as a result, across long and noisy interaction histories in environments that keep changing. Existing embodied benchmarks do not directly test whether an agent can build, update, and reuse this kind of memory. The paper fills that gap with a benchmark whose episodes force an agent to consolidate an offline interaction history into memory and then act on it in the environment to complete a later task.

Key Contributions

1. A four-way diagnosis of embodied memory failure. By manually inspecting 100 failed trajectories from Gemini-3-Flash and Qwen3-VL-32B on EmbodiedBench's EB-ALF and EB-Hab suites, the authors group failures into forgetting fine-grained visual cues (35%), overlooking changes in the environment (18%), failing to record world state revealed by interaction outcomes (31%), and failing to generalize from prior experience (7%) — about 91% of failures in total.

2. EMem-Bench, a memory-centric interactive benchmark. It contains 2,554 episodes across four task families (Passive Observation, Dynamic Tracking, Interaction Failure, Experience Generalization). Each episode provides a multimodal interaction history, a target task, and an action space; the agent must extract the needed evidence from the history and then act in the simulator until the environment reaches a state satisfying the task.

3. A systematic evaluation. The authors evaluate 16 open-source and proprietary MLLMs along with representative multimodal memory systems (MIRIX, MemVerse, TeleMem, MMA) under a fixed GPT-5.4-mini backbone, and add two metrics: success rate (SR) per task family with an equally weighted mean, and Error Recurrence Rate (ERR), which flags actions that reuse stale locations, repeat known failed interactions, or violate learned regularities.

4. Embodied-Memorizer (EMem) and EMem-8B. EMem is an external memory system with spatial, event, and scene memories. EMem-8B is a Qwen3-VL-8B policy trained via supervised fine-tuning to decide what to write, which memories to query, and how to turn retrieved information into action sequences.

Main Findings

• Current models are weak and uneven. Gemini-3-Flash, the strongest full-context model, reaches only 64.2% average SR, and all but one of the evaluated open-source models score below 45%. Profiles vary sharply: Qwen3-VL-32B obtains 20.3% SR on Dynamic Tracking but 63.1% on Experience Generalization.

• Embodied specialization does not automatically buy memory ability. RynnBrain-8B reaches 3.6% and RoboBrain2.5-8B-NV 20.0% average SR, both below Qwen3-VL-8B-Instruct at 25.4%; Cambrian-S-7B reaches 0.8%; MiMo-Embodied-7B reaches 38.8%; Robotics-ER 1.5 reaches 52.4% (evaluated on a 10% subset because of API service instability).

• EMem leads among memory systems. With a fixed GPT-5.4-mini backbone, EMem reaches 58.9% average SR, 21.1 points above the strongest alternative, TeleMem, while reducing ERR by 11.4 points.

• EMem generalizes across backbones. It improves average SR by 10.8, 14.1, and 16.4 points on three open-source backbones and by 23.6 points on proprietary GPT-5.4-mini. Training Qwen3-VL-8B to operate the memory loop gives a further 9.8-point gain, producing EMem-8B at 46.0% average SR, 20.6 points above its backbone.

• More raw visual history hurts. On a fixed 800-episode subset (200 per family) with GPT-5.4-mini, adding as many historical RGB observations as the context budget permits drops average SR from 36.6% to 26.2% — a paired decrease of 10.4 points — while raising input tokens from 41,770 to 206,489 (4.9 times) and model-step latency from 4.1 to 39.1 s/step (9.5 times). Performance declined in every history-length quartile.

• All three memories matter, and they play different roles. Removing spatial memory drops average SR from 58.9 to 47.9, removing event memory to 46.9, and removing scene memory to 47.3. Dynamic Tracking depends most on spatial memory, Interaction Failure and Experience Generalization most on event memory, and Passive Observation most on scene memory.

• Failure analysis points to memory use, not action execution. Across 400 failed GPT-5.4-mini episodes (100 per family) under full context, memory-use errors dominate Passive Observation, Dynamic Tracking, and Experience Generalization; Interaction Failure failures are mostly contextual reasoning errors; action errors remain uncommon across all four families. Separately, among 400 sampled failures, 61.3% begin when the model acts on a memory contradicted by its interaction history.

• Memory training transfers selectively. EMem-8B improves EB-ALF average SR by 5.0 points over Qwen3-VL-8B (23.3 vs 18.3) and BLINK macro accuracy by 1.3 points (56.9 vs 55.6), gains 0.5 points on MMMU-Pro (45.1 vs 44.6), and is comparable on HallusionBench with a 0.4-point decrease (65.8 vs 66.1).

Methodology in Plain English

The authors start from evidence rather than assumption: they watch real failures, categorize them by hand, and turn those categories into testable task families.

Benchmark construction. Using AI2-THOR single-room scenes, they compose multi-room virtual homes and extend the action space with LeaveRoom, EnterRoom, and MoveToRoom; they also filter ProcTHOR multi-room homes for room connectivity, navigable space, object visibility, and executable interactions. In each environment they designate a cue room supporting the required memory relation and use other connected rooms for unrelated activity.

Building the memory cue. Each task family gets a differently constructed cue: for Passive Observation, an object visible in the current view that must later be retrieved; for Dynamic Tracking, an object whose location or state is changed in the simulator so that the later task depends on the updated state; for Interaction Failure, an executed failed action whose feedback reveals an otherwise invisible state (a locked drawer, a blocked receptacle); for Experience Generalization, Claude-4.6-Sonnet generates a structured experience with an underlying regularity, several correction cases, and a target task, which is then grounded in concrete scene objects and executed.

Distractors and validation. A PDDL planner inserts unrelated trajectories before and after the cue so the cue can appear anywhere in the episode, and any trajectory that touches critical objects, reveals the answer, alters required state, or cannot be executed is discarded. Automated checks confirm navigability, executable interactions, completability, and that decisive evidence is present without being revealed in distractors; manual review rejects ambiguous instructions, implausible trajectories, and non-unique cues. This process rejected 143 episodes, leaving 2,554 that pass both automated and manual screening.

The memory system. EMem keeps three episode-local stores, each initialized empty per trajectory. Spatial memory is a scene-scoped entity graph tracking entities, properties, and relations such as in, on, held_by, and seen_in; when a new relation arrives, conflicting relations in the same family are replaced so the graph reflects latest known state. Event memory is an append-only sequence of action outcomes (successes and failures, with action, feedback, summary, scene scope, and supporting identifiers), and it also stores aggregate rules formed after at least two related corrections, or after one explicit persistent failure. Scene memory stores selected visual snapshots with model-generated captions, identified entities, scene scope, and a reference to the original RGB observation, using an admission gate that suppresses near-duplicate views.

Evaluation. Predicted action sequences are executed in the simulator; an episode counts as successful only when the environment reaches the target terminal state. SR is reported per family and averaged with equal family weights to prevent the two larger families from dominating.

Why This Matters

Impact on research. The paper reframes embodied memory from a vague capability into four separately measurable demands, and its negative result about raw visual context is a direct challenge to the common assumption that longer multimodal context substitutes for memory. It also supplies a concrete baseline — a modular memory backed by a deterministic store plus a supervised policy — for others to beat.

Real-world applications (plausible settings implied by the work):

• Home robotics, where an assistant must remember where a specific object was seen among many similar ones and in which room each instance lives.
• Assistive or service robots that must track objects moved by people or by other robots and act on the newest state rather than what they last saw.
• Manipulation systems in which failed grasps, locked drawers, and unusable receptacles reveal constraints that vision cannot show, and where that feedback must shape later choices.
• Multi-task agents that should carry a correction learned on one object (say, where cutlery belongs) over to a new object of the same category.

Industry relevance. The benchmark gives teams a way to compare memory components on equal backbones rather than confounded by model scale, and the ablation results give a practical allocation guide: spatial memory for tracking, event memory for interaction outcomes and experience transfer, scene memory for visually dense observation tasks. The context experiment also has direct cost implications, since 4.9 times the tokens and 9.5 times the per-step latency bought a 10.4-point performance loss.

Future Directions

• Closing the gap between storing and using. The failure analysis locates most errors in memory use and contextual reasoning rather than action execution; what training signal or architecture would make a model reliably act on retrieved evidence it already has?
• Generalizing the four families beyond the constructed settings. Experience Generalization is the smallest family at 203 episodes and the one where several models already score highest (Gemini-3-Flash at 75.4%), so how the picture changes with more varied regularities is untested here.
• Making memory training transfer more broadly. EMem-8B's gains are largest on interactive embodied execution (EB-ALF, +5.0) and smaller on general multimodal benchmarks, which raises the question of whether memory-operation training can be scaled without trading off general perception.
• Evaluating at greater scale and with real hardware. The benchmark runs in simulated AI2-THOR and ProcTHOR homes, and one model was only evaluated on a 10% subset because of API instability, so robustness of these conclusions to real robots and to fuller API coverage remains open.

Target Audience

Researchers and engineers working on embodied agents, multimodal LLMs, and robot learning who need a way to measure whether an agent actually remembers and updates its world state; benchmark designers interested in evaluation methodology for memory; and practitioners building long-horizon robotic assistants who want evidence on which memory components contribute to which task types, and on why simply feeding more images into the context window is not a solution.

Authors’ abstract

Long-horizon embodied interaction requires agents to retain and continually update information about the environment as they observe, act, and encounter change. Yet current agents struggle to maintain such memory reliably. Our analysis traces this limitation to four key deficiencies: weak fine-grained visual memory, unreliable dynamic world-state tracking, failing to record world state revealed by interaction outcomes, and limited generalization from prior experience. However, existing benchmarks do not directly assess these memory capabilities during long-horizon embodied interaction. To address this gap, we introduce EmbodiedMemory-Bench (EMem-Bench), comprising 2,554 interactive episodes across four task families. EMem-Bench requires agents to build and update memory from interaction history, then use it to complete a later task by acting in the environment. We further present Embodied-Memorizer (EMem), an external memory system that organizes embodied experience into spatial, event, and scene memories. We also train EMem-8B, an 8B policy that manages and uses these memories. We evaluate a diverse range of open-source and proprietary MLLMs and representative multimodal memory systems. Results show that current models remain weak and uneven across the four challenges. Under matched backbones, EMem achieves the best overall performance among the evaluated memory systems and improves both open-source and proprietary models, while EMem-8B further improves over its backbone. Project page: https://zju-omniai.github.io/EmbodiedMemoryBench/