Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Exploration

AI summary

Overview

• Research area: Embodied AI, specifically long-horizon embodied exploration, multimodal large language models (MLLMs), and reinforcement learning.
• Technical level: Advanced. The paper assumes familiarity with vision-language navigation, reinforcement fine-tuning (GRPO), episodic memory banks, and embodied simulators such as Habitat-Sim.
• Scope in one sentence: The paper defines a new task (Long-term Memory Embodied Exploration), releases a paired benchmark (LMEE-Bench), and proposes MemoryExplorer, an MLLM fine-tuned with reinforcement learning to actively retrieve episodic memories while performing multi-goal navigation and memory-based question answering.

What This Paper Is About

Most embodied exploration benchmarks measure only whether an agent reached a goal or answered a question correctly, treating memory as a passive store and ignoring how exploration itself builds understanding. The authors argue that a truly capable embodied agent should accumulate episodic memories during long-horizon navigation and then actively recall them to answer follow-up questions and guide future decisions. Their goal is to unify the process of exploration with the outcome of task completion, and to train a model that proactively queries its memory rather than passively consuming pre-filtered snapshots.

Key Contributions

1. LMEE (Long-term Memory Embodied Exploration) task paradigm. A formulation that couples multi-goal navigation (where the agent builds an on-the-fly episodic memory bank) with memory-based question answering (where the agent must retrieve and reason over those memories).
2. LMEE-Bench dataset and benchmark. Built from HM3DSem scenes (145 training and 36 test scenes), containing 246 object categories, over 9,000 goals and questions, and 1,982 exploration trajectories. The full dataset comprises 1,982 tasks with 377,311 entries, split into 1,816 training tasks and 166 test tasks, with 11,684 training instances after sampling. Questions fall into five types (attribute, counting, location, relationship, state) with open-ended and multiple-choice answer formats.
3. MemoryExplorer. An MLLM-based framework trained with Reinforcement Fine-Tuning (GRPO) that generates code to invoke an external memory retrieval tool, then produces a single-step action, a frontier selection, and an answer.
4. Multi-Task Reward function. A reward combining action accuracy, frontier correctness, answer precision, and output format completeness, plus a consistency coefficient that penalizes inconsistent action-frontier pairs and a scaling factor that rewards successful tool use.

Main Findings

• LMEE-Bench subset results: MemoryExplorer reaches Total SR 23.53, SPL 14.99, MLLM-Score 43.62, and multiple-choice accuracy 65.52, versus RA-Mem (20.96 / 12.18 / 35.52 / 58.62), 3D-Mem (16.91 / 6.86 / 32.59 / 41.38), and Explore-EQA (13.24 / 7.66, with no QA evaluation since it lacks memory).
• Difficulty breakdown on LMEE-Bench: MemoryExplorer scores SR 31.91 / SPL 21.11 / Score 35.71 / Acc 68.57 on Easy; 21.35 / 14.03 / 48.14 / 63.83 on Medium; and 23.40 / 12.51 / 34.38 / 68.75 on Hard.
• GOAT-Bench "Val Unseen" results: MemoryExplorer achieves 46.40 Success Rate and 28.03 SPL, ahead of RA-Mem (42.81 / 21.95), 3D-Mem (37.05 / 20.26), and Explore-EQA (23.02 / 14.43). Reference GOAT-Bench baselines reported in the paper are Modular GOAT (24.9 / 17.2), Modular CLIP on Wheels (16.1 / 10.4), SenseAct-NN Skill Chain (29.5 / 11.3), and SenseAct-NN Monolithic (12.3 / 6.8).
• Retrieval-augmented QA comparison: Among off-the-shelf MLLMs answering questions from a post-exploration memory bank, Qwen2.5-VL-7B leads on the Total MLLM-Score (41.72, Acc 55.86), while Qwen3-VL-8B (37.93 / 60.69) and LLaVA-OneVision-7B (38.62 / 59.31) perform better on multiple-choice accuracy. The authors attribute this split to misalignment between cognitive understanding and action decision-making.
• Reward design matters: Removing the consistency penalty drops Total SR from 23.53 to 22.43; removing the tool-usage penalty drops it to 21.32.
• Training task design matters: Supervising navigation only yields SR 20.59 / SPL 12.34 / Score 39.14 / Acc 58.62, versus 23.53 / 14.99 / 43.62 / 65.52 once the memory-retrieval tool is included.
• Question-type variety has a non-linear effect: The "All" setting (combining task-progress questions and multiple-choice questions) gives the best GOAT-Bench results (SR 48.20, SPL 29.36), while the plain Multiple-choice setting gives the best LMEE-Bench results (SR 23.53, SPL 14.99, Score 43.62).
• Subset vs. full test set: On the full 166-task LMEE-Bench, MemoryExplorer records SR 22.05, SPL 14.26, Score 38.98, Acc 64.78, with the authors reporting no significant difference in conclusions between the 58-task subset and the full set.
• Real-world transfer: The agent was deployed on a ROSMASTER X3 robot with an Orbbec Astra Pro depth camera, a local computer, and a remote NVIDIA H200 server, and completed multi-goal navigation and memory QA in a meeting room and a reception room. These results are qualitative; no quantitative real-world metrics are reported.
• Failure modes: Memory-based QA failures stem from ambiguities in generated data, spatial understanding limits of MLLMs (retrieving the wrong memory), and incorrect descriptions even when the right memory is retrieved. In a reported full-task example with five navigation goals, the agent found the refrigerator, coffee machine, and nightstand but failed on the dresser and picture due to incorrect memory retrieval.

Methodology in Plain English

The researchers started by building a dataset rather than only a model. Using the real-world HM3DSem scene dataset, they fed region and object information into Qwen3-235B-A22B-Instruct to generate natural-language multi-goal navigation instructions, and used Habitat-Sim to plan step-by-step exploration trajectories from the agent's start position to each target. Each step records an action, observation, position, and rotation. An image tagging model labels objects in every image, producing a memory bank of (position, text feature, image feature) triples. Qwen3-VL-235B-A22B-Instruct then generates question-answer pairs about the navigation targets, which keeps questions answerable and lets the model gauge task progress. Tasks are graded easy, medium, or difficult based on the number of regions and goals and the distance to targets. Because simulators often cannot execute multi-step continuous actions, the authors sample from continuous runs of identical actions to form training data.

For the model, they fine-tune Qwen2.5-VL-7B-Instruct with GRPO. Given an instruction, multi-view observations, a goal-oriented question, and access to long-term memory, the model first emits a tool call with a query. A Python-side retrieval tool encodes the query and memory entries with CLIP and returns the top-k most similar memories by cosine similarity. The model then produces three outputs: an action, a frontier selection, and an answer. Training uses a multi-task reward over these outputs, scaled up when the tool is used successfully and scaled down when it fails, and clipped to [0, 1]. Training ran on 8 NVIDIA H200 GPUs for 160 steps with a global batch size of 128, a learning rate of 1e-6, a KL penalty coefficient of 0.1, top-k of 3, and a consistency coefficient of 0.5.

Evaluation uses Success Rate and SPL (a navigation episode counts as successful if the final position is within 1 meter of the target), plus MLLM-Score for open-ended answers — an MLLM (Qwen3-VL-30B-A3B-Instruct) scores each prediction from 1 to 5 given the ground truth, target observation, and prediction, with scores averaged and rescaled to 0–100.

Why This Matters

The paper shifts the evaluation of embodied agents from outcome-only metrics toward the exploration process and memory use, which the authors argue is a prerequisite for lifelong learning. It provides a benchmark that graders can use to compare memory-centric embodied agents, and a training recipe that turns memory retrieval into an active, learned behavior rather than a fixed filter.

Real-world applications:

• Home service robots that must remember where objects were seen during earlier tasks and answer follow-up questions from a user, such as the paper's water-bottle and tripod examples.
• Facility inspection and maintenance robots that patrol large, multi-room spaces and later report on the state of specific equipment observed during a prior round.
• Assistive and eldercare robotics, where an agent needs to recall the state of an appliance (open or closed) after navigating a home.
• Search and rescue or warehouse retrieval, where multiple goals must be visited in sequence and memory of earlier observations informs later decisions.

Industry relevance: the work targets the robotics and embodied AI stack, combining off-the-shelf multimodal LLMs, tool calling, and reinforcement fine-tuning with simulation-to-reality transfer demonstrated on commercial hobbyist-grade hardware.

Future Directions

• Multi-round memory retrieval. The current method focuses on single-round tool invocation due to multi-image input limits; extending to multi-round querying is an open problem.
• Reducing memory-retrieval errors. The failure analysis shows agents sometimes retrieve the wrong memory or misdescribe a correctly retrieved one, pointing to better spatial grounding and retrieval filtering as needed.
• Broader sim-to-real evaluation. Real-world testing was qualitative on two office environments; quantitative physical-robot benchmarks remain to be established.
• Scaling question types and task difficulty. The ablation shows non-linear interaction between question types and performance, and the full LMEE-Bench required 29 hours per method to evaluate, suggesting efficiency and coverage both need further work.

Target Audience

Researchers and graduate students working on embodied AI, vision-language navigation, and memory-augmented agents; practitioners building multimodal LLM policies with reinforcement fine-tuning; and benchmark designers interested in evaluating the exploration process rather than only task outcomes. Readers should be comfortable with reinforcement learning terminology and MLLM tool-calling pipelines; those without that background will find the task motivation and results accessible, but the methodology harder to follow.

Authors’ abstract

An ideal embodied agent should possess lifelong learning capabilities to handle long-horizon and complex tasks, enabling continuous operation in general environments. This not only requires the agent to accurately accomplish given tasks but also to leverage long-term episodic memory to optimize decision-making. However, existing mainstream one-shot embodied tasks primarily focus on task completion results, neglecting the crucial process of exploration and memory utilization. To address this, we propose Long-term Memory Embodied Exploration (LMEE), which aims to unify the agent's exploratory cognition and decision-making behaviors to promote lifelong learning. We further construct a corresponding dataset and benchmark, LMEE-Bench, incorporating multi-goal navigation and memory-based question answering to comprehensively evaluate both the process and outcome of embodied exploration. To enhance the agent's memory recall and proactive exploration capabilities, we propose MemoryExplorer, a novel method that fine-tunes a multimodal large language model through reinforcement learning to encourage active memory querying. By incorporating a multi-task reward function that includes action prediction, frontier selection, and question answering, our model achieves proactive exploration. Extensive experiments against state-of-the-art embodied exploration models demonstrate that our approach achieves significant advantages in long-horizon embodied tasks. Our dataset and code will be released at https://wangsen99.github.io/papers/lmee/