EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents

AI summary

Overview

• Research area: Continual learning and embodied AI — specifically, benchmarking whether self-evolving embodied agents can accumulate experience and transfer it to novel tasks.
• Technical level: Intermediate. Readers should be comfortable with agent evaluation protocols, in-context learning, and embodied simulation concepts such as state-transition graphs and action primitives.
• Scope: The paper introduces EmbodiedCLBench, the first embodied-simulation benchmark for evaluating the continual learning capability of self-evolving agents, and reports multi-harness, multi-method experiments showing that experience from basic tasks helps on harder unseen tasks but does not scale with the amount of experience.

What This Paper Is About

Continual learning research has mostly been confined to non-embodied settings — knowledge-intensive tasks, interactive digital environments, and long-term simulated worlds — while embodied benchmarks focus on perception, navigation, manipulation, and long-horizon planning rather than continual learning. The paper's goal is to fill that gap by building a benchmark in embodied simulation that measures whether an agent can learn from simple basic household tasks and then recombine the acquired skills to solve complex, unseen advanced tasks. The authors operationalize continual learning through compositional generalization, using basic tasks as test-time learning data that the agent accumulates before attempting advanced tasks.

Key Contributions

1. EmbodiedCLBench: The first benchmark in embodied simulation for evaluating the continual learning capability of self-evolving agents, built on the TongSim household simulation platform.
2. Compositional generalization as the evaluation principle: The key innovation is using compositional generalization to construct tasks, design evaluation protocols, and measure continual learning. Tasks are organized into three levels: atomic actions from TongSim, basic tasks (each composing a small number of atomic actions into a practical household operation), and advanced tasks (each combining elements from multiple basic-task categories).
3. A structured, offline, reproducible benchmark: 268 tasks total — 168 basic tasks spanning six categories and 100 advanced tasks — with each task represented as a state-transition graph whose nodes pair an intermediate state with first-person-view images and action options.
4. A comprehensive evaluation: Experiments across four agent harnesses (OpenClaw, Claude Code, Codex, Hermes) and two learning methods (In-Context Learning and Skill Learning), with two backbone models (Qwen3.8-Flash and GPT5.6-Terra), revealing consistent limitations and distinctive behaviors of current agents.

Main Findings

• Experience from basic tasks consistently helps on unseen advanced tasks: Zero-shot success on advanced tasks is around 10%, while both In-Context Learning (ICL) and Skill Learning raise it above 30% across all harnesses and models. Zero-shot success on basic tasks is around 50%. Example rows from the main results: Hermes with Qwen3.8-Flash goes from 8.0% zero-shot to 44.0% (ICL) and 33.0% (Skill Learning) on advanced tasks; Codex with GPT5.6-Terra goes from 11.0% to 52.0% (ICL) and 26.0% (Skill Learning).
• No consistent winner between ICL and Skill Learning: The paper reports no consistent advantage of one method over the other, conjecturing that both can extract meaningful information from agent experience.
• The benefit is not sensitive to task difficulty: As compositional complexity increases from 3 to 6 atomic actions, zero-shot success drops sharply (from 28.0% to 4.0%), and ICL falls from 72.0% to 42.0% between 3 and 4 atomic actions before leveling off. The gain over zero-shot drops initially but then stabilizes at around 30%. At the highest complexity of 6 atomic actions, zero-shot reaches 4.0% while ICL and Skill Learning still achieve 32.0% and 26.0%.
• Improvement does not scale with the number of learning samples: Going from k=0 to k=6 basic tasks, the largest jump occurs between k=0 and k=1, where a single learning sample raises success from 8.0%–10.0% to 28.0%–40.0%. Beyond k=1, success rate fluctuates without a clear upward trend across both harnesses and both learning methods.
• Continual learning changes decision behavior: Experience sharply reduces invalid (unexecutable) actions by 13%–20% across all harnesses and increases optimal actions, with suboptimal actions rising slightly by 3%–8%. ICL and Skill Learning shift behavior in the same direction, but Skill Learning consistently produces smaller shifts, which the authors suggest may reflect a difference in how experience is represented (trajectories versus skill summaries).
• Prior experience cannot be replaced by more tolerance: With the default allowance of 3 extra steps, and when varying extra steps from 0 to 6, ICL ranked highest, followed by Skill Learning, then zero-shot, at every budget. At the strictest budget of 0 extra steps, zero-shot success is 0.0%, while ICL and Skill Learning reach 15.0% and 7.0%. Experienced agents show a steeper increase as the budget grows, and the gap over zero-shot remains substantial even at 6 extra steps.

Methodology in Plain English

The researchers started from the intuition that a truly self-evolving agent should not relearn every new task from scratch — it should reuse skill elements it already mastered. To make that measurable, they built tasks in layers. Atomic actions are single interactions such as picking up a cup, composed of steps like walking to the cup, looking at it, and picking it up. Basic tasks chain a few atomic actions into household operations. Advanced tasks chain more atomic actions drawn from multiple basic-task categories.

Because they used the closed-source version of the TongSim simulator, which offers more diverse actions than the open-source version, online simulation was impractical. Instead they built each task as an offline state-transition graph. Each node is a unique environment state (for example, "near the desk and looking at the lamp"), paired with an image captured from TongSim and a set of action candidates. The shortest path between initial and goal states is found with breadth-first search, and distractor paths are created by expanding alternative executable actions from nodes along that shortest path. This gives three candidate types at each step: optimal actions that follow the shortest path, suboptimal actions that are executable but deviate, and invalid actions that cannot be executed in the current state.

Task descriptions and per-state action options were generated by Qwen3.7-Plus through a three-stage prompting pipeline (task description rewriting, per-state option rewriting, and cross-state option hardening designed to prevent linguistic shortcuts), then manually verified. Three annotators independently verified each generated sample for captured images, task descriptions, and per-state action options; a sample was retained only if all three annotators approved.

Evaluation has a learning phase and a testing phase. In the learning phase the agent completes basic tasks; in the testing phase it attempts unseen advanced tasks. Three settings are compared: Zero-shot (no learning phase), In-Context Learning (the full interaction trajectories of two related basic tasks per advanced task are kept in context), and Skill Learning (the agent writes a concise Markdown skill summary, and only that summary is provided in context). The primary metric is Success Rate. Step budgets are the shortest path length plus extra steps — 3 extra steps in testing and 6 in learning.

Why This Matters

The paper reframes continual learning as a first-class, measurable capability for embodied agents rather than an abstract aspiration, and it shows that current agents capture only a small fraction of the benefit that accumulated experience should provide. Because the benchmark is entirely offline and deterministic, results are reproducible and can be computed in parallel without running the simulator, which lowers the barrier for follow-up research.

Real-world applications:

• Household service robots, which must learn routines from simple operations and recombine them into multi-step chores such as tidying a room.
• Assistive and elder-care robots, where an agent that has learned basic manipulation and appliance control should transfer that knowledge to new, unanticipated requests.
• Smart-home and appliance control systems, which need to sequence actions like opening cabinets, switching devices on and off, and restoring rooms to a tidy, energy-efficient state.
• General-purpose embodied agents in simulation-based training pipelines, where a benchmark like this can serve as a pre-deployment check on whether an agent actually reuses experience or merely memorizes tasks.

Industry relevance: the findings suggest that simply supplying more experience to an agent is not enough — the bottleneck is extracting distinct, reusable information from additional interactions. That is a direct design constraint for anyone building self-improving agents, and it also implies that the surrounding agent harness matters, since the paper notes that performance across harnesses is close yet far from saturation and that designing harnesses for continual learning is a future direction.

Future Directions

• Designing continual learning methods that scale: Since a single learning sample gives most of the improvement and additional samples yield no consistent further gain, the paper identifies scalable continual learning — where success rate keeps improving as learning samples increase — as an open challenge.
• Building better agent harnesses: The paper calls designing harnesses specifically for continual learning a future direction, given that performance across the four harnesses is close but unsaturated.
• Understanding how experience is represented: The observed difference between ICL (larger behavioral shifts) and Skill Learning (smaller, possibly more focused shifts), despite no consistent success-rate advantage, raises the question of how experience should be stored and surfaced to an agent.
• Consolidating, generalizing, and reusing experience: The conclusion highlights the need for stronger mechanisms that can continuously consolidate, generalize, and reuse experience across tasks, and notes that how to enable agents to accumulate and exploit additional interactions remains open.

Target Audience

Researchers and engineers working on continual learning, embodied AI, and self-evolving agent systems; benchmark designers interested in compositional generalization and reproducible offline evaluation; and practitioners building agent harnesses or applying large language models to sequential decision-making in simulated physical environments. It is also useful for readers who follow household robotics and smart-home autonomy, since the task categories are grounded in everyday domestic activities.

Authors’ abstract

Continual learning requires models to accumulate past experience and transfer it to new tasks. Despite substantial research on continual learning, existing work remains largely confined to non-embodied settings, without extending to embodied environments. To address this gap, we introduce EmbodiedCLBench, the first benchmark in embodied simulation for evaluating the continual learning capability of self-evolving agents. With compositional generalization as the core evaluation principle, our benchmark evaluates whether agents can first learn from basic tasks and then recombine the acquired skills to solve complex, unseen advanced tasks. Based on EmbodiedCLBench, we conduct extensive experiments across multiple agent harnesses and learning methods, revealing consistent limitations and distinctive behaviors. Our results show that while advanced tasks can be hardly solved under zero-shot setting, the experience from basic tasks enables consistent improvement. Such benefit from the experience remains stable even if the task complexity increases. However, this improvement does not scale accordingly with the number of learning samples, and current agents cannot extract meaningful information from additional experience. Collectively, our findings highlight continual learning as a fundamental yet underexplored capability, and our benchmark offers valuable resources for designing effective self-evolving agents in embodied environments. Our code and dataset are available at https://pku-value-lab.github.io/EmbodiedCLBench-homepage.