Embodied Scene Rearrangement Planning

AI summary

Overview

Research area: Embodied AI and robotics, specifically scene-level rearrangement planning, task-and-motion planning (TAMP), and foundation-model-driven embodied agents.

Technical level: Intermediate to Advanced. The paper assumes familiarity with reinforcement learning (PPO), imitation learning / behavior cloning, diffusion policies, vision-language models, and task-and-motion planning. The task definition and benchmark statistics are accessible to a general reader, but the baseline architectures and evaluation design are aimed at robotics and embodied-AI researchers.

Scope (one sentence): The paper defines and benchmarks a new task — ESRP — in which an embodied agent must rearrange furniture in a full 3D indoor scene to match a top-down target layout using only egocentric observations, and shows that four representative baselines (imitation learning, reinforcement learning, a vision-language model agent, and a privileged planner) all perform far from complete.

What This Paper Is About

The paper addresses a gap between two existing lines of rearrangement research: tabletop manipulation of small objects, and scene-level rearrangement that is typically simplified to a 2D plane with full global observations. The authors introduce ESRP (Embodied Scene Rearrangement Planning), a task where an agent must reconfigure furniture in a realistic 3D scene to match a target configuration, given only a first-person RGB view, a top-down target layout image, and a binary signal indicating whether it is currently holding an object. The goal is to test whether embodied agents can align partial egocentric perception with a global spatial target and plan long-horizon action sequences to realize it.

Key Contributions

1. The ESRP task definition. A new embodied rearrangement task in which agents rearrange furniture in 3D scenes using only egocentric observations and a top-down target layout, with no access to global state and no ground-truth localization. The paper highlights three resulting challenges: partial observability, complex inter-object dependencies (including mutual occlusions and physical deadlocks), and dynamic scene evolution during the task.

2. ESRP-Bench, a benchmark built on OmniGibson. The benchmark contains over 5,400 initial–target scene layout pairs and over 8,200 objects, sourced from the 3D-FRONT dataset. In the dataset section the paper reports 5,495 distinct scenes spanning 24 room types, with 1 to 6 rearrangeable objects per scene, and 8,213 movable objects. Each scenario includes an initial layout, a target layout as a top-down image, and expert demonstration trajectories for imitation learning.

3. Three multi-level evaluation metrics. Success Rate (SR), Object Success Rate (OSR), and Relative Distance Reduction (RDR), plus evaluation stratified by difficulty level (Easy: 1 object; Medium: 2–3 objects; Hard: 4–6 objects).

4. Four baselines across three paradigms. ESRP-BC (imitation learning with a conditional diffusion policy), ESRP-PPO (reinforcement learning with PPO), ESRP-VLM (a ReAct-style multimodal agent over a pretrained vision-language model), and ESRP-PLAN (a hierarchical task-and-motion planning method with privileged global information), together with comprehensive experimental results and a failure-mode decomposition.

Main Findings

• The strongest baseline reaches only 30.20% SR. On the test set, ESRP-PLAN achieves SR 30.20%, OSR 36.59%, and RDR 0.6916. ESRP-PPO reaches SR 20.18%, OSR 20.35%, RDR 0.9845; ESRP-BC reaches SR 19.09%, OSR 20.31%, RDR 1.0170; and ESRP-VLM reaches SR 17.64%, OSR 18.97%, RDR 0.9830.

• Privileged global state helps most. ESRP-PLAN outperforms all other baselines on all three metrics, which the authors attribute to its access to the global environment state (current and target positions of all objects, the agent's pose, and full obstacle occupancy). Learning-based approaches relying on partial observations perform worse.

• RL edges out imitation learning. ESRP-PPO consistently outperforms ESRP-BC across all three metrics. The two are comparable in SR and OSR, but ESRP-BC's worse RDR indicates that errors accumulate and are amplified over long-horizon execution.

• A pretrained VLM alone is not enough. ESRP-VLM performs comparably to ESRP-BC but does not surpass ESRP-PPO, suggesting that off-the-shelf foundation-model priors are insufficient for the long-horizon spatial reasoning ESRP requires.

• Performance collapses as difficulty rises. On Easy scenes, ESRP-PLAN reaches SR 35.73% and OSR 35.73%, ESRP-PPO 33.33% SR/OSR, ESRP-BC 29.44% SR/OSR, and ESRP-VLM 27.64% SR/OSR. On Medium scenes, ESRP-PLAN gets 22.40% SR and 39.71% OSR, while ESRP-PPO falls to 0.54% SR, ESRP-BC to 0.91% SR, and ESRP-VLM to 0.00% SR. On Hard scenes, every baseline records 0.00% SR; only ESRP-PLAN and ESRP-BC place any objects, at 22.00% and 4.44% OSR respectively.

• Placement, not navigation or grasping, is the dominant failure mode. Partitioning rollouts into four stages (navigation failure, engagement failure, placement failure, complete success), Stage 3 placement failure accounts for 60.85% of ESRP-PPO rollouts, 62.85% of ESRP-BC rollouts, and 66.55% of ESRP-VLM rollouts. Only 4–6% of episodes terminate at the engagement stage and 11–13% stall during navigation.

• A stronger VLM backbone helps somewhat. Swapping Qwen3-VL-2B-Instruct for Gemini 2.5 Flash-Lite improves ESRP-VLM from 17.64% SR / 18.97% OSR / 0.9830 RDR to 23.01% SR / 23.75% OSR / 0.9450 RDR.

• Long-horizon planning is a bottleneck independent of perception. Even with perfect perception, ESRP-PLAN still achieves 0% SR on Hard-level scenes, showing that the combinatorial complexity of multi-object rearrangement is a challenge separate from the difficulty of inferring global state from local views.

• Qualitative behavior. In the successful example shown, the agent identifies the designated object and moves it to the target location within half of the available steps. In failure cases, the agent still lowers RDR by placing objects closer to their goals even when the scene is not fully rearranged.

Methodology in Plain English

The authors first define the task precisely. At every timestep the agent sees a 128×128 egocentric RGB image, a 128×128 top-down image of the desired layout, and a binary flag for whether it is grasping something. Its actions are discrete: move_forward, move_backward, turn_left, turn_right, fetch, and release. Turns are fixed at 22.5 degrees and translations at 0.25 meters. An object counts as correctly placed if the IoU between its current 2D bounding box and its target bounding box exceeds 0.3 on the horizontal plane. An episode ends when the timestep horizon is exhausted or every object is placed.

To build the benchmark, the authors take professionally designed layouts from 3D-FRONT as targets and render them as top-down orthographic images. Initial layouts are generated by working backward from the target: a rule-based agent iteratively moves each rearrangeable object away in random steps until it rests in a non-overlapping, collision-free position. This backward construction makes a feasible solution likely to exist, and the reverse of these trajectories becomes the expert demonstrations for imitation learning. Filtering criteria excluded rooms too cramped for a furniture-carrying agent, selected furniture slightly smaller than the agent so it remains maneuverable, and capped scenes at 6 rearrangeable objects.

The simulation is built with OmniGibson on NVIDIA Isaac Sim, using a Fetch-derived robot with the arm removed and a camera mounted overhead. Collision checking is continuous against the full 3D meshes of the robot, furniture, and environment, following HomeRobot. Grasping is abstracted: the agent can fetch objects within a 1.0 m radius of its center (the robot's diameter is 0.6 m), and objects attach instantly on interaction.

Four baselines span three paradigms. ESRP-BC uses a conditional diffusion policy with ResNet-50 encoders over egocentric history and the goal image, a 1D U-Net noise predictor, and a combined denoising plus cross-entropy objective. ESRP-PPO uses two visual encoders, a learnable gripper-state embedding, and a single-layer LSTM feeding an MLP, trained with a dense reward that includes a +10 success bonus, a ±1 arrival/leave signal, a potential term based on L2 distance from the held object to its target, a +0.01 per-step grasping bonus, and a living term set to 0. ESRP-VLM wraps a pretrained Qwen3-VL-2B-Instruct model in a ReAct loop with no task-specific fine-tuning. ESRP-PLAN assumes perfect perception and runs a TAMP framework: a task planner chooses the rearrangement order, and a motion planner executes a four-phase routine of BFS to the object (on a state space discretized at 0.05 m and π/32 rad), fetch, A* to the goal with a Euclidean heuristic, and release.

Training details include PPO with a learning rate of 1.5×10⁻⁴, value loss coefficient 5×10⁻³, batch size 2048, minibatches of 64, context length 64, and 6 million environment steps. The IL model trains for 500 epochs with 50 diffusion timesteps, AdamW at 10⁻⁴, weight decay 10⁻⁶, batch size 1024, cross-entropy weight 0.2, prediction horizon 8, history length 4, and DDIM sampling with 16 denoising steps; it predicts 8 actions, executes only the first 6, and takes a random action with probability 0.1. The dataset is split 90% training and 10% test. The IL model was trained on an NVIDIA A100 GPU; all other training and inference ran on an NVIDIA RTX 4090 GPU.

Why This Matters

This work pushes embodied rearrangement beyond tabletops and 2D abstractions into full 3D rooms under realistic partial observability, and shows quantitatively that current methods are far from solving it. By publishing the task, benchmark, metrics, and four baselines together, it gives the community a shared, reproducible target to measure progress against — including a diagnostic breakdown that points at precise placement as the dominant failure mode rather than navigation or grasping. The paper also frames its simulation-only and abstracted-grasping choices as deliberate, arguing that the isolated capabilities (long-horizon planning and spatial reasoning under partial observability) are exactly what future real deployments will demand.

Real-world applications:

• Domestic service robots: rearranging furniture in homes to match a preferred or requested layout.
• Hospitality and facility management: resetting banquet halls, hotel rooms, or event venues to a specified configuration.
• Warehouse and retail staging: repositioning large items such as cabinets, shelves, and desks into dictated arrangements.
• Assistive robotics: reconfiguring a living space for accessibility or mobility needs on request.

Industry relevance: the task sits at the intersection of mobile manipulation, semantic scene understanding, and long-horizon task planning, all of which are active investment areas for robotics companies building general-purpose indoor robots. The finding that a strong general-purpose VLM backbone improves results (17.64% to 23.01% SR) but still trails a privileged classical planner is directly relevant to teams deciding how much to rely on foundation models versus explicit planning.

Future Directions

• Overcoming placement under partial observability. Since placement failure accounts for roughly 60–67% of rollouts across learning-based baselines, targeted methods for precise object positioning from egocentric views are the most direct next step.

• Scaling to harder, longer-horizon scenes. Every baseline records 0.00% SR on Hard scenes, and even the privileged planner fails there, so new approaches to combinatorial multi-object ordering and cascading dependencies are needed.

• Additional sensing modalities and larger scope. The authors explicitly propose extending ESRP to house-level planning and incorporating depth, bird's-eye-view cameras, and LiDAR.

• Bridging simulation to real hardware. The paper notes that the absence of real-robot results mainly reflects a hardware bottleneck, since commodity platforms cannot yet reliably manipulate furniture-scale objects. Whether policies trained under the abstracted fetch/release primitives transfer with routine sim-to-real adaptation remains an open question.

Target Audience

Robotics and embodied-AI researchers working on rearrangement planning, mobile manipulation, and long-horizon task planning; researchers building or evaluating vision-language-model-based embodied agents; and benchmark designers who need a well-specified, difficulty-stratified task with multiple metrics and failure-mode analysis. Practitioners in indoor service robotics will benefit from the honest baseline numbers and the argument about which abstractions are worth making when isolating high-level planning. The paper is less suited to readers looking for solved methods or deployment-ready systems, since no baseline exceeds 30.20% SR overall and none succeeds on Hard scenes.

Authors’ abstract

This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout. Unlike prior rearrangement tasks, ESRP precludes global state access and introduces mutual object occlusions, reflecting the practical constraints of real-world robotic deployment. These factors make aligning partial egocentric observations with the global target layout particularly challenging for long-horizon planning. To facilitate research, we present ESRP-Bench, a comprehensive benchmark built on OmniGibson featuring over 5,400 scene pairs and 8,200 objects. We define three multi-level metrics to evaluate rearrangement quality and provide four baselines: a hierarchical task-and-motion planning method, a vision-language-model-based method, and two learning-based approaches (IL and RL). Experimental results demonstrate that current methods struggle to complete the task efficiently, highlighting ESRP as a challenging frontier for embodied agents in scene understanding and long-horizon task planning. This work serves as a stepping stone toward deploying intelligent agents in real-world scenarios. Project page: https://pie-lab.cn/ESRP/.