Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation

AI summary

Overview

Research area: Embodied AI and computer vision — specifically vision-language-model (VLM) driven mobile manipulation agents that must operate in open-world physical environments.

Technical level: Advanced. The paper assumes familiarity with POMDPs, vision-language models, supervised fine-tuning, reinforcement learning (GRPO/GSPO), sim-to-real transfer, and VLA (vision-language-action) policies.

Scope: The paper presents REAL, a framework plus benchmark plus physical deployment for training an embodied agent that actively explores a scene through RGB perception (no privileged simulator state), asks a simulated user to resolve ambiguous instructions, and transfers zero-shot to a real dual-arm mobile robot.

What This Paper Is About

Most embodied-agent benchmarks give the agent an unfair advantage: they hand it a global list of objects, exact 3D target coordinates, or special "find object by category" APIs that do not exist on real robots. They also assume the user issues a perfectly clear instruction. The REAL framework removes both crutches — the agent must discover objects by physically driving around a cluttered room using only camera images, and must ask clarifying questions when the instruction is ambiguous. The goal is to train an agent that survives the trip from simulation to a real household robot without the reality gap breaking it.

Key Contributions

1. REAL framework: A sim-to-real-consistent embodied agent framework with no oracle perceptual APIs. The agent receives only physically obtainable priors (a map and receptacle labels) and must ground objects through active exploration of raw RGB observations, combined with a simulated user that supplies ambiguous instructions and answers follow-up queries.

2. Two-stage training pipeline: An automated task-generation and filtering pipeline produces expert trajectories, which are used to align Qwen3-VL-8B-Instruct to the tool interface via supervised fine-tuning, followed by online reinforcement learning with Group Sequence Policy Optimization (GSPO) to build adaptive replanning and error recovery.

3. REAL-Bench: A benchmark of 241 task instances across four families — Furniture-Distractor Pick-and-Place (FDP, 72), Furniture & Object-Distractor Pick-and-Place (FODP, 56), Furniture-Distractor Open/Close (FDO, 48), and Simulator-User-Loop (SUL, 65).

4. Physical validation: Zero-shot deployment of the high-level policy on an ARX LIFT2 dual-arm mobile robot, achieving 78.3% end-to-end success across 60 real-world episodes.

Main Findings

• Trained agent beats commercial closed-source VLMs on interactive tasks: The SFT+RL agent reaches 56.9% success on the SUL split, exceeding the best zero-shot baseline (gemini-3-pro-preview at 53.8%) and every other proprietary model tested, including gpt-5 and gemini-2.5-pro.

• Zero-shot base model is essentially non-functional: Qwen3-VL-8B-Instruct scores 0.0% on FDP, FODP, and FDO and 1.5% on SUL before training, and even the much larger Qwen3-VL-235B-A22B scores only 22.2% on FDP — showing that scale alone does not solve tool-interface alignment.

• Behavior cloning overfits with extended training: One epoch of SFT gives 45.8% on FDP; two epochs raises FDP to 65.3% but drops FODP from 30.4% to 28.6%. Strict imitation of expert trajectories weakens robustness under open-vocabulary clutter.

• RL restores generalization lost to overfitting: Applying GSPO to the one-epoch checkpoint lifts FODP to 33.9% while keeping standard-task performance intact, and produces the largest gain on the interaction-heavy SUL split (36.9% to 56.9%). The paper argues social interaction emerges from environmental feedback rather than prompt engineering.

• Vision grounding is the persistent bottleneck: Manual failure analysis over 100 episodes from the best SFT+RL checkpoint found 53 failures, dominated by Object Confusion (25), followed by Key Action Missing (17), Lost Memory (9), and Other (2). The frozen vision encoder means RL can only influence perception indirectly.

• Real-world transfer works, but interaction remains hardest: On the physical robot, overall success is 78.3% over 60 episodes. FODP reaches 100%, FDO 80.0%, but SUL only 55.0%. Low-level VLA primitives executed successfully in 85.3% of 600 attempts, with zero unrecoverable system crashes.

• Long-horizon causal planning transfers: In a food-preparation task the agent opened the microwave before fetching the bread, rather than navigating to the cabinet first — evidence that the compact structured memory prevents temporal hallucination and respects causal ordering.

• Efficiency cost is real: Per-episode VLM inference latency ranges from 53.5s (FODP) to 80.5s (SUL), averaging 68.4s, with path efficiency (SPL) of 63.1% overall.

Methodology in Plain English

The researchers built a simulated household environment on top of the GRUtopia platform, drawing on 7 high-fidelity scenes, 11 room types, 100 interactive receptacles, and more than 2,500 object instances across 639 categories. The agent never sees a global object list, target poses, or simulator state. It receives only a pre-built map with labeled receptacles (like "table_2") and must use a chain of navigation and perception tools to find things.

That toolchain has four exploration stages: nav_to drives between receptacles using the map; walk_around orbits a receptacle to see past occlusions using open-vocabulary 2D detection; show_object_by_category isolates objects of a requested category in the current camera view; and gaze_at approaches and centers a chosen object. Detected objects are back-projected from 2D masks into 3D and stamped with visual IDs that appear in the agent's RGB observation, so downstream manipulation commands (pick, place, open, close) can reference them.

A simulated user, driven by gemini-3-flash, is embedded directly in the environment. It issues deliberately vague instructions, watches the agent's trajectory, and answers questions when the agent calls the ask tool. This creates the multi-turn intent-alignment challenge.

Decision-making is framed as a POMDP. Rather than appending every past observation to the prompt (which would explode context and cause hallucinations), the policy keeps a compact memory: a running history summary and a current task-phase analysis, plus the immediately preceding action. Each step it outputs a reasoning trace, an updated memory, and a tool call.

Training happens in two phases. First, a rule-based planner generates expert trajectories in the simulator, and gemini-3-pro annotates the reasoning traces and memory states; the model is then fine-tuned by behavior cloning on these prompts. Second, GSPO — a critic-free RL method that applies sequence-level rather than token-level ratio clipping — optimizes the policy using rewards grounded in a World Graph representation of scene state. The reward matches each observed state transition against remaining goal deltas, with extra penalties for wasted time and failed actions, plus a difficulty-conditioned budget of at most three user queries.

For physical deployment the high-level policy runs unchanged behind an MCP (Model Context Protocol) server, so the real robot is simply an alternative backend. Abstract manipulation commands are executed by a fine-tuned π0.5 vision-language-action model handling the low-level motor control.

Why This Matters

Impact on research: The paper makes a methodological argument that is easy to overlook but hard to argue with — benchmarks that hand agents oracle perception are measuring the wrong thing. By showing that a properly aligned 8B model outperforms much larger zero-shot commercial VLMs on interactive tasks, it suggests the field's bottleneck is interface alignment and closed-loop feedback, not raw model scale. The finding that behavior cloning overfits while RL restores generalization is a concrete, quantified data point for the ongoing debate about imitation versus reinforcement in embodied learning.

Real-world applications:

• Household service robots that must find objects in cluttered, unfamiliar homes without a curated inventory — the FODP setting directly mirrors "find my keys somewhere in the kitchen."
• Assistive robotics for elderly or disabled users, where the user often cannot precisely describe object locations and the robot must ask clarifying questions rather than guess.
• Warehouse and logistics manipulation, where articulated tasks (opening containers, cabinets, appliances) and distractor-rich shelves are routine.
• Robot fleet deployment, where the MCP-based tool interface means one trained high-level policy can be swapped across different physical backends without retraining.

Industry relevance: The standardized tool interface decoupling reasoning from hardware is directly commercially useful — it means a cognitive policy can be developed once and ported across robot form factors. The 78.3% real-world success rate with zero system crashes and 85.3% primitive executability represents a credible reliability signal for companies evaluating whether VLM-driven agents are production-ready. The 68.4s average VLM latency per episode, however, is also a clear signal that inference cost remains a deployment barrier.

Future Directions

1. Fix the perception bottleneck. The dominant failure mode is object confusion, but the vision encoder is frozen throughout training. Unfreezing it selectively, or adding explicit perceptual supervision and fine-grained grounding objectives, is the most obvious next lever.

2. Move beyond monolithic receptacles. The current environment treats a cabinet as a single entity; adding part-level segmentation (individual shelves, drawers, compartments) would sharpen both exploration and manipulation precision.

3. Encode richer compositional goals. The task space is limited to cross-receptacle rearrangement. Temporal ordering constraints and spatial relational goals ("place the cup to the left of the plate") are not yet represented.

4. Make the simulated user harder. The current user is bounded — preferences are restricted to in-scene objects and responses are anchored to the agent's trajectory. Real users introduce new goals mid-task and communicate through implicit cues, which the framework does not yet test.

5. Address long-horizon memory loss. Nine of 53 analyzed failures were "Lost Memory" errors, suggesting the compact summary field degrades over extended episodes and could benefit from more structured or hierarchical memory.

Target Audience

This paper is most valuable to embodied AI and robotics researchers working on vision-language agents, sim-to-real transfer, and mobile manipulation — particularly those building benchmarks or training pipelines for VLM-based planners. It is also relevant to RL researchers interested in sequence-level policy optimization applied to multimodal agents, and to industry engineers evaluating whether a VLM high-level planner plus a VLA low-level controller is a viable architecture for commercial robot deployment. Readers without background in POMDPs, behavior cloning, or policy-gradient methods will find the training sections dense, though the environment design and experimental results are accessible to a broader robotics audience.

Authors’ abstract

Real-world deployment of embodied agents requires active exploration, visual grounding, and interactive intent disambiguation. However, existing frameworks often rely on privileged simulator states or assume complete instructions, bypassing realistic deployment challenges. To bridge this gap, we present REAL, an agentic framework for open-world mobile manipulation. REAL establishes sim-to-real-consistent environment APIs without oracle perception and integrates a simulated user to enable human-in-the-loop interaction. Within this environment, we design diverse task compositions to drive data collection, supervised fine-tuning, and online reinforcement learning, systematically optimizing agent performance. To comprehensively evaluate this approach, we introduce REAL-Bench, a benchmark spanning 241 tasks across active exploration, visual distraction, articulated manipulation, and interactive disambiguation. Experimental results demonstrate that our trained agent outperforms leading commercial closed-source VLMs on interactive tasks with a 56.9% success rate. Further empirical analysis reveals that our hierarchical training pipeline successfully aligns the model's tool-use capabilities while maintaining robust open-vocabulary reasoning under extended exploration horizons. Finally, we deploy and evaluate our framework on a physical dual-arm mobile robot, where it achieves a 78.3% end-to-end success rate over 60 real-world episodes. These physical trials demonstrate robust zero-shot transferability to unseen household scenarios, validating that our sim-to-real-consistent design successfully bridges the reality gap for long-horizon mobile manipulation. Code is available at https://github.com/InternRobotics/REAL.