World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

AI summary

Overview

Research area: Robotics — robot manipulation with general-purpose vision-language models (VLMs), agentic control, and embodied skill learning.

Technical level: Intermediate. The paper assumes familiarity with VLAs, VLM prompting, motion planning, and standard manipulation benchmarks, but its central idea (change what the model sees and how its decisions take effect, rather than retraining it) is explained in largely conceptual terms.

Scope: The paper introduces World Action Agent (WAA), a multi-agent harness that lets a general-purpose VLM pilot a robot through basic action primitives inside a shared visual action workspace, and evaluates it on LIBERO-Pro and robosuite.

What This Paper Is About

Most systems that put VLMs to work in robot manipulation use them indirectly — to predict constraints, write programs, or choose among options shown in a picture of the scene — so the model never actually gets a world in which to act. The paper argues that three things are missing from these visual interfaces: observation is not centered on the interaction, actions take effect the instant they are issued and cannot be tried first, and offsets seen in an image must be rewritten as coordinates before they can be corrected. WAA's goal is to let a general-purpose VLM make and revise decisions throughout execution using only basic action primitives, serving as a robot pilot without being retrained for action prediction.

Key Contributions

1. World Action Agent. A multi-agent harness through which general-purpose VLMs pilot robots with basic tools, observing, rehearsing, and correcting every action within a single visual action workspace composed of a global view plus automatically selected Contact views.

2. Learning through the harness. The same workspace supports non-parametric skill evolution from expert videos and human teaching (via Learner, Editor, and Reviewer roles), and the distillation of interaction traces into smaller VLM pilots.

3. State-of-the-art results. WAA reaches 75.6% average success on LIBERO-Pro and transfers to robosuite without further learning, and a 9B VLM (Qwen3.5-9B) can be trained on harness traces to pilot the same interface.

Main Findings

• Best average on LIBERO-Pro with frozen, out-of-domain skills. WAA with skills evolved only from LIBERO-90 reaches 75.6% average success across the six splits, above ASPIRE (72.0%), beating end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone. The skill library is frozen before evaluation: no LIBERO-Pro or robosuite rollout, failure log, or label updates skills, prompts, or model parameters.

• Largest gains appear on the Spatial splits. WAA reaches 80.0% and 73.3% on Spatial Pos. and Task, where success depends on centimeter-scale placement relations, compared with 51.0% and 60.0% for ASPIRE. ASPIRE remains strongest on both Object splits and on Goal Pos. (98.0% and 95.0% Object; 81.0% Goal Pos.).

• Even zero-shot, the harness works. Without any skills, WAA reaches 28.9% average, above π0.5 (12.8%) and both CaP-Agent0 evaluations, with 53.3% and 46.7% on the two Object splits.

• Skills contribute the largest single gain. Average success rises from 28.9% without skills to 43.3% with text-only seed skills and 75.6% with evolved skills; Spatial success rises from 13.3% and 6.7% to 80.0% and 73.3%. Text-only seed skills help inconsistently and even slightly lower Goal Pos. success, whereas evolved skills improve on zero-shot in all six splits.

• One demonstration can be enough for a new skill. Starting from text-only seed skills with no stove-specific procedure, one LIBERO-90 demonstration of stove activation yields a skill with which WAA succeeds in all ten executions of the LIBERO-Pro stove task. The Reviewer rejects a first draft that treats knob motion as success; the accepted skill requires an explicit activation signal. Two runs take 23 and 37 decision steps.

• Frozen LIBERO skills transfer to robosuite. WAA without skills already succeeds in every cube lifting and stacking trial (100.0%) and reaches 60.0% on restacking; adding the frozen LIBERO skills raises restacking to 100.0% and the average to 100.0%, versus 63.3% for CaP-Agent0 with RATs skills and 49.3% for CaP-Agent0 without skills.

• WAA is cheaper and faster than the visual-harness baseline. WAA averages $0.1996 API cost, 31.05 model calls, 150.47 seconds, 193.62K input tokens, and 5.45K output tokens per episode, against $0.5021, 119.80 calls, 874.12 seconds, 347.10K input tokens, and 62.90K output tokens for Show-Harness.

• The harness works across point-cloud sources. Replacing the simulator's point cloud with fused RGB-D views or VGGT reconstruction lowers average success only to 71.1% and 68.9% — drops of 4.5 and 6.7 points — and both still surpass every baseline on the two Spatial splits.

• A 9B VLM can learn to pilot the same harness. Fine-tuning Qwen3.5-9B with LoRA (rank 8, 10 epochs, learning rate 5×10⁻⁵) on 1,774 main-agent tool calls from 112 successful Gemini 3.7 Flash episodes raises success from 0.0% to 55.0% in domain and from 1.7% to 43.3% out of domain, replacing only the main agent while sub-agents and grounding still use Gemini 3.7 Flash.

Methodology in Plain English

WAA does not retrain the VLM. It changes what the model sees and how its decisions take effect, wrapping a general-purpose VLM in a harness with three coordinated designs.

Interaction-centered Canvas. Instead of a fixed global camera, the harness actively solves for camera parameters for two "Contact views" that expose the interaction region, subject to constraints: the two views' horizontal projections must be orthogonal so an alignment error can be read along two independent directions, while redundancy and viewpoint jumps are penalized so spatial relations stay comparable across steps. Combined with the global view, these form the Canvas, with every view keeping its projection calibration so image-space annotations map directly to 3D. The method only needs a scene point cloud in the robot base frame, which can come from a simulator, fused calibrated RGB-D cameras plus forward-kinematic hand geometry, or a feed-forward reconstruction model such as VGGT.

Action rehearsal. An action is an editable proposal, not a one-shot output. The agent localizes a target on the Canvas and specifies a pose; the harness solves inverse kinematics, plans with cuRobo, overlays a translucent robot at the target configuration in every view, and reports feasibility. The agent can inspect approach direction, prospective contact, and clearance, and revise. For finer work, the main agent delegates to an Imagination Agent running in a separate context, which iteratively edits and replans while the physical scene stays unchanged. Only a proposal with an executable plan becomes motion. Tools split into query, proposal, and execution classes; query and proposal tools leave the physical world unchanged.

In-view correction. After a target-level motion, the agent drags from a reference point (the gripper or a visible point on the held object) to the desired location in a Contact view, and the harness converts this image-space intent into a bounded end-effector displacement using the view's calibration. Local motions are bounded — 0.03 m horizontal/downward, 0.08 m upward, and 0.08 m total diagonal — and the two orthogonal Contact views support corrections along complementary directions.

Skill evolution. The library starts from text-only seed skills written by Codex (GPT-5.5) from the harness API. It is then enriched from one expert trajectory per LIBERO-90 source task, processed by three GPT-5.5-driven roles: a Learner that segments video at state changes and extracts knowledge candidates citing their source frames (without reading existing skill text, so new evidence is not steered by prior conclusions), an Editor that adds, revises, retires, or defers skills, and a Reviewer that independently checks changes against source evidence. Human teaching through a Canvas-GUI — where people use the same tools as the agent — supplies recovery guidance that goes through the same Editor–Reviewer loop.

Learning to pilot. Interaction traces from successful, clean episodes (plus separately reviewed recovery segments and human corrections) pair the pre-action Canvas, task instruction, control context, and skill guidance with the executed tool call, with no teacher reasoning or privileged simulator state. The model is trained by maximum likelihood on the policy, updating only the main policy while the workspace and sub-agents stay unchanged.

Evaluation. Six LIBERO-Pro splits (Object, Goal, Spatial, each under position swaps and task perturbations), ten tasks per split, 60 episodes per split (six per task); Show-Harness rerun for 30 episodes per split. Each episode starts in a fresh process and is limited to 50 main-agent turns, 50 physical operations, and one hour, with at most six turns per Imagination Agent call.

Why This Matters

The paper's core claim is that the generality of a VLM can be preserved for manipulation by changing the interface rather than fine-tuning the model for action prediction — which prior work suggests can weaken general understanding and reasoning. That reframes the design space: instead of asking whether a VLM can output precise metric poses, WAA asks whether a harness can make precise poses checkable before they happen. It also shows the same workspace can be used both to grow reusable skills and to produce training data for smaller, cheaper pilots.

Real-world applications:

• Household and service robotics, where a robot must align objects to centimeter-scale relations (placing a bowl on a plate) and recover from a failed grasp without task-specific training.
• Low-cost robotic deployments, where a small fine-tuned VLM (Qwen3.5-9B) pilots the harness rather than a large proprietary model being called tens of times per episode.
• Learning from human demonstration and correction, using the Canvas-GUI, which lets people operate the robot with the same tools the agent uses so that corrections are recorded in the same format as agent traces.
• Cross-embodiment and cross-simulation transfer, since scene point clouds can come from simulation, fused RGB-D, or VGGT reconstruction, and skills learned in LIBERO remain effective in robosuite without further learning.

Industry relevance. The cost comparison matters for deployment economics: $0.1996 versus $0.5021 in API cost, and 150.47 versus 874.12 seconds per episode. Avoiding a VLA executor or generated policy code, and using numerical planners and controllers to realize decisions, gives a simpler control stack. The paper also notes the Flash-class backbone costs less than methods driven by frontier proprietary models.

Future Directions

• Overcoming the backbone's multi-view limits. The paper states Gemini 3.7 Flash still integrates multi-view information imperfectly on some tasks, where WAA is less stable, and that this limitation lies in the backbone's perception rather than the harness interface — so WAA should benefit directly from VLMs with stronger multi-view understanding.

• Reducing cost and latency. An episode still requires tens of model calls; training smaller VLMs to pilot the harness is positioned as the promising route to bring both down.

• Improving efficiency of learned skills. In the stove-activation case, two runs take 23 and 37 decision steps, which the paper notes shows contact adjustment and recovery can still be inefficient.

• Scaling skill evolution and its review process. The pipeline is demonstrated with one expert trajectory per LIBERO-90 source task and one human-teaching route; how far the evidence-based Editor–Reviewer loop scales across many tasks and how much human teaching is needed remain open.

Target Audience

Robotics and embodied-AI researchers working on VLM-driven manipulation, agentic robot control, and skill libraries; engineers building practical robot systems who need to weigh API cost and latency against success rate; and students or practitioners with a background in robot learning who want a concrete example of using an interface, rather than retraining, to make a general-purpose VLM a competent robot pilot.

Authors’ abstract

General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.