Agent as Policy for Robotic Manipulation

AI summary

Overview

Research area: Robot manipulation with foundation multimodal large language models (MLLMs); agentic robot systems, program synthesis for control, and vision-language-action policy learning. The paper is posted on arXiv under the Natural Language Processing category (cs.CL).

Technical level: Intermediate. The core idea is conceptually simple, but following the interface design, budgets, and metric accounting requires some familiarity with robot control interfaces and LLM agent workflows.

Scope: The paper introduces AGP (Agent as Policy), a system that uses a general-purpose coding agent as the robot policy itself, and evaluates it on five real-world manipulation task groups (assembly, block construction, dice flipping, targeted throwing, and bimanual towel folding) with success, time, token, and cost measurements.

What This Paper Is About

Robot manipulation systems typically either generate executable programs in advance or orchestrate a library of separately trained learned policies. Both approaches limit how the robot can reinterpret evidence or generate motion when something unexpected happens. This paper asks whether a general-purpose agent, with no task-specific or environment-specific training and with fixed model weights, can serve directly as the robot policy: reading a task specification, writing programs at runtime, issuing motion commands, and revising its actions based on physical feedback.

Key Contributions

1. AGP: the agent itself as the robot policy. Instead of pre-written programs or policy orchestration, a general-purpose coding agent decides at runtime what to observe, how to interpret observations, which motion to request, and how to recover, with model parameters held fixed throughout execution.

2. An agent-robot bridge. A documented robot interface exposes calibrated camera observations, proprioceptive state, geometric queries, and seven commands (state, status, help, frames, move_ee, move_joints, gripper), with motion validation, inverse kinematics, trajectory generation, and measured state returned after each motion.

3. A real-robot evaluation across five task groups. Zero-shot trials on assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding, spanning long horizon reasoning, action diversity, and object diversity under video, image, and language instructions.

4. A study of experience accumulation and strong-to-weak transfer. The agent saves procedures, measurements, corrections, and scripts to files for reuse; experience collected by a stronger model (GPT-6 Astra) is then transferred to a weaker model (GPT-5.6 Terra).

Main Findings

• Strong zero-shot success. AGP succeeds in at least 80% of trials in seven of eight task configurations, with 100% observed success in five. Specifically: 8/10 on four pair assembly, 29 of 30 trials across the three block construction configurations (pyramid 10/10, two towers 10/10, six block tower 9/10), 10/10 on dice flipping, 2/2 on targeted throwing to a potato, 5/5 on sequential towel folding, and 3/5 on simultaneous towel folding.

• Execution overhead grows with task complexity. Time and cost are high in absolute terms: four pair assembly averages 37.2 minutes, 13.02 million tokens, and USD 16.62 over successful trials; dice flipping averages 37.9 minutes, 17.08 million tokens, and USD 21.07. The six block tower costs more than the pyramid and two towers (11.57 versus 9.24 and 7.94 million tokens).

• Verification itself consumes substantial time. In the potato throwing task, the throw is completed an average of 13.9 minutes after task delivery, and subsequent trajectory analysis and verification account for approximately 40% of the reported 22.8 minutes.

• Deformable object manipulation is the hardest case. Sequential towel folding succeeds in all five trials but requires 50.8 minutes and USD 24.14 on average, among the highest expenditures in the task suite. Simultaneous folding has the lowest observed success rate at 3/5, highlighting the difficulty of coordinating dynamic manipulation of deformable material.

• Model choice matters within an agent. On two pair assembly, GPT-6 Astra succeeds 5/5 at low, medium, and high thinking effort (9.9, 9.2, and 9.2 minutes respectively), suggesting limited benefit from additional thinking effort on this task. GPT-5.6 Sol succeeds 5/5 with longer time (14.2 minutes) but lower cost (USD 3.94) than Astra; GPT-5.6 Terra succeeds 1/5; GPT-5.6 Luna succeeds 0/5. Under Claude Code, Claude Opus 5 succeeds 5/5 and Claude Fable 5.1 succeeds 3/5 while using fewer tokens on successful trials yet incurring greater time and cost.

• Accumulated experience reduces execution time. Over five executions of two pair assembly, task time decreases by 29.3% from the first to the fifth execution. Response latency (reasoning plus programming) decreases by 47.0%, while tool execution increases by 9.3%. Disassembly and reassembly of a ridged ring becomes faster after the first cycle and stays near that lower time.

• Strong-to-weak transfer improves both success and efficiency. Transferring Astra's saved experience to Terra on two pair assembly raises success from 1/5 to 4/5, and among successful trials mean completion time and token usage fall by 34.3% and 11.4%. Mean inference cost also decreases, excluding Astra's experience collection.

• Cost of the approach remains a barrier. The authors state that substantial execution time and inference cost remain obstacles to practical deployment.

Methodology in Plain English

The setup has two agents. A preparation agent runs once per task type and turns the user's request into a reusable task definition describing the goal, reference materials, constraints, allowed variations, completion criteria, interface rules, budgets, and reporting requirements. An execution agent then runs each instance of the task with that definition.

At runtime the execution agent receives the task specification and a documented robot interface. It works in a persistent local workspace of scripts, notes, and observations. It can call tools to capture images, read joint positions and end effector pose, write and run Python programs to compute object geometry, and submit arm or gripper commands. After each motion, the interface returns updated state and target errors; the agent uses these plus new images to refine its estimates and decide the next move. The agent stops when it reports completion or when the task budget is exhausted. Motion execution and monitoring run independently of agent inference, so the robot can execute a submitted motion while the agent reasons.

The hardware is an I2RT YAM arm with six revolute joints and a parallel gripper (a linear_4310 gripper with a nominal maximum inner jaw opening of 95.5 mm), driven through the i2rt Python library over a 1 Mbit/s CAN connection. Cameras are a wrist-mounted Intel RealSense D405 with aligned RGB and depth at 640 × 360 pixels and a fixed overhead Logitech BRIO at 1920 × 1080 pixels, both at 30 frames per second. Cartesian targets use Mink for inverse kinematics, and ordinary motion uses joint velocity and acceleration limits of 20°/s and 40°/s², a Cartesian translation limit of 0.03 m/s, and an orientation limit of 10°/s. Safety envelopes restrict targets radially to 0.12 to 0.65 m, grasp heights to −0.050 to 0.60 m, and single Cartesian steps to 0.25 m. Bimanual towel folding uses two arms in a shared coordinate frame; targeted throwing uses a separate timed-motion runtime interpolated on a 50 Hz grid, with joint 4 limits of 180°/s and 360°/s².

Evaluation protocols use predefined initial scenes, fixed budgets for elapsed time and observation and action requests, matched scenes and counterbalanced order for model and experience comparisons, and completion criteria fixed before evaluation. Success is judged from the final physical state and execution video. Assembly parts are adapted from the AutoMate dataset, with nominal radial clearances as small as 0.73 mm, which illustrates the precision required for successful insertion.

Why This Matters

Impact on research. The paper tests a different boundary than trained visuomotor policies, vision-language-action models, or agents that sequence learned skills: it asks whether the agent's own reasoning and programming, interacting with a robot interface, can constitute the policy. If that holds, training data collection and policy training for each new task may not be required, and adaptation can happen within a session through updated measurements, programs, and decisions.

Real-world applications:

• Precision assembly. Submillimeter-to-low-millimeter clearance insertions, using human demonstration videos and timestamped textual guides, as in the AutoMate-derived assembly parts.
• Structured construction and packing. Building target structures from goal images (a pyramid, two towers, a six block tower), where the agent must infer object identities, orientations, and support relations and choose a stable placement order.
• Throwing and timed release tasks. Coordinating a circular joint swing with gripper release so an object travels through the air and lands in a target region.
• Deformable object handling. Bimanual towel folding, where the agent must reproduce demonstrated folds with either arm and keep arm clearance during staging.

Industry relevance. The work points to deployments where robots must handle repeated, varied physical tasks without per-task training. The cost figures reported here (for example, USD 24.14 average for sequential towel folding and USD 21.07 for dice flipping) and completion times (up to 50.8 minutes) indicate that economics and latency, not just success rate, determine whether this is deployable. The experience-transfer result, where a costly stronger model establishes procedures that a cheaper weaker model reuses, is directly relevant to operating budgets.

Future Directions

1. Reducing execution time and inference cost. The authors identify these as key barriers to practical deployment; cached procedures and experience files reduce them only partially.

2. Improving coordinated deformable object manipulation. Simultaneous towel folding was the weakest configuration at 3/5, and the authors flag reliability in coordinated deformable manipulation as an open challenge.

3. Broadening experience accumulation and transfer. The studies used two task pairings from the assembly suite; whether accumulated procedures transfer across different task types, object sets, and agents beyond the Astra-to-Terra pairing is not reported.

4. Generalizing the reported evaluation. Only the two pair assembly task was used for the model and thinking-effort comparison and for both experience studies; scaling these comparisons to the full task suite, and to more models and agents, remains open.

Target Audience

Researchers and engineers working on robot learning, LLM-based agents, and embodied manipulation who want to understand what a general-purpose MLLM agent can do as a robot policy without task-specific training. It is also relevant to practitioners evaluating deployment trade-offs, since the paper reports success rates alongside time, token usage, and inference cost, and to readers interested in experience accumulation and strong-to-weak knowledge transfer in agentic systems.

Authors’ abstract

We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. Across assembly, block construction, and dice flipping, AGP succeeds in at least eight of ten trials for each evaluated task configuration. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.