OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies

AI summary

Overview

• Research area: Robotics — robot-use agents, LLM coding agents, zero-shot visuomotor policies, and the ROS 2 robot software stack.
• Technical level: Advanced.
• Scope: The paper proposes OpenRUA, a minimalist "workspace-as-harness" design that gives off-the-shelf coding agents nothing but terminal access to a robot's native ROS 2 interface, and evaluates whether a coding agent can serve as a zero-shot visuomotor policy on CaP-Bench, LIBERO-PRO, and RoboCasa365 in simulation.

What This Paper Is About

Most prior robot-use agent systems spend substantial engineering effort building custom tool interfaces, primitive libraries, and prescribed agentic workflows so that an LLM can operate a robot. The authors ask whether that harness engineering is actually necessary. OpenRUA instead hands a coding agent a workspace containing ROS 2 documentation and a few basic starter tools, plus terminal access to the robot's native ROS 2 interface, and lets the agent organize its own perception and control work as if the robot were an ordinary software project.

Key Contributions

1. A zero-abstraction, workspace-as-harness design. OpenRUA bypasses bespoke abstraction layers and provides an off-the-shelf coding agent with only terminal access to the robot's native ROS 2 interface, plus a workspace holding ROS 2 documentation and starter tools. No prescribed agentic workflow is orchestrated; the agent decides when to observe, what to write, and how to verify completion.
2. A reframing of robot use as software work. Perception is recast as file I/O (acquiring data from native ROS 2 sensor topics and saving it as workspace files the agent can inspect and process), and manipulation is recast as CLI coding (writing and executing ros2cli commands or rclpy programs that send control requests to the robot's controllers).
3. Empirical evidence that a coding agent is a zero-shot visuomotor policy. With Claude Code powered by Claude Opus 5, OpenRUA reaches 99.0% on CaP-Bench and 87.0% on LIBERO-PRO, exceeding ASPIRE by 18.0 percentage points and Harness VLA by 4.6 percentage points, without robot-policy training, prior task experience, or access to the benchmark's success predicate during execution.
4. Documentation of emergent agent behaviors and generalization under distribution shift. The authors quantify how often the agent writes its own image-processing, measurement, motion-control, and feedback-control programs, and show on RoboCasa365 that OpenRUA achieves 28.1% on unseen composite tasks, exceeding Harness VLA by 13.1 percentage points.

Main Findings

• Coding agents can act as zero-shot visuomotor policies. Claude Code with Claude Opus 5 (high reasoning effort) achieves 99.0% on CaP-Bench and 87.0% on LIBERO-PRO. On CaP-Bench, per-task success is 100.0% for Lift, Stack, Wipe, Nut, and Restack, 99.0% for 2A-Lift, and 94.0% for Handover. On LIBERO-PRO, the 6-cell average is 91.8% and the 8-cell average is 87.0%.
• The Nut task exposes a specific limitation of primitive-based systems. OpenRUA reaches 100% success on Nut, compared with 0–9% for CaP-X, RHO, and ASPIRE. The authors attribute this to native ROS 2 access letting the agent incorporate force feedback directly into its insertion program, whereas accumulating skills over predefined primitives does not supply sensing capabilities absent from their interface.
• The agent spontaneously writes its own perception programs. Across 1,500 CaP-Bench and LIBERO-PRO episodes, the agent processes sensor images with self-written code in 72.67% of episodes (47.43% on 700 CaP-Bench episodes, 94.75% on 800 LIBERO-PRO episodes) and runs a self-written metric program in 96.80% of episodes (97.43% and 96.25% respectively).
• The agent spontaneously writes its own control programs. Across the same 1,500 episodes, the agent writes motion-control clients in 95.87% of episodes (97.00% on CaP-Bench, 94.88% on LIBERO-PRO) and executes self-written feedback-control programs in 50.13% (60.14% and 41.38% respectively).
• Generalization holds where learned policies degrade. On RoboCasa365, OpenRUA scores 61.1% on Atomic-Seen, 26.3% on Comp.-Seen, 28.1% on Comp.-Unseen, and 39.4% overall. RLDX-1 drops from 21.3% on seen composite tasks to 5.0% on unseen ones, and Harness VLA (which orchestrates RLDX-1) drops from 61.0% to 13.8%. GR00T N1.5 scores 41.9, 0.0, 0.2, and 15.1 in a no-training, no-experience, no-oracle setting, and 68.5, 40.6, 42.1, and 51.1 in another setting.
• More test-time computation helps, then saturates at model-dependent levels. On the LIBERO-10 suite of LIBERO-PRO, raising the agent-turn budget from 100 to 200 lifts Claude Code with Opus 5 from 50% to 70%. Codex with GPT-6 Astra reaches 69.0% at high reasoning effort versus 62.5% at medium effort. Sonnet 5 reaches 24.0% despite gaining successes up to about 400 turns, compared with 72.5% for Opus 5 and 75.0% for Fable 5.1.
• Failure analysis points to task criteria and undocumented machine behavior. Of 413 failed episodes, class A (success criterion not stated in the instruction) accounts for 79, class B (predicate contradicts the instruction) for 19, and class C (budget exhausted while handling undocumented machine behavior) for 140 — 238 of 413 failures, or 58%. Class D (object or force beyond the machine's limits) accounts for 75 failures (18%), and class E (perception, control, or waiting error of the agent) for 100 (24%), split into 46 control errors, 27 perception errors, and 27 waiting errors. In 14 drawer episodes the agent pulls the drawer 62–136 mm and stops, whereas the predicate requires more than 140 mm.

Methodology in Plain English

The authors start from the observation that frontier coding agents already operate autonomously on unfamiliar codebases: they navigate repositories, edit and run code, issue command-line commands, read terminal output, and recover from errors. Their hypothesis is that the elaborate tool interfaces in existing robot-use agent systems compensate for limited agent capability rather than reflecting a requirement of robot control, and that a capable agent needs only a workspace directory.

So they give the agent a workspace on the robot's onboard computer. The opening prompt states the task and asks the agent to survey the machine, find out what robot it is, complete the task, verify it, and finish. A README.md file points the agent to machine.yaml for the robot's configuration — sensors, control interfaces, joint limits — and to documentation. A small set of starter tools provides ready-to-run implementations of basic operations such as saving camera images and sending joint trajectories; these are supplied as source files the agent can inspect, adapt, or ignore in favor of writing its own rclpy programs.

Perception becomes file I/O: the agent subscribes to native ROS 2 sensor topics and saves visual data, proprioceptive and kinematic states, or force and effort feedback as workspace files, which it opens with its ordinary file tools. Manipulation becomes CLI coding: the agent writes and executes ros2cli commands or rclpy programs that send trajectory, velocity, or gripper control requests to the robot's controllers, then inspects terminal output and new observations to assess outcomes. The workspace also acts as persistent external memory, holding images, programs, notes, and logs in folders such as snaps/, scripts/, and work/, which the agent can revisit beyond its context window.

Evaluation runs on three simulation benchmarks: CaP-Bench (seven single-arm and bimanual manipulation tasks, 100 trials per task), LIBERO-PRO (four LIBERO suites, ten trials per task, testing robustness to position and task perturbations), and RoboCasa365 (mobile manipulation; 50 target tasks with ten trials per task, split into 18 atomic tasks, 16 seen composite tasks, and 16 unseen composite tasks). Each episode begins with a fresh workspace, no robot demonstrations, no task-specific rehearsal, no memory from earlier episodes, no privileged simulator state, and no access to the benchmark's success predicate. Agents and models evaluated include Claude Code with Claude Opus 5, Sonnet 5, and Fable 5.1 at high reasoning effort, and Codex with GPT-6 Astra at medium and high reasoning effort.

Why This Matters

Impact on research. The paper challenges a widely used design pattern in robot-use agent research: that agents need hand-crafted perception and control primitive libraries, or learned visuomotor policies wrapped as tools, to operate robots. The reported results show an off-the-shelf coding agent reaching 99.0% on CaP-Bench and 87.0% on LIBERO-PRO with no bespoke primitives, no learned policy, and no access to the success predicate, while its RoboCasa365 numbers hold steady across seen (26.3%) and unseen (28.1%) composite tasks where RLDX-1 falls to 5.0% and Harness VLA to 13.8%. The failure study also identifies a mismatch between task instructions, machine behavior, and benchmark completion criteria, with 238 of 413 failures attributed to classes A, B, and C.

Real-world applications (the paper's evaluation is simulation-only):

• Household and kitchen manipulation, where the agent handles drawers, dishwasher racks, freezer doors, baskets, and objects such as a fallen ketchup bottle — the RoboCasa365 task families the paper evaluates.
• Mobile manipulation on a base, since RoboCasa365 covers mobile manipulation and the failure analysis discusses a mobile base displaced by contact forces.
• Force-sensitive insertion and assembly, illustrated by the Nut task, where OpenRUA's ability to use force feedback from native ROS 2 interfaces yields 100% success against 0–9% for CaP-X, RHO, and ASPIRE.
• Industrial robot software integration, because ROS 2 is described as the de facto standard of the robotics community and is used in both simulation and on physical robots, so an agent that works directly through ros2cli and rclpy fits existing deployments rather than requiring a new abstraction layer.

Industry relevance. Companies building robot fleets on ROS 2 could in principle reduce the engineering cost of bespoke agent harnesses, since the agent works with documentation, a configuration file, and starter tools rather than a purpose-built tool library. The test-time computation results — success rising from 50% to 70% when the agent-turn budget goes from 100 to 200 for Opus 5, and from 62.5% to 69.0% for Codex with GPT-6 Astra when reasoning effort goes from medium to high — give a practical dial for trading interaction cost against task success, though the paper notes that performance plateaus at levels that vary across models.

Future Directions

• Physical robot experiments. The authors state that their evaluation is limited to simulation and that experiments on physical robots are needed to assess practical viability. Sensor noise, missing depth values, and calibration errors may affect both visual interpretation and programmatic measurement.
• Real-time responsiveness. Simulation pauses time during agent deliberation, so the evaluation does not capture inference latency in a physical environment that continues to evolve. Agent-written feedback controllers can react within a running program but do not eliminate delays between agent-level decisions, leaving dynamic tasks untested.
• Safety guarantees. Task success does not establish safe execution; the failure analysis includes motions that would collide with the table, and the authors note that agent-written force thresholds and stopping conditions are not guarantees.
• Closing the benchmark gap. The failure study finds that task criteria not stated in the instruction (79 failures), predicates contradicting the instruction (19), and budgets exhausted while handling undocumented machine behavior (140) together account for 58% of failures, pointing to a need for better-specified task instructions and completion criteria — and to the question of how much of the remaining gap comes from agent capability versus benchmark design.

Target Audience

Robotics researchers working on visuomotor policies, robot-use agents, and language-conditioned manipulation; LLM agent researchers interested in embodied applications and test-time scaling; and engineers building robot software on ROS 2 who want to know how much harness engineering is actually required. Readers familiar with benchmarks such as CaP-Bench, LIBERO-PRO, and RoboCasa365, and with tools like Claude Code and Codex, will get the most from the empirical comparisons. The paper is written at an advanced technical level, and its code is available at https://github.com/terminalworld/OpenRUA.

Authors’ abstract

Coding agents are extending their reach into the physical world by writing and executing robot control programs. One might expect the agents to use the existing mature software stack that engineers have developed over decades to access sensors and control motion. Yet prior work primarily engineers complex custom harnesses to orchestrate agents for robot use, particularly by prescribing specialized workflows and providing bespoke interfaces. This raises the question: "Is such additional harness engineering necessary?" We introduce OpenRUA, a zero-abstraction harness that bypasses bespoke abstraction layers by providing off-the-shelf coding agents with only terminal access to the robot's native software interface ROS 2. OpenRUA employs a minimalist workspace-as-harness design, only offering ROS 2 documentation and basic tools while leaving the coding agent to organize its own work without orchestrating any agentic workflow. Within this workspace, OpenRUA recasts perception as file I/O and manipulation as coding. With Claude Code powered by Claude Opus 5, OpenRUA achieves success rates of 99.0% on CaP-Bench and 87.0% on LIBERO-PRO, demonstrating that an off-the-shelf coding agent can serve as a zero-shot visuomotor policy through the robot's native interface, without bespoke primitives or task-specific training. Under this minimalist design, further analysis reveals striking emergent behaviors of coding agents: (1) For perception, the agent spontaneously writes programs that process raw sensory inputs and derive metric measurements in 96.80% of episodes. (2) For manipulation, the agent spontaneously builds motion-control clients (e.g., gripper control) in 95.87% of episodes and closed-loop control programs (e.g., adjusting motion based on sensor feedback) in 50.13% of episodes. Our code is available at https://github.com/terminalworld/OpenRUA.