Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens

AI summary

Overview

Research area: Robot learning and embodied AI agents — specifically vision-language model (VLM) agents that control robots by composing classical motion/perception primitives with learned vision-language-action (VLA) policies.

Technical level: Intermediate. The interface-design argument is accessible, but the evaluation involves three robot benchmarks, VLA policies, token accounting, and ablations.

Scope: This paper introduces PyRUA-Lean, an interactive Python code-execution interface for robot agents, and compares it against a tool-calling baseline (RPent) using the same GPT-6 Astra planner across 700 simulated task instances from LIBERO-PRO, RoboTwin 2.0, and RoboCasa365.

What This Paper Is About

VLM agents can drive robots by calling motion, perception, and grasping primitives, but tool-calling interfaces make dependent operations expensive: an action that depends on a previous result generally needs another model invocation, and intermediate results (including multiple camera images returned after every move) accumulate in the conversation. This paper asks whether letting the agent write and execute Python programs — composing primitives, checking outcomes, retrying locally, and requesting only the observations it needs — can both raise task success and cut token consumption. The authors build PyRUA-Lean on RPent's robot stacks and primitive implementations and test it head-to-head against RPent's tool-calling agent under equal LLM-call budgets.

Key Contributions

1. An interactive code-execution interface for robot primitive composition. PyRUA-Lean exposes the robot as a Python object (robo) and lets the agent generate code cells that chain primitives, condition later actions on earlier outcomes, define helper functions, and retain variables in a persistent namespace within an episode. Both agents run without cross-episode memory.

2. Comprehensive evaluation on three robot benchmarks. Across 700 simulated task instances (LIBERO-PRO, RoboTwin 2.0, RoboCasa365), PyRUA-Lean raises overall success from 63.1% to 71.7%, and on instances solved by both agents uses 49% fewer LLM calls and 65% fewer input tokens.

3. Selective observation and context management. Primitive results stay in the runtime; only explicitly printed outputs and camera requests (for example robo.show) are returned at cell end for replanning. A recorded placement cell replaces baseline steps 14–17, reducing four LLM turns to one and returning five printed lines and no image.

4. Analysis of where the efficiency gains come from and where they stop. Trace decompositions, ablations removing VLA policies and operating guides, and a RoboDojo pilot identify fewer LLM invocations as the main source of token savings, show the gains persist without VLA policies or guides, and show they diminish on VLA-dominated tasks such as RoboCasa365 atomic tasks.

Main Findings

• Higher success under equal LLM-call budgets. Overall success rises from 63.1% to 71.7%, an improvement of 8.6 percentage points, or approximately 14% relative. Gains by benchmark group: +11.0 points on LIBERO-PRO (83.0% to 94.0%), +9.2 on RoboTwin 2.0 (60.0% to 69.2%), +7.8 on RoboCasa365 atomic (78.9% to 86.7%), and +5.0 on RoboCasa365 composite (34.4% to 39.4%).

• More exclusive wins. PyRUA-Lean alone solves 96 instances, whereas tool calling alone solves 36.

• Fewer calls and tokens on jointly solved instances. Mean LLM calls fall from 17.0 to 8.7 and mean input tokens from 788k to 276k, corresponding to reductions of 49% and 65%. Estimated cost falls from $1.63 to $0.74 per episode; the smaller monetary reduction partly reflects caching discounts on the baseline's repeatedly processed history.

• Per-benchmark call and token ratios. LIBERO-PRO: 21.4 to 8.4 calls (2.5×) and 1.23M to 273k tokens (4.5×), cost $2.49 to $0.79 (3.1×). RoboTwin 2.0: 12.2 to 8.9 calls (1.4×) and 415k to 245k tokens (1.7×), cost $1.00 to $0.66 (1.5×). RoboCasa365 atomic: 10.4 to 5.6 calls (1.8×) and 261k to 171k tokens (1.5×), cost $0.62 to $0.54 (1.2×). RoboCasa365 composite: 28.4 to 14.9 calls (1.9×) and 1.22M to 603k tokens (2.0×), cost $2.01 to $1.25 (1.6×).

• Token-budget cutoffs on recorded trajectories. On LIBERO-PRO, PyRUA-Lean reaches the baseline's final success rate with 564k tokens per episode, compared with 3.18M for tool calling. The authors note the agents were not rerun with these token budgets specified beforehand.

• Fewer invocations, not uniformly smaller prompts, drive the savings. On LIBERO-PRO the 4.48× token ratio decomposes into 2.55× fewer calls and 1.76× fewer tokens per call. On RoboCasa365 atomic tasks, PyRUA-Lean's calls are larger on average, yet fewer invocations still reduce total token usage.

• Selective images alone are not enough. Modifying the tool-calling baseline on LIBERO-PRO to return images only on request lowers success from 83.0% to 68.5%. On the 131 instances solved by all three configurations, that modified baseline makes more calls and consumes approximately the same total input tokens as the original baseline.

• Savings persist without VLA policies or guides. Without VLA policies, PyRUA-Lean reaches 68.8% success on RoboTwin 2.0 versus 42.8% for tool calling, with traces showing classical primitives composed into approach, gripper control, and state checks. On RoboCasa365 atomic tasks without VLA policies, success is 65.6% versus 45.6%; on composite tasks, 20.0% versus 7.5%. On LIBERO-PRO without guides, success is 88.0% versus 84.0%.

• Guidance effects depend on the interface. On RoboTwin 2.0 without VLA policies, removing guides increases tool-calling success from 42.8% to 60.4%, suggesting guidance is not uniformly beneficial.

• Where the extra successes come from. Of the 96 instances solved only by PyRUA-Lean, the baseline exhausts its LLM-call budget in 52 and ends unsuccessfully before exhausting it in 44. The latter include geometric reasoning and scripted recovery, but also cases where a VLA succeeds in one run and fails in the other, so the difference cannot be attributed entirely to better recovery logic.

• Efficiency gains are smallest where a VLA does most of the work. Including failed episodes, the estimated VLM inference cost of tool calling on RoboCasa365 atomic tasks is approximately 1.1 times that of PyRUA-Lean, and PyRUA-Lean's larger API description offsets part of the benefit from fewer calls.

• RoboDojo pilot. On three RoboDojo tasks (cover blocks, press by number, stack blocks by language), PyRUA-Lean achieves 26.7% success (8/30) versus 3.4% (1/29 scored; one infrastructure failure excluded); both fail button pressing. Total input and output tokens across all attempts per success fall from 11.56M to 1.07M (90.7%), while total-token usage on 29 matched scored pairs falls by 25.3%.

• What the code cells actually do. Over all 700 episodes, 11,145 cells were counted. A code cell runs 2.1 acting primitives on average (45% run two or more), whereas an LLM call of tool calling runs 0.65; of tool-calling calls that call a tool, 89% call exactly one, and only 45 of all 18,585 run two acting tools together. Named habits include guarded chains (19% of cells, 45% on LIBERO-PRO), retries (8% of cells, 18% on RoboCasa365 atomic), NumPy-based perception or geometry (30%), defining helper functions (2%) or calling one from an earlier cell (7%), asking for a camera image (84%), looking only if the task is not done (20%), and single steps (35%, 54% on RoboTwin 2.0).

• A worked paired episode. In the median-cost episode for both agents (LIBERO-PRO, spatial swap, task 7, seed 2), tool calling solves the task at call 17 after 33 images in all and 816k tokens, while the code agent solves it at call 6 with 5 images and 156k tokens.

Method in Plain English

The researchers took one existing robot agent framework (RPent, part of Harness VLA) and changed only how the planner talks to the robot. In the baseline, the planner issues tool calls: state and success flags come with every move, three camera images come back after every move, and any step that depends on a previous step's output needs a new model turn. In PyRUA-Lean, the planner instead writes Python cells against a robo object whose methods are the same primitives with the same names and arguments. A cell can chain several primitives, check whether each one reached its target, retry a VLA policy, compute geometry with NumPy from point clouds, and print or request only what it wants to see; variables and helper functions persist across cells within an episode.

Both agents use GPT-6 Astra at high reasoning effort through the Codex CLI, the same robot stacks, the same frozen VLA policies (π0.5 on LIBERO-PRO, LingBot-VLA on RoboTwin 2.0, RLDX-1 on RoboCasa365), SAM 3 for segmentation, the same operating guides where available (RoboCasa365 has none), and no cross-episode memory. Evaluation covers 40 tasks from LIBERO-PRO's four perturbed suites, 50 dual-arm RoboTwin 2.0 tasks, and 50 RoboCasa365 Target50 tasks, each at five environment seeds, one episode per agent per task–seed pair: 700 paired task instances and 1,400 episodes. Budgets are 40 LLM calls per episode (100 for RoboCasa365 composite tasks) and two hours; an uncaught exception ends a cell and returns its traceback, an external watchdog enforces the two-hour limit, and the Codex CLI compacts the conversation only near 272K tokens while returning a tool-call timeout after 300 s without cancelling the cell. Success is measured with each benchmark's own check, and calls, cumulative input tokens (including cached tokens and post-success calls), and estimated list-price dollars are compared on instances both agents solved.

Why This Matters

The paper reframes robot agent efficiency as an interface question rather than a model question: with the same planner, primitives, and call budget, the shape of the interface determines how much work gets done per model invocation. That is a useful, testable claim for anyone building embodied agents, and it comes with an honest account of where it breaks down — VLA-dominated atomic tasks, ambiguous guidance effects, and successes that are not all attributable to better recovery logic.

Real-world applications:

• Warehouse and logistics picking, where per-episode inference cost and latency scale with the number of robot actions, and where local retries and geometric checks can recover from near-misses without a new model turn.
• Household and kitchen manipulation, the setting of RoboCasa365 and LIBERO-PRO, where composite multi-step tasks (washing lettuce, loading a dishwasher, stacking bowls) benefit most from composing many primitives in one program.
• Bimanual assembly and handover, exercised by RoboTwin 2.0 tasks such as placing containers, opening laptops, and rotating QR codes, where guarded chains and state checks reduce repeat motion.
• Fleet deployment under API cost or rate limits, where the same call budget yields more completed tasks — relevant for long-running robots that must operate within fixed inference spend.

Industry relevance: the measured cost per solved episode ($2.49 to $0.79 on LIBERO-PRO; $1.63 to $0.74 pooled as means per solved episode) is the kind of number that decides whether a frontier-model planner is economically viable for robot fleets. The code-execution pattern also aligns with how general-purpose coding agents already work, which lowers the barrier to adopting it.

Future Directions

• Generalization beyond the tested setup. The authors state the evaluation is limited to GPT-6 Astra, RPent's robot stacks and primitive libraries, and simulation, so other planners, other primitive configurations, and physical-robot performance remain untested.
• Run-to-run variability. The study uses one run per agent per task instance, and some exclusive wins are cases where a VLA succeeds in one run and fails in the other; repeated runs would clarify how much of the success gap is stable.
• Isolating interface components. The comparison evaluates the complete interfaces, so the separate effects of primitive composition, persistent state, and selective feedback are not fully isolated. The token decomposition is described as an accounting analysis that does not independently isolate causal contributions.
• Measurement and reuse. The authors propose measuring execution latency and recovery on physical robots, and examining how primitive granularity and cross-episode reuse of validated routines affect success, inference cost, and adaptability.

Target Audience

Researchers and engineers working on embodied agents, VLM-based robot control, and agent interface design; practitioners deploying LLM planners on robot fleets who care about inference cost per solved task; and readers of code-as-policies or CodeAct-style work who want a controlled comparison of code execution versus tool calling on identical robot primitives. Readers need basic familiarity with robot manipulation primitives and LLM token accounting to follow the tables.

Authors’ abstract

Vision language model (VLM) agents can control robots through visual feedback and action primitives, but repeated model invocations and redundant observations incur substantial token overhead. We introduce PyRUA-Lean, an interactive code-execution framework that couples feedback-driven primitive composition with selective observation: the agent composes classical robot primitives and learned vision-language-action (VLA) policies into Python cells that perform conditional checks and local retries, returning only explicitly requested images and state feedback for replanning. Across 700 simulated task instances from LIBERO-PRO, RoboTwin 2.0, and RoboCasa365, we compare PyRUA-Lean with a tool-calling baseline using the same GPT-6 Astra planner and underlying robot primitives. Under equal LLM-call budgets, PyRUA-Lean increases overall success from 63.1% to 71.7%. On instances solved by both agents, it uses 49% fewer LLM calls and 65% fewer input tokens.