CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents

AI summary

Overview

Research area: Robotics (cs.RO) intersecting with language-model post-training — specifically code-as-policy (CaP) agents that control robots by writing Python against perception and motion APIs. Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, multi-turn reinforcement learning, generalized advantage estimation, and actor–critic methods. Scope: An open, end-to-end SFT-then-RL pipeline that turns a small open vision-language model (Qwen3.5-4B) into a standalone robot-manipulation agent operating over 22 low-level primitives, with no VLA, no frontier model, and no privileged state at test time.

What This Paper Is About

Language-model agents for coding and tool use are routinely trained with a simple recipe: supervised fine-tuning to learn the interface, then multi-turn reinforcement learning to master it. Code-as-policy robotics should fit that recipe — the policy writes code against robot APIs and reads back structured results. But existing CaP agents skip training entirely: they either prompt closed frontier models or hand the actual manipulation to frozen vision-language-action models and high-level "grasp"/"place" skills, while small open VLMs fail outright. This paper asks whether a small open model can be post-trained into a capable CaP agent using the same SFT-and-RL recipe.

Key Contributions

1. CaPtain, an open end-to-end SFT-and-RL pipeline that turns a small open VLM into a standalone CaP agent, with the SFT data, training code, and CaPtain-4B weights released.
2. CaPtain-Harness, a portable execution harness exposing only 22 low-level perception, geometric-measurement, and guarded-motion primitives — no grasp/place/open-drawer skills and no VLA — with grounded text receipts returned after every robot call, and an unchanged policy-facing API across LIBERO, RoboTwin 2.0, RoboDojo, SimplerEnv, CaP-Bench, and a physical robot.
3. A simulator critic for multi-turn VLM agents that converts privileged physical-state transitions into dense per-request credit, with analyses showing credit concentrates on physical manipulation decisions rather than perception or verification calls.
4. A comprehensive evaluation showing RL gains persist across three perturbation suites, that a 4B policy surpasses several frontier-model agents at a small fraction of their cost, and that it transfers zero-shot across benchmarks and embodiments and to a physical robot without real-world data.

Main Findings

• Zero-shot to competent: The base Qwen3.5-4B solves 0% of held-out LIBERO episodes (at most 0.3% on any suite), exhausting its request budget by misusing the interface. SFT raises this to 55.4% host success and RL to 73.8%.
• RL gains are large and consistent: On LIBERO, RL adds +18.3 host points (55.4 → 73.8; 55 episodes gained, 11 lost) and +15.4 strict points (52.1 → 67.5). Mean requests per episode fall from 39.1 to 32.6, and false finishes drop from 7.1% to 2.1%.
• Gains survive perturbation: +19.0 on LIBERO-PRO across the three short-horizon suites (59.5 → 78.5), +19.4 across all four suites (49.2 → 68.6), +17.6 on LIBERO-Plus (48.6 → 66.1), +21.2 on LIBERO-MAX Dynamic (40.0 → 61.2) and +18.8 on LIBERO-MAX Base (51.2 → 70.0). All seven LIBERO-Plus perturbation dimensions improve at p < 0.05, and six of seven survive a Bonferroni correction.
• Per-suite variation: Largest gains on the goal suite (+26.7 host, p = 1.4×10⁻⁴) and spatial suite (+18.3); smallest on the long-horizon suite (+13.3). On LIBERO-Plus, lighting gains most (+23), camera/object layout/robot initial state +18 each, sensor noise least (+13).
• Beats frontier-model agents on LIBERO-PRO: 78.5% versus K1 (77.2%), WAA (75.6%), and ASPIRE (71.7%). On held-out initial state 11 specifically, 74.6% (SFT: 57.8%).
• Low-level RL beats skill-level RL in gain size: +19.4 points over its SFT model on the four LIBERO-PRO suites, versus +5.9 points for AgenticRoboRL (65.1 → 71.0) over a frozen VLA — though the authors note the two start from different baselines.
• Zero-shot SimplerEnv WidowX: 83.3% (spoon on towel 75.0, carrot on plate 79.2, block stacking 91.7, eggplant in basket 87.5), close to ManiAgent with GPT-5 (86.8) and above PhysBrain 1.0 (80.2) and π0.5 (57.1).
• Zero-shot CaP-Bench: 75.0, above Gemini-3-Pro (73.5), GPT-5.2 (55.5), Claude Opus 4.5 (54.0), and the in-domain CaP-RL 7B coder (62.0). At the full episode horizon it completes 20/20 Cube Lift and 19/20 Cube Stack (97.5% mean), indicating the 75.0 reflects execution speed against CaP-Gym's step cutoff rather than task failure.
• Weak spots: RoboTwin 2.0 reaches 22.6% (30.0% on the single-arm subset) versus 46.0% for π0.5 and 48.7% for OpenWAM-α; RoboDojo reaches Score 1.7 / 0.8% success (2.0 / 1.0% on 32 single-arm tasks) versus GPT-6 Astra's 29.0 / 22.5. The authors attribute this to required dual-arm coordination, unfamiliar arm kinematics, and a ~0.5 m top-down workspace limit of the 6-DoF X5 arm.
• Cheap: 32.6 requests (median 26), 16.6 physical actions, and 7.3k output tokens per LIBERO episode; 417 s wall time on a GPU shared by eight episodes, about 52 GPU-seconds of policy compute, or an estimated $0.03–0.06 per episode at $2–4 per GPU-hour, excluding perception services. Published code agents report $0.18–0.24 (SPARK, WAA, RACaP) up to $8–22 (OpenRUA) per episode.
• Runs on real hardware: After simulation-only adaptation in a coarse simulated twin, CaPtain-4B completes tasks on a physical robot. Success rates for the real-robot experiments are not reported in the available text.

Methodology in Plain English

The robot never gets a "grasp" or "place" button. Instead, the model gets a sandboxed Python session pre-loaded with a robot module containing 22 tools: cameras and segmentation, geometric measurement, and motion commands that take an explicit end-effector pose or gripper command. Each episode is a ReAct-style loop of at most 60 requests; in each request the model thinks, writes code, and the harness executes it and returns text "receipts" (stop reason, distance travelled, peak force, jaw opening) plus images. Variables persist across requests, so the model can save detections and helper functions. The harness also manages context — truncating receipts to at most 20 lines per cell, keeping only the 12 most recent images, and compacting old turns — identically during training and testing.

Stage 1 (SFT): Trajectories are generated in the harness itself using GPT-6 Astra and rendered in exactly the format the policy will see. Only agent-produced tokens are supervised; turns with repeated failures or recovery loops are masked out (weight 0), and visual question-answering examples get weight 0.5. Because episodes share layouts, earlier coordinates are replaced with a mask token with probability 0.2 to stop the model copying pixel values instead of looking. The vision tower is frozen and all language-model weights are fine-tuned.

Stage 2 (multi-turn RL): The hard part is credit assignment — an episode spans tens of requests whose physical effects surface turns later, with one success signal at the end. Since training happens in simulation, the authors use a simulator critic: before each request the simulator records a hidden snapshot, from which 36 features are computed in five groups (end-effector/jaw state, goal bookkeeping, distances, target lift and displacement, articulated joints and held flags). Deliberately excluded are counters of the policy's own behavior, which otherwise absorb credit from grasping. The critic is an ensemble of five small MLPs predicting host success, warm-started on earlier-policy episodes and co-trained online. Each request is credited by how much it changes the predicted success probability at a fixed time index — so requests that only perceive or verify get near-zero credit — combined with GAE (λ = 0.9) and a group-relative outcome term over G = 4 rollouts per start state. Terminal returns are graded (1 for success with finish(), 0.25 for success without, 0 for other failure, −0.1 for budget exhaustion, −0.3 for a false finish). Training uses 64 start states per round, up to 8 sampled requests per episode for the policy gradient, gradient accumulation over 1,024 episodes, an optimizer step every 4 rounds, and the round-300 checkpoint (75 optimizer updates) as the final model. The critic is discarded at test time.

Why This Matters

The paper shows that robot manipulation over a low-level execution harness can be post-trained like any other language agent — no VLA, no frontier API, no privileged state at deployment — and that the resulting small model is cheaper by orders of magnitude than frontier-API code agents while outperforming several of them on a standard perturbation benchmark.

Real-world applications:

• Onboard robot deployment: a 4B policy that runs on local compute rather than calling a paid API every episode, relevant for cost-sensitive or connectivity-limited settings.
• New robots and tasks without retraining from scratch: the policy-facing API is unchanged across five simulators and one physical robot, so a new backend adapter is the main integration cost.
• Warehouse and lab automation: pick-and-place and articulated-object tasks specified as natural-language instructions plus a TASK.md file, where the agent must reason about approach direction and grasp point rather than call a canned skill.
• Robustness-critical settings: the perturbation results suggest code-writing policies degrade less than end-to-end VLAs when object positions, lighting, cameras, or instructions shift.

Industry relevance: the cost comparison ($0.03–0.06 vs. $0.18–0.24 up to $8–22 per episode) and the release of code, data, and weights make this a practical alternative to VLA-centric stacks, especially where demonstration data is scarce. The asymmetric critic also offers a reusable recipe for anyone training language-model agents on long-horizon physical tasks in simulation.

Future Directions

• Dual-arm coordination and new embodiments: RoboTwin 2.0 (22.6%) and RoboDojo (0.8%) remain weak; the authors point to required two-arm coordination and the ~0.5 m reach of the 6-DoF X5 arm as bottlenecks, leaving cross-embodiment transfer an open problem.
• Real-world data and evaluation: the physical-robot results follow adaptation in a coarse simulated twin alone; scaling to real-world data and quantifying real-robot success are natural next steps (the paper's real-robot success rates are not reported in the available text).
• Critic portability: the simulator critic depends on privileged physical state, so extending the credit mechanism to settings without full simulator access — or learning it from cheaper proxies — is unresolved.
• Scaling and data efficiency: the SFT trajectory dataset size and the effect of model scale beyond 4B are not reported here, leaving open how far the recipe extends and how much synthetic data it needs.

Target Audience

Researchers and engineers working on embodied language-model agents, robot learning, and LLM post-training (SFT plus multi-turn RL). It is most useful to those building code-as-policy systems, evaluating manipulation robustness under perturbation, or looking for a cheaper alternative to VLA-based or frontier-API robot stacks. Readers without background in RL or robot manipulation will find the systems design accessible but the training details demanding.

Authors’ abstract

Across coding, web, and tool-use tasks, language-model agents are now routinely trained with the same recipe: supervised fine-tuning (SFT) to learn the interface, then multi-turn reinforcement learning (RL) to master it. Code-as-policy (CaP) casts robot manipulation in the same form, as the policy writes code against robot APIs and reads back the results, so the same recipe should apply. Yet existing CaP agents sidestep training: they either prompt closed frontier models or delegate manipulation to pre-trained vision-language-action models (VLAs) and high-level skills, while small open vision-language models (VLMs) fail outright. We present CaPtain, an open post-training pipeline that turns a small open VLM into a capable CaP agent, with no VLA, frontier model, or privileged state at test time. CaPtain trains and deploys the policy inside CaPtain-Harness, which exposes low-level perception, geometry, and guarded-motion primitives across simulation and real hardware. We first align the base model via SFT on synthetic trajectories generated with GPT-6 Astra, then apply multi-turn RL. Multi-turn RL must credit tens of agent requests, whose physical effects often surface turns later, from a single end-of-episode outcome. We therefore introduce a physical critic that reads privileged simulator state, never seen by the policy, and credits each request by how much it changes the predicted chance of success. Starting from Qwen3.5-4B (0% zero-shot), CaPtain-4B reaches 55.4% held-out LIBERO success after SFT and 73.8% after RL. On LIBERO-PRO, it reaches 78.5%, surpassing several frontier-model agents, and its RL gains persist across three perturbation suites (+17.6 to +21.2 points). Without target-domain training, it attains 83.3% on SimplerEnv WidowX, exceeds frontier models on CaP-Bench, and runs on a real robot after simulation-only adaptation. We release all code, data, and model weights.