AgenticRoboRL: Bringing Coding-Agent Post-Training to Agentic Robotics

AI summary

Overview

• Research area: Robotics, specifically agentic robot systems where a language-model "planner" drives a robot through a harness that executes VLA skills or generated control code; closely tied to LLM post-training (SFT + reinforcement learning) as used for coding agents.
• Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, multi-turn RL, PPO-style clipped objectives, and vision-language-action (VLA) policies, though the core idea is stated plainly.
• Scope (one sentence): The paper introduces AgenticRoboRL, a framework that post-trains the planner of a robot agent (not the robot's controllers) using supervised fine-tuning on teacher trajectories followed by multi-turn RL, and evaluates it on LIBERO-PRO and LIBERO-Plus with two different harnesses (RPent, which invokes VLA skills, and CaP-X, which generates control code).

What This Paper Is About

Agentic robotics systems pair a language-model planner with a "robot harness" that carries out the planner's commands over multiple turns, and recent work has improved these systems mainly by building better harnesses and external memory while relying on powerful closed-source planners such as Codex and Claude Code. That leaves the planner's own capabilities unchanged. The authors ask whether the post-training recipe that improved coding agents—supervised fine-tuning plus multi-turn reinforcement learning—can also improve robot planners, and they build a pipeline that works across harnesses with very different execution mechanisms (calling learned VLA skills vs. writing executable control code).

Key Contributions

1. A general post-training framework for robot planners. AgenticRoboRL is presented as, to the authors' knowledge, the first general post-training framework for agentic robotics systems. Only the planner is optimized; perception modules, motion planners, VLA executors, tools, prompts, and control loops stay fixed.
2. A harness-agnostic training interface built at the model-call boundary. Instead of requiring a shared low-level action representation, a recording proxy captures each planner request (context, generated token IDs, rollout log-probabilities) and a lightweight, parameter-free harness-specific adapter reports five fields per episode (task, initial-state identifier, attempt index, reward, validity flag) so recorded calls can be matched to outcomes.
3. A two-stage recipe with memory-aware supervision. SFT on successful teacher trajectories initializes the planner, then multi-turn RL with terminal task success as the only reward refines it, using a leave-one-out advantage within groups of attempts on the same task and initial state. For RPent, teacher responses that depend on memory the student lacks are rewritten and judge-filtered rather than simply discarded.
4. Strong empirical results and an open release. A 9B planner improves from 18.1% to 71.0% on LIBERO-PRO without external memory (versus 72.1% for Codex (GPT-5.5) with memory), reaches 72.2% on LIBERO-Plus without extra training, and improves from 2.7% to 28.5% under the CaP-X harness. The authors state they will release the framework and trained planner checkpoints (the conclusion also mentions teacher trajectories).

Main Findings

• SFT does most of the work, RL adds a meaningful margin (LIBERO-PRO). With the RPent harness, SFT raises the 9B planner from 18.1% to 65.1%, and RL further raises it to 71.0%—an additional 5.9 percentage points—without external memory.
• The post-trained 9B planner beats larger and memory-equipped references. It outperforms the original 27B planner without memory (57.9%) and comes within 3.0 percentage points of the 27B teacher with memory (74.0%). It also beats the standalone VLA policy (π_RLinf, 50.0% overall) in seven of eight LIBERO-PRO suites.
• The RL gain is statistically supported. On the 800 paired episodes, the RL planner solves 121 episodes the SFT planner fails and fails 74 that SFT solves (exact sign test, p = 0.001).
• Gains transfer to unseen perturbations (LIBERO-Plus). Without any LIBERO-Plus training, the same checkpoints go from 19.2% (original 9B) to 66.7% (SFT) to 72.2% (SFT + RL) across the 10,030 evaluation instances and seven perturbation categories. RL adds between 2.2 percentage points (object layout) and 7.0 points (lighting); on matched instances RL wins 1,547 where SFT loses and loses 999 where SFT wins, a net gain of 548 episodes.
• Best among methods not trained on LIBERO-Plus. The RL planner has the highest overall success in that comparison group, with its largest margin on robot initial-state perturbations: 75.2% versus 46.2% for the best compared baseline.
• The recipe generalizes to control-code generation (CaP-X). SFT lifts the 9B planner from 2.7% to 18.7% and RL to 28.5%, exceeding the 27B teacher's 20.3%. Gains concentrate in the two Object suites, where RL raises success from 39.0–49.0% to 67.0–72.0%, while Goal and Spatial remain low.
• Teacher data coverage limits SFT in CaP-X. Teacher data covers only 26 of the 60 tasks (19 of 20 Object, five of 20 Goal, two of 20 Spatial) and yields 979 episodes for SFT; RL needs no teacher data and still pushes the student above the teacher.
• Supervision recipe matters. On LIBERO-PRO, mask-only supervision (27,110 target turns, 424 optimizer steps) reaches 55.2%, while rewrite-and-filter (40,047 target turns) reaches 60.2% at step 438 and 64.6% after one epoch at 626 steps; the main result uses the step-563 checkpoint at 65.1%. The authors caution this comparison does not isolate the effect of rewriting alone, since the recipes differ in both response content and supervision coverage.

Methodology in Plain English

The researchers treat the robot's language-model planner the way coding-agent researchers treat a code model: as the thing to be trained, while the rest of the system is left alone.

• Record at the model call, not the robot action. Because one harness calls VLA skills and another writes Python control code, the two have no common low-level action format. Instead, a proxy sits at the model-serving boundary and saves exactly what the planner saw and generated (including images and tool specifications, where applicable), plus the log-probabilities from rollout. The trainer re-renders the prompt and compares it against the recorded tokens; mismatched samples get zero weight. This matters because a harness may prune images or drop earlier reasoning between turns.
• Translate outcomes with a tiny adapter. For each episode, a harness-specific adapter reports the task, initial-state identifier, attempt index, reward, and a validity flag. Task plus initial state defines a comparison group; the validity flag separates genuine task failures (kept, reward zero) from infrastructure failures (excluded). The adapter has no learnable parameters and does not change the harness.
• Stage 1: learn from a stronger teacher. Successful trajectories from a 27B teacher are turned into training examples, pairing the teacher's response with the context the 9B student would actually see. For RPent, the teacher had memory the student will not have, so memory-dependent turns are either dropped or rewritten (removing memory calls and results, rewriting reasoning, keeping tool arguments) and only judge-approved rewrites are used as targets. For CaP-X, teacher and student share a configuration, so prompts are simply re-rendered in the student's template and the teacher's tokens are used directly. Loss is computed only on response tokens; context tokens are masked.
• Stage 2: reinforce from task success alone. The planner runs full multi-turn episodes with fixed weights, six attempts per task/initial-state group (64 groups, 384 episodes per round). The reward is 1 for task success and 0 otherwise—no intermediate rewards, no length penalty, no learned critic. Each attempt's advantage is its reward minus the average of the other attempts in its group, so groups where everything succeeds or everything fails contribute nothing. At most 16 responses per episode are selected, and updates use a dual-clipped PPO token loss with truncated importance sampling to correct for mismatch between the rollout and training backends.
• Train and evaluate on disjoint initial states. Each LIBERO task has 50 fixed initial states; the agent explores on state 0 to build memory, is evaluated on states 1–10, teacher demonstrations and RL rollouts come from states 11–44, and states 45–49 serve only as validation for checkpoint selection. Hyperparameters: SFT at learning rate 10⁻⁵ with a batch of 64 target turns for one epoch (626 optimizer steps with RPent, 37 with CaP-X); RL at 10⁻⁶ with a batch of 128 selected responses; the reported RPent RL checkpoint follows round 6 of nine rounds, after 2,304 episodes, 26,052 selected responses, and 206 optimizer steps. Episodes end after at most 100 planner turns or 7,200 seconds in RPent.

Why This Matters

• It reframes where improvement comes from. Instead of only engineering better harnesses, memory stores, or perception around a fixed closed-source planner, this work shows that training the planner itself is a viable and complementary lever—one that can let a 9B open model approach a frontier closed model on a hard manipulation benchmark.
• It solves a real integration problem. The model-call recording plus adapter design means a single RL pipeline can serve harnesses with incompatible execution semantics, which is a prerequisite for anyone wanting to train planners across robot stacks.
• Real-world applications (illustrative):
  • Warehouse pick-and-place and bin manipulation, where a planner must retry, recover from failed grasps, and adapt when object positions change (the position-swap and task-perturbation settings tested here).
  • Household or service robots that must cope with changed lighting, camera viewpoints, backgrounds, and sensor noise—the LIBERO-Plus perturbation categories.
  • Flexible automation cells where engineers write control code over perception APIs rather than relying on pretrained skills (the CaP-X setting).
  • Lower-cost deployment: substituting a post-trained 9B planner for a much larger model or for paid closed-source API calls, while keeping existing controllers untouched.
• Industry relevance. The approach requires no retraining of VLA policies or motion controllers, so organizations with existing robot stacks can treat planner post-training as an add-on pipeline; it also suggests a data flywheel where a strong model's successful episodes become supervision for a smaller deployable planner.

Future Directions

• Combine post-training with external memory. The student is trained on a memory-using teacher but evaluated without memory; whether a post-trained planner plus memory exceeds either alone is not reported.
• Fix the weak spots in code-generating harnesses. Under CaP-X, Goal and Spatial suites stay near the floor (10.0%, 16.0%, 6.0%, 0.0% across suites for the RL planner), attributed to sparse teacher success; better data collection or exploration for rarely solved tasks is the obvious next step.
• Scale and cross-harness transfer. Only a 9B student and a 27B teacher are studied, and the RPent-trained checkpoints are explicitly not transferred to CaP-X; whether one planner can be trained jointly across harnesses, or whether larger planners gain more, remains open.
• Validate outside simulation. The paper reports only simulated benchmarks (LIBERO-PRO, LIBERO-Plus); real-robot experiments and deployment cost/latency measurements are not reported and would be needed before practical claims.

Target Audience

Researchers and engineers working on embodied AI, robot learning, and LLM agent post-training—particularly those building planner-plus-harness manipulation systems, those interested in multi-turn RL with verifiable task-success rewards, and practitioners who want to improve robot behavior without retraining VLA policies or control stacks. Readers need comfort with RL terminology (advantages, PPO clipping, importance sampling) to follow the method section, though the high-level argument and benchmark tables are accessible to a broader robotics audience.

Authors’ abstract

Agentic robotics systems combine planners with robot harnesses to perform complex manipulation tasks. Recent work improves these systems through better harnesses and external memory, but relies on powerful closed-source models as planners and does not improve the planners' own capabilities. In contrast, coding agents have improved substantially by post-training the model itself. We therefore investigate whether post-training can also improve robot planners. However, applying it across robot harnesses requires accommodating different execution mechanisms, including VLA skill invocation and control-code generation. To this end, we introduce AgenticRoboRL, a framework for post-training robot planners across different harnesses. The training recipe combines supervised fine-tuning with multi-turn reinforcement learning. To accommodate different execution mechanisms, harness-specific adapters connect recorded planner interactions with task outcomes, enabling a common RL training pipeline while keeping execution components fixed. On LIBERO-PRO, AgenticRoboRL raises the success rate of a 9B-parameter planner from 18.1% to 71.0% without external memory, approaching the 72.1% achieved by Codex (GPT-5.5) with external memory. These results demonstrate post-training as an effective approach for improving robot planners. We will release the framework and trained planner checkpoints.