From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

AI summary

Overview

• Research area: Robotics — real-world reinforcement learning for long-horizon robot manipulation, specifically adapting pretrained vision-language-action (VLA) policies.
• Technical level: Advanced. The paper assumes familiarity with VLA foundation models, residual RL, TD3+BC, replay buffers, and sparse-reward credit assignment.
• One-sentence scope: The paper introduces PARTS, a framework that spends real-robot RL practice only on the few subtasks where a frozen pretrained policy fails, using agent-authored selectors, verifiers, and reset programs instead of human corrections.

What This Paper Is About

A pretrained robot policy can perform most of a long-horizon task yet fail repeatedly at a few critical subtasks, and because complete-task success requires every stage to succeed, those bottlenecks cap overall performance. Collecting more full-task demonstrations for supervised fine-tuning forces operators to re-demonstrate behaviors the policy already handles, while full-task RL with only a sparse end-of-episode reward gives almost no learning signal when early stages keep failing. PARTS addresses this by keeping the pretrained VLA frozen and training small residual policies only inside the identified bottleneck subtasks, with local rewards that make every attempt informative.

Key Contributions

1. A formulation for bottleneck-localized policy adaptation: The paper formalizes long-horizon tasks as sequences of subtasks with binary outcome criteria, where the terminal reward factorizes as the product of all subtask outcomes, and defines bottlenecks as the low-success subtasks that bound complete-task success. Learning targets these bottlenecks while evaluation is measured on complete long-horizon execution.
2. The PARTS framework: PARTS connects executable subtask supervision (policy selectors, success verifiers, and auto-reset policies authored by coding agents), residual RL with online TD3+BC, and success-reweighted retraining with redeployment. Training rollouts require no human corrections, no human switching decisions, and no human reward labels in the automated cases.
3. Success-reweighted retraining and redeployment: Each residual policy is periodically retrained on a curated dataset containing every successful episode plus a uniformly sampled fraction ρ of failures, then redeployed to collect further experience — meaning retraining changes the policy used for subsequent data collection rather than only extracting a final policy.
4. Real-world experiments on two platforms: Evaluation on two bimanual YAM tasks (earbud insertion, LEGO sorting) and one single-arm Franka FR3 task (cable unplug-and-plug) shows gains over a π0.5-SFT base policy and over existing real-world RL fine-tuning methods under a matched robot-rollout budget.

Main Findings

• Full-task success improves on both platforms: PARTS raises complete-task success from 32% to 61% on bimanual YAM tasks and from 50% to 95% on single-arm Franka tasks, using tens of minutes of real-world RL rollouts per task on average.
• Better than real-world RL fine-tuning baselines at equal rollout budget: Under the same robot-rollout budget, PARTS raises full-task success by more than 25% compared with existing real-world RL fine-tuning methods while requiring less human involvement. Baselines compared are SFT, DSRL, EXPO-FT, and RLT.
• EXPO-FT can degrade the base policy: EXPO-FT ultimately underperforms the base policy and occasionally degrades previously reliable behaviors such as cable unplugging, suggesting that backbone updates driven by sparse full-task rewards can disrupt non-bottleneck subtasks.
• DSRL and RLT do not significantly beat SFT on average: Both are described as stronger baselines, but neither significantly improves on the SFT policy on average under limited real-world rollout budgets. DSRL's exploration is confined to behaviors the frozen VLA can generate; RLT targets a single critical phase after a VLA-to-RL handoff.
• Cable insertion is the most rollout-efficient gain: Insertion success nearly doubles after only 29 min of RL rollouts, because the base policy already unplugs reliably and RL is concentrated on insertion.
• Earbud task shows the largest relative gain: On the most challenging task, requiring precise bimanual manipulation, PARTS achieves four times the base policy's full-task success rate, with per-stage success improved by 20–35 percentage points for both insertions over the base policy.
• LEGO grasp correction improves progress substantially: Correcting gripper closure alone increases LEGO progress from 54% to 82% with only 17 min of RL rollouts. The same two grasp residual policies are reused across all ten bricks.
• Success-reweighted retraining matters: Ablating retraining on the earbud task (applied to RL1 and RL2 only) reduces both subtask and full-task success. RL0 was not retrained because its base success is already high and its reset is the task's initial condition.
• Local rewards give denser signal than terminal rewards: PARTS uses local rewards over 3–15 s subtask windows with initial success rates above 25%, versus baselines that receive sparse feedback only at the end of 20–120 s episodes.
• Human involvement is reduced, not eliminated: Compared with the baselines, PARTS needs no corrective interventions or handoff decisions during RL rollouts, no success labeling on LEGO and cable, and automated resets on LEGO and cable. On earbuds, a human checks the verifier's labels, and a human resets when the robot cannot restore the scene.

Methodology in Plain English

The researchers start from a frozen pretrained VLA policy (π0.5) that has been supervised fine-tuned on a small set of task demonstrations, giving a task-specific base policy. A human watches this base policy fail on the real robot and identifies the bottleneck subtasks, writing a "contract" for each one: the entry conditions, which action coordinates may be corrected and within what bounds, the outcome criterion for success, a motion budget, and a few subtask demonstrations.

Coding agents (Claude Fable 5 and GPT-6 Astra) turn each contract into three ordinary programs that run alongside the control loop. A policy selector decides when to activate a residual policy based on images, proprioception, and motion predicates. A success verifier checks the contract's postcondition after human-defined event gates are satisfied and returns a local reward of one or zero, requesting a human label when uncertain. An auto-reset policy decides whether to retry from the current state or reset the scene, with the robot resetting when feasible and a human doing so otherwise. Perception for these programs comes from promptable segmentation with SAM3 and from a VLM (Gemini-3.7-flash) answering asynchronous queries about object states.

For each bottleneck, a lightweight residual actor-critic is trained with online TD3+BC. The frozen base policy supplies the nominal action chunk at every replan; the residual outputs a normalized correction that is masked to a few action dimensions and bounded physically, so all other dimensions follow the base policy. Untrained actors output a zero residual, and exploration adds Gaussian noise. Each bottleneck has its own actor, twin critics, and replay buffer, and rewards are the verifier's local outcome labels. Reference dropout zeroes the nominal chunk for a random subset of each batch to prevent the actor from merely copying the base policy.

Periodically, each residual policy is retrained on a curated dataset keeping all successful episodes and a sampled fraction ρ of failures, then redeployed to collect more experience. At evaluation, residuals are activated at their entries and hand control back to the base policy at their exits so the full task can continue. Evaluation is on 20 complete episodes per task from full-task initial states, with a normalized progress score also reported.

The three tasks are earbud insertion (opening a charging case, holding it in the left gripper, inserting two earbuds with the right arm, closing the case; bottlenecks RL0 case preparation, RL1 first insertion, RL2 second insertion), LEGO sorting (sorting ten bricks into three color bins within 150 s, with SFT demonstrations from the public ABC-130k dataset whose bricks are larger; bottleneck is the grasp by either arm), and cable unplug-and-plug on a single-arm Franka FR3 (unplugging from a source router and inserting into a destination router; bottleneck is alignment and insertion).

Why This Matters

Impact on research: The paper reframes real-world RL fine-tuning of robot foundation policies as failure-localized policy repair rather than global policy improvement. It shows that a frozen pretrained policy plus narrow, verifier-supervised residual corrections can beat full-task RL methods under the same robot-rollout budget, and it introduces executable contracts and coding-agent-authored supervision programs as a way to remove recurring human involvement (corrections, switching decisions, labels, resets) from the training loop.

Real-world applications:

• Precision assembly and insertion tasks, such as fitting small connectors, earbuds, or components into tight slots where millimeter-level clearance matters.
• Warehouse and logistics sorting, where a repeated pick-and-place must run reliably over many objects within a fixed time budget.
• Cable and connector handling in data center or electronics rework, where insertion requires alignment within a port tolerance followed by a decisive contact motion.
• Rapid adaptation of general-purpose robot policies to customer-specific objects that are out of distribution for the pretrained model.

Industry relevance: Because training needs only tens of minutes of real-world rollouts per task and can reuse learned corrections across repeated occurrences of the same bottleneck, the approach targets deployments where robot time and human supervision are the scarce resources. Framing bottlenecks as contracts implemented by coding agents also points to a workflow in which engineers specify what to fix rather than hand-correcting the robot, which matters for scaling real-world policy improvement across many tasks.

Future Directions

• Automating bottleneck identification so that agents discover the limiting subtasks from observed policy failures instead of humans specifying them.
• Generating dense subtask rewards automatically rather than relying on human-written outcome criteria and, on earbuds, human label checking.
• Constructing recovery phases so the robot can practice behaviors after subtask failures, reducing reliance on human resets.
• Accelerating learning in regimes where successful outcomes are rare, since real-robot training time remains a hard budget constraint.

Target Audience

Robotics researchers and engineers working on real-world reinforcement learning, robot foundation models, and vision-language-action policy adaptation; practitioners deploying manipulation systems that must execute long-horizon tasks with limited robot time and limited human supervision; and readers interested in how agent-authored executable programs can replace recurring human oversight in robot training loops.

Authors’ abstract

A pretrained robot foundation policy may execute most of a long-horizon task yet repeatedly fail at a few critical subtasks. Collecting additional full-task demonstrations for supervised fine-tuning (SFT) requires operators to repeat behaviors the policy already performs well. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches struggle to solve long-horizon tasks using only sparse rewards. We present PARTS (Policy Adaptation with RL on Targeted Subtasks), a real-world subtask RL framework that concentrates practice at these bottlenecks while allowing training rollouts to proceed with minimal human intervention. The frozen pretrained policy supplies nominal actions throughout execution, while agent-generated selectors and success verifiers activate residual corrections and provide local outcome rewards. These rewards support learning from successful subtasks even when complete-task successes are scarce. Training combines online RL with success-reweighted retraining, and each retrained residual policy is redeployed to collect further experience. Humans identify bottlenecks during setup and perform physical resets when needed. On bimanual YAM and single-arm Franka tasks, PARTS improves complete-task success from 32% to 61% and from 50% to 95%, respectively, using tens of minutes of real-world RL rollouts per task on average. Compared with existing real-world RL fine-tuning methods, PARTS raises full-task success by more than 25% under the same robot-rollout budget while requiring less human involvement.