DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation

AI summary

Overview

Research area: reinforcement learning for dexterous robotic manipulation, specifically the state-policy stage of trajectory-guided pipelines such as ViViDex.

Technical level: Intermediate. The paper assumes familiarity with PPO-style on-policy RL, but its central idea — scheduling the action-noise scale — is conceptually simple and explained concretely.

Scope: A paired, within-method study of whether making exploration scale an explicit function of training steps improves final deterministic manipulation success across three policy-optimization settings (PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant, FPO), in simulation on YCB objects and on a real RealMan RM75 arm with an Inspire/RH56 hand.

What This Paper Is About

Dexterous manipulation requires two things that pull in opposite directions: broad action noise helps the robot discover finger–object contacts, but that same noise destroys the contacts once they are found. In the ViViDex trajectory-guided setting, the authors observed that baseline PPO barely reduced its learned action standard deviation — it ended at 0.185–0.186 after 5M steps from a nominal 0.20 initialization — which motivated taking direct control of exploration scale. DexPolicy therefore makes the noise scale an explicit function of training steps, annealing from broad to narrow exploration while leaving loss, architecture, reward, and optimizer settings fixed, and asks whether the final deployed policy improves.

Key Contributions

1. A scalar training-step exploration schedule that raises mean deterministic Target success in all three policy-optimization settings (PPO, GRPO continuation, FPO), in simulation and on hardware, under paired training and a common zero-noise execution rule.

2. A component screen on PPO showing the improvement comes from noise control rather than the tested optimizer contraction, plus a shape screen showing the selected schedule reaches higher mean Target success than uniform linear decay between the same endpoints on the three tested objects.

3. Evidence that training return, deterministic Target success, and tolerance to execution noise dissociate, so schedules should be validated on terminal task success under the intended execution conditions, per task and policy-optimization setting.

4. A real-robot demonstration on a RealMan RM75 arm with an Inspire/RH56 right hand across 360 trials, where all conditions execute policy means, so gains cannot be attributed to sampling less noise at deployment.

Main Findings

• Simulation Target success rises in all three settings. Across five YCB objects (mustard bottle, mug, banana, sugar box, tomato can) and three training seeds, mean deterministic Target success rises from 32.0% to 35.7% for PPO, 14.1% to 45.4% for GRPO, and 49.4% to 68.1% for FPO.

• Hardware gains are larger. On 360 trials over three objects with one model per object–method condition and 20 trials each, mean Target success rises from 8.3% to 43.3% (PPO), 10.0% to 63.3% (GRPO), and 25.0% to 85.0% (FPO). Mean Lift success rises from 38.3% to 58.3% (PPO), 46.7% to 95.0% (GRPO), and 45.0% to 93.3% (FPO). PPO Target success on sugar box remains zero.

• Noise control beats the tested update contraction. In a mustard seed-0 screen over 5M steps, noise scheduling alone raises reward AUC from 19.96 to 27.36 while improving contact (0.325 to 0.427) and lift (0.02995 m to 0.03139 m); update contraction alone lowers reward AUC to 12.35 and lift AUC to 0.00293 m; combining both gives 19.13 reward AUC and fails to recover the baseline.

• Schedule shape matters, but not every detail. A stagewise reference passes; uniform linear decay (91.9% reward retention) and continuous front-loaded decay fail, as does removing the final drop (78.2% lift retention). Removing only the intermediate 2M jump passes (100.3% reward, 95.5% contact, 108.3% lift retention), so the final low-noise plateau is retained.

• Linear decay is not equivalent. A separate uniform linear control (0.20 to 0.04 over 5M steps, three seeds, 100 deterministic episodes/model) reaches 83.7 ± 10.3% on mustard, 29.7 ± 3.5% on mug, and 1.3 ± 0.6% on banana, versus 84.0%, 25.3%, 8.7% for learned-std PPO and 94.7%, 40.7%, 2.3% for DexPolicy.

• Aggregates hide seed-level failures and reversals. FPO scheduling removes failed seeds on mustard (88/67/0% to 100/100/100%) and tomato (95/0/92% to 95/88/95%) but not on mug (65/0/11% to 0/0/93%), which drives the large mug standard deviation. For GRPO, deterministic mustard Target is 0/0/92.5% for the baseline and 100/0.5/3.0% with DexPolicy across seeds 0/1/2; excluding seed 0, the five-object mean still rises from 15.3% to 37.2%.

• Deterministic gains do not imply tolerance to execution noise. For GRPO under matched common noise, five-object Target at σ = 0.03 is 14.8 ± 8.5% versus 41.8 ± 14.1%, but at σ = 0.10 it reverses to 16.1 ± 4.6% versus 8.7 ± 1.6%. Neither nonzero scale was selected as optimal.

• Target and Lift can move in opposite directions. GRPO deterministic Lift declines on mug (79.3% to 76.8%), banana (53.8% to 42.2%), and tomato (95.2% to 90.7%), while the five-object mean rises from 78.9% to 80.6%. In the eight-object PPO/FPO study, Lift falls from 41.9% to 36.0% for PPO but rises from 46.6% to 50.8% for FPO.

• Return improves even where tasks stay unsolved. Eight-object Target means are 20.0% to 22.3% for PPO and 30.9% to 42.5% for FPO, with pitcher base, wood block, and extra-large clamp averaging at most 0.34% Target success in every PPO/FPO execution mode. Reward AUC improves on every object mean in the six-object transfer table (16/18 paired wins overall), with pitcher base lift declining 13.4%.

• Gains persist over a longer budget. In a 10M-step mustard-bottle confirmation, reward AUC 0–10M rises from 36.57 to 47.79 (+30.7%), the 5–10M window rises from 53.09 to 66.44 (+25.2%), contact AUC from 0.620 to 0.704 (+13.5%), and lift AUC from 0.0654 m to 0.0731 m (+11.7%), all winning 3/3 seeds.

Methodology in Plain English

The authors take an existing trajectory-guided pipeline (ViViDex) and change exactly one thing: instead of letting the policy learn how much action noise to use, they freeze that parameter and prescribe a noise schedule based on cumulative environment steps. For PPO the schedule starts near the baseline initialization, decays in two stages, and ends at a low plateau (0.20 → 0.10 over 2M steps, 0.10 → 0.05 over the next 2M, then 0.04), with learning rate 10⁻⁵, clipping range 0.20, five update epochs, and a gradient-norm cap of 0.50 held fixed. The same idea is applied to a critic-free GRPO continuation that uses group-relative reward-to-go advantages (γ = 0.95), no learned value baseline, and no KL penalty, and to an FPO variant where an eight-step conditional flow produces a deterministic action mean with a Gaussian likelihood.

Crucially, the primary outcome is deterministic Target success: both conditions execute their policy means (σ_eval = 0), which separates learned control from evaluation-time noise. Target requires specified contacts and final position within 3 cm of the target; Lift requires specified contacts and height above 5 cm. Policies are trained per object, so the study tests whether the scheduling rule transfers across objects, not whether a single policy transfers. Paired comparisons hold environment, reward, architecture, optimizer, rollout size, and budget fixed within each method pair. Supporting evidence includes common-noise evaluations for GRPO, schedule-shape retention gates against a reference curve, a longer 10M-step run, and hardware trials on a RealMan RM75 arm with an Inspire/RH56 hand using FoundationPose for object poses at 20 Hz.

Why This Matters

Impact on research: The paper reframes exploration magnitude as a first-class design variable rather than something to be learned as a by-product of the update, and it shows that the standard proxy — training return — can rise while terminal task success stays near zero or while seed-level results reverse. That is a methodological warning for how RL manipulation results are reported and validated.

Real-world applications:

• Robotic pick-and-place and bin picking, where a fixed gripper noise level is often tuned by hand and rarely re-validated per object.
• Warehouse and logistics automation, where success rate under repeated physical trials matters more than simulated return.
• Household and service robots that must grasp objects with varied geometry, where the paper shows per-object schedule reuse without per-object retuning.
• Industrial assembly or tool use requiring sustained contact, where the paper's observation that Target gains can coexist with Lift declines is directly relevant to task specification.

Industry relevance: The intervention is cheap — it changes only the noise scale, not the loss or optimizer — so it can be layered onto existing RL pipelines with minimal engineering cost. The hardware result, an increase from 25.0% to 85.0% for FPO, indicates the effect is not confined to a simulator.

Future Directions

• Establishing zero-shot or cross-object policy transfer. Current policies are object-specific with one reference trajectory per object, which the authors explicitly flag as insufficient to demonstrate generalization.

• Addressing the unsolved objects. Three of eight PPO/FPO objects remain effectively unsolved, and extended training was only tested on mustard bottle.

• Separating the mechanisms bundled into the intervention. Resetting, freezing, and annealing are not unseparated in this work, and the shared credit-assignment rule is not separately ablated.

• Building matched-noise diagnostics beyond GRPO. Native stochastic PPO/FPO evaluations mix policy quality with execution noise, and common

Authors’ abstract

Reinforcement learning (RL) for dexterous manipulation must discover finger-object contacts and then control the object precisely; the action noise that serves the first goal can interfere with the second. In trajectory-guided settings such as ViViDex, where RL refine hand-object trajectories from human video, our baseline PPO runs end near their initial action noise after 5M steps, motivating explicit control of exploration scale. DexPolicy makes that scale an explicit function of training steps, annealing from broad to narrow exploration while holding loss, architecture, reward, and optimizer settings fixed. We study three policy-optimization settings: PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO). Across five YCB objects and three training seeds, mean deterministic Target success rises from 49.4% to 68.1% (FPO), 14.1% to 45.4% (GRPO), and 32.0% to 35.7% (PPO). On a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials over three objects raise mean Target success from 25.0% to 85.0% (FPO), 10.0% to 63.3% (GRPO), and 8.3% to 43.3% (PPO), with one trained model per object-method condition. PPO component screening favors noise control over the tested optimizer contraction; the selected PPO schedule yields higher mean Target success than linear decay with the same endpoints on three tested objects. Training return, deterministic Target success, and tolerance to execution noise dissociate; schedules should therefore be judged by terminal task success under the intended execution conditions, per task and policy-optimization setting. Code: https://github.com/AIGeeksGroup/DexPolicy. Website: https://aigeeksgroup.github.io/DexPolicy.