Dynamic Policy Learning for Legged Robot with Simplified Model Pretraining and Model-Homotopy-Inspired Transfer

AI summary

Overview

Research area: Reinforcement learning for dynamic legged locomotion, specifically the transfer of policies trained on reduced-order models to full-body (and real-robot) dynamics. Secondary areas: trajectory optimization, continuation/homotopy methods in numerical analysis, and sim-to-real deployment.

Technical level: Advanced. The paper assumes familiarity with reduced-order models (Single Rigid Body, Linear Inverted Pendulum, centroidal dynamics), Proximal Policy Optimization, imitation learning, numerical continuation, and rigid-body dynamics parametrization.

Scope (one sentence): The paper proposes a two-stage learning framework that pretrains a dynamic-motion policy on a Single Rigid Body model and then transfers it to a full-body simulator along a λ-parameterized path that gradually redistributes mass and inertia between the trunk and legs, validated on flips and wall-assisted maneuvers and deployed on a Unitree Go1.

What This Paper Is About

Training legged robots to perform highly dynamic motions such as flips is difficult: end-to-end reinforcement learning needs extensive reward tuning and often converges to poor solutions, while imitation learning requires hard-to-obtain demonstrations. Reduced-order models such as the Single Rigid Body (SRB) model make training easier and faster, but the mismatch between the simplified model and full-body dynamics—the "model gap"—means a policy trained on the SRB model cannot simply be deployed.

The paper's goal is to bridge that gap smoothly rather than abruptly, by defining a continuous family of intermediate dynamics models between the SRB model and the full-body model, and letting the policy adapt along that path.

Key Contributions

1. A continuation-based learning framework that combines simplified-model (SRB) pretraining with a "model-homotopy-inspired" transfer stage to generate and refine complex dynamic behaviors.
2. A principled parameterized transition path that smoothly moves from simplified to full-body dynamics by gradually redistributing mass and inertia between the trunk and legs, improving both convergence speed and learning stability.
3. Validation across a variety of dynamic behaviors in simulation — gaits, backflip, sideflip, yawspin, wall-assisted jump, and wall-assisted backflip — including a comparative analysis against Direct Transfer, Imitation Transfer, and Vanilla RL baselines.
4. Deployment of learned policies on a real quadrupedal robot (Unitree Go1), including a trot gait and a backflip.

Main Findings

• Faster convergence and higher return than Imitation Transfer (IT): On the wall-assisted backflip task, Model-Homotopy-Inspired Transfer (MHIT) achieved a 19.0% higher normalized return and converged twice as fast as IT (2,240 vs. 4,610 iterations).
• Greater learning stability than Direct Transfer (DT): MHIT maintained a max-min variation range of 0.0042 compared to 0.2285 for DT.
• Better rotational motion quality than IT (best-performing seeds): lower keyframe orientation error (0.129 vs. 0.185), higher pitch rotational velocity (2.004 vs. 1.794 rad/s), and reduced cross-axis angular velocity error (0.368 vs. 0.493 rad/s).
• Vanilla RL consistently failed to discover meaningful motions when trained from scratch without pretraining, highlighting the value of SRB pretraining.
• DT lost behaviors learned during pretraining due to an initial performance collapse caused by the model gap; IT converged to suboptimal policies because the SRB-to-full-body gap persisted.
• Superior disturbance robustness than IT: success rate was measured across combinations of force and torque disturbances, with 100 different disturbance directions tested at each specified magnitude; MHIT showed greater robustness. No disturbances were experienced during training.
• Real-world trot tracking improvement: The SRB policy directly deployed without fine-tuning achieved a velocity tracking RMSE of 0.53 m/s and a yaw rate RMSE of 0.29 rad/s; the transferred policy achieved 0.28 m/s and 0.11 rad/s.
• Real-world backflip success: The SRB policy failed in the real world (insufficient angular momentum from unmodeled composite inertia during leg extension, causing severe under-rotation and landing on its back), whereas the transferred policy executed the backflip, with close alignment between simulated and real trajectories, achieving a 90% success rate over 10 consecutive trials.

Methodology in Plain English

The framework has two stages.

Stage 1 — Simplified model pretraining. The robot is modeled as a single floating rigid body with lumped mass and inertia, massless legs, and ideal point contacts (the SRB model). The policy is a multi-layer perceptron trained with PPO, and it outputs ground reaction forces during stance and residual foot-target offsets during swing (added to a nominal Bézier swing trajectory, then clipped to the leg workspace and against ground penetration). This idealized environment removes leg inertia, leg collisions, and contact instabilities, so the policy can focus purely on body momentum control. Data is collected from 100 parallel environments over a 4-second rollout and trained for eight epochs per iteration. Rewards take an exponential form, r(x;a,b) = a exp(-b‖x‖²), with a linear rotational term for flipping motions, and the total reward is (1 + Σ r_pos) × (Σ r_neg). Tasks are defined by a contact plan plus a nominal swing trajectory across three motion classes: gaits, flips, and wall-assisted maneuvers.

Stage 2 — Model-homotopy-inspired transfer. Rather than jumping straight to full-body dynamics, the authors build a family of intermediate models indexed by a continuation parameter λ ∈ (0, 1], defined as the ratio of leg mass to its original value. As λ grows, leg mass and inertia scale up linearly, while trunk mass, inertia, and center of mass interpolate between the composite values from the nominal pose and the original full-body values—keeping total mass constant so the ground reaction force actions do not change drastically. At λ → 0 the model is effectively the SRB model. Because the equations of motion depend continuously on these inertial parameters, training along λ forms a numerical continuation process: λ moves from 0.01 to 1.0 over 900 iterations using a piecewise linear schedule (increment 0.01 for λ < 0.6, reduced to 0.005 over the next range, and to 0.0025 as λ reaches 1.0), with the simulation timestep adaptively adjusted to avoid numerical instability at low λ. After the continuation phase, the policy is trained further in the full-body environment.

Action conversion and estimation. In the full-body environment, stance-phase GRFs are converted to joint torques via the Jacobian transpose, and swing-phase foot targets are tracked with task-space PD control (Kp = 500, Kd = 5). A two-layer MLP estimator, trained concurrently with the policy, infers linear velocity and contact states from proprioceptive histories to replace privileged simulation information; contact state is thresholded at 0.5 probability.

Evaluation. The method was compared against DT, IT (imitation learning with Reference State Initialization and Early Termination, using an SRB-recorder trajectory as the kinematic reference), and Vanilla RL. Each method was trained five times with distinct initial policies and seeds. Real-world validation used a Unitree Go1 with rotor inertia, sensor and motor command latency, realistic actuator characteristics, and domain randomization over initial poses, friction, trunk inertia, motor strength, and sensor noise.

Why This Matters

Impact on research. The work reframes the reduced-order-model-to-full-body transfer problem as a continuation problem: instead of treating the model gap as something to be tracked through or imitated across, it is traversed gradually through a parameterized family of dynamics. This offers an alternative to hierarchical planner-plus-tracker architectures, which the authors argue constrain the full-body system to the simplified plan and can yield suboptimal behaviors. It also connects legged locomotion learning to continuation/homotopy ideas from numerical analysis and optimization.

Real-world applications:

• Agile quadruped and humanoid locomotion requiring flips, spins, and other high-momentum maneuvers.
• Wall-assisted locomotion, where a robot uses environmental contact for rapid redirection, upward propulsion to reach a higher platform, or propulsion into an aerial maneuver.
• Sim-to-real deployment of learned controllers where full-body effects (such as leg inertia during extension) determine whether a motion succeeds.
• Reduced-cost controller development workflows, since pretraining in the simplified environment is described as computationally inexpensive and serves as a shared initialization.

Industry relevance. The framework targets the practical bottleneck of getting dynamic behaviors out of RL without extensive reward engineering or expensive motion-capture demonstrations. Because the SRB pretraining stage is cheap and the transfer stage is stable, the pipeline is a plausible template for robot manufacturers building whole-body dynamic controllers. The reported real-robot results—improved velocity tracking and a successful backflip on a Unitree Go1—are direct evidence that the approach survives the reality gap. The work was supported by the Korean MOTIE Robot Industry Technology Development program for dexterous and agile humanoid robots for industrial applications.

Future Directions

1. Extending the approach to humanoids and other platforms. The framework was validated on a quadruped (Unitree Go1) and demonstrated on quadrupedal flips and wall maneuvers; whether the mass/inertia redistribution path remains well-behaved for higher-dimensional full-body systems with floating arms is not reported.
2. Handling richer contact dynamics. The authors explicitly note that the formal topological conditions of a homotopy are not strictly satisfied because of the hybrid nature of contact dynamics; addressing contact discontinuities within the continuation path is an open problem.
3. Understanding why the transfer preserves reactivity. The paper attributes improved robustness to reactive behaviors retained from pretraining and to broader state-space exploration without early termination, but a more systematic characterization of which behaviors survive transfer and why is not reported.
4. Reducing dependence on hand-designed task structure. Each motion still requires a designed contact plan, nominal swing trajectory, and keyframe targets; automating the construction of these task definitions is not addressed.

Target Audience

Researchers and graduate students in legged robotics and reinforcement learning who work on dynamic locomotion, reduced-order model transfer, or sim-to-real deployment. It is also relevant to practitioners who need to produce agile behaviors on real hardware without extensive reward engineering or motion-capture data, and to readers interested in applying continuation and homotopy ideas from numerical optimization to learning-based control. The paper assumes prior familiarity with PPO, SRB/LIP reduced-order models, and rigid-body dynamics, so it is not an introductory read.

Authors’ abstract

Generating dynamic motions for legged robots remains a challenging problem. While reinforcement learning has achieved notable success in various legged locomotion tasks, producing highly dynamic behaviors often requires extensive reward tuning or high-quality demonstrations. Leveraging reduced-order models can help mitigate these challenges. However, the model discrepancy poses a significant challenge when transferring policies to full-body dynamics environments. In this work, we introduce a continuation-based learning framework that combines simplified model pretraining and model-homotopy-inspired transfer to efficiently generate and refine complex dynamic behaviors. First, we pretrain the policy using a single rigid body model to capture core motion patterns in a simplified environment. Next, we employ a continuation strategy to progressively transfer the policy to the full-body environment, minimizing performance loss. To define the continuation path, we introduce a parametric transition path from the single rigid body model to the full-body model by gradually redistributing mass and inertia between the trunk and legs. The proposed method achieves faster convergence and demonstrates superior stability during the transfer process compared to baseline methods. Our framework is validated on a range of dynamic tasks, including flips and wall-assisted maneuvers, and is successfully deployed on a real quadrupedal robot.