Agile perceptive multi-skill locomotion for quadrupedal robots in the wild

AI summary

Overview

Research area: Legged robotics, specifically reinforcement learning (RL) and trajectory optimization for quadrupedal locomotion over unstructured terrain.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning, trajectory optimization, variational autoencoders, transformers, and sim-to-real transfer.

Scope: The paper presents APT-RL (Action Pretrained Transformer-based Reinforcement Learning), a unified framework that lets a single onboard policy on the KAIST HOUND quadruped autonomously select gaits and motor skills for high-speed perceptive locomotion across indoor, urban, and wild terrain.

What This Paper Is About

Quadrupedal robots need many different motor skills and smooth transitions between gaits to cross rubble, stairs, gaps, logs, and steep steps at speed. Existing methods mostly split along a trade-off: high-speed work is limited to relatively flat ground, while controllers that handle obstacles are typically limited to moderate speeds of roughly 1 m/s, and many strong demonstrations rely on external state estimation (such as motion capture) rather than onboard sensors. This paper's goal is a single control framework that runs only on onboard perception and computation, yet can autonomously choose and switch between gaits and skills while traversing complex real-world terrain at high speed.

Key Contributions

1. APT-RL, a unified three-phase control architecture. The framework combines a representation learning phase (a Transformer-based variational autoencoder, or TVAE, trained on torque-annotated motion data from trajectory optimization), a reinforcement learning phase (a policy that selects latent actions and refines them with an auxiliary action), and a perceptual distillation phase (a student exteroceptive encoder trained to mimic a teacher encoder based on privileged height maps).

2. A large-scale, torque-annotated 2D motion dataset generated cheaply with simplified dynamics. Using Single Rigid Body Dynamics (SRBD) in trajectory optimization, the authors produced 180,000 trajectories with a total duration of 15.5 hours in just 8 minutes. Because trajectory optimization jointly generates motions and their control inputs, no separate RL stage is needed to train a reference-tracking policy, and no online trajectory optimization is used during training.

3. An auxiliary action mechanism that extends the pretrained decoder beyond its dataset. Rather than a traditional residual policy that refines a fixed base policy, the policy jointly learns latent actions and auxiliary actions. This lets the robot generate behaviors absent from the pretraining data, such as jumping over logs, recovering from a broken leg, and in-place rotation.

4. Real-world validation of autonomous multi-skill, multi-gait locomotion on the KAIST HOUND robot. Using only onboard LiDAR, a depth camera, and onboard computation, the robot traversed a 1.1 km urban campus environment and a 0.34 km forest trail, and cleared obstacles including stairs, hurdles, high steps, logs, fallen branches, gaps, and stepping stones.

Main Findings

• Instantaneous peak speeds and Froude numbers. The robot reached an instantaneous peak speed of 4.25 m/s while overcoming a high step involving combined jump-up and drop-down events, and 6 m/s while jumping down a three-step staircase (each step 30 cm high and 58 cm deep), measured immediately before ground impact. The corresponding Froude numbers were 3.85 (at 4.25 m/s) and 7.69 (at 6 m/s), which the authors describe as establishing a new benchmark for perceptive quadrupedal locomotion in challenging real-world conditions.

• Autonomous gait selection depends on terrain, state, and commanded speed. In a fixed 2 m/s command test, the policy chose trot for a lower obstacle (0.175 m) and bound for a higher obstacle (0.44 m). On the same terrain it used trot at 1 m/s and switched to bound above 4 m/s. In the urban course it trotted down a staircase at a 1 m/s command but bounded up stairs at a 4.3 m/s command. In the indoor course it trotted to climb stairs, then switched to bounding at 4.41 seconds before jumping off a 90 cm vertical step.

• Trot and bound are the most complementary gaits. In a five-gait comparison (trot, bound, pace, gallop, pronk) evaluated on rough and discrete terrain, stairs, high steps, and stepping stones using success rate, velocity tracking reward, and inverse cost of transport (1/COT), trot showed strong stability and low cost of transport, while bound achieved higher success rates at high speed and on terrain with large height variations such as high-step obstacles. Pace was often energy-efficient but had lower success rate and velocity tracking than trot; gallop and pronk showed larger variance and lower success rates.

• The pretrained latent space is reused and extended. PCA of latent actions across pretraining, policy training, and real-world deployment showed the policy initially concentrating in a subset of the pretrained cluster and expanding outward as learning progressed. During deployment it used latent actions overlapping pretrained regions for familiar gaits and extended into novel areas for challenging or unseen terrain. t-SNE embeddings of real-world latent actions on flat, stepping stones, stairs, high-step, and log terrain clustered by gait type, with sub-clusters corresponding to motion variations.

• Auxiliary torque enables behaviors outside the pretraining data. In a log-jump scenario, decoder torque dominated on flat ground before the log, but auxiliary torque became more prominent as the jump began. After a leg fracture (an out-of-distribution event), the policy reduced decoder torque on the hip flexion-extension motor and increased auxiliary torque to alter leg trajectory and maintain balance. During in-place rotation, the hip abduction-adduction (HAA) motor had no decoder torque at all because the pretraining dataset lacked torque data for it, so the policy relied on auxiliary torque to produce yaw rotation.

• The automatic gait controller outperformed fixed-gait baselines. Compared against trot-only and bound-only controllers across seven terrains and three command-speed groups (low: 0-3 m/s, high: 3-6 m/s, mixed: 0-6 m/s), the Auto controller scored higher on best-performance rate, average relative performance regret, and worst-case normalized score. Bounding fraction generally increased with command speed and difficulty, but at different rates by terrain: high-step, hurdle, and gap settings showed elevated bounding fractions even at low speeds, while discrete and rough terrain kept more trotting until higher velocities.

• Motion priors from flat terrain beat imitation and from-scratch baselines. Against Adversarial Motion Priors (AMP) and vanilla RL on flat terrain, hurdles, and stepping stones, the method achieved comparable success rates with consistently lower cost of transport. AMP had a higher COT across all three terrains, while vanilla RL showed substantial variation in success rates across terrains under a single reward setting.

• Better reuse and sample efficiency than hierarchical RL with a residual policy. On previously unseen 3D terrain, APT-RL achieved higher velocity-tracking reward and reached higher terrain difficulty levels than an HRL baseline augmented with a residual policy, while using fewer overall training samples because it did not require an expert pretraining phase.

• More robust dynamic gait transitions. Under externally triggered switches between trot and bound at 2 Hz with 50% probability on terrain difficulty 4 to 10, APT-RL achieved higher transition success rates (defined as the proportion of transitions during which the robot remains alive) across command velocities. The HRL baseline failed to stabilize, showing stumbling and reduced survival time.

• Both perception modalities matter. Comparing LiDAR-only, depth-only, and combined policies at command velocities from 1 to 7 m/s, the combined method achieved the highest success rates. LiDAR-only was relatively strong on low stairs, rough terrain, gaps, and high steps; depth-only was particularly effective on hurdles and stepping stones. The 2D LiDAR offers more than three times the vertical field of view of the depth camera and a faster 40 Hz update rate.

Methodology in Plain English

The researchers built the controller in three stages, gradually moving from simple to complex behavior:

1. Generate motion data cheaply. They used trajectory optimization with a simplified model (Single Rigid Body Dynamics) that captures the essential whole-body dynamics at much lower computational cost. This produced a huge library of 2D flat-terrain motions together with the torque commands that produce them.

2. Learn a shared latent space and a torque decoder. A Transformer-based variational autoencoder was trained on that dataset so that different speeds and gaits map into a structured, continuous latent space. A separate supervised step trained a decoder that turns a latent representation into joint torque commands. Because the dataset contains torques, there is no need for a second RL stage to recover control commands, and no online trajectory optimization during training.

3. Train a policy that uses the decoder plus an auxiliary action. During reinforcement learning, the policy picks latent actions from its observations, which are decoded into torque, and simultaneously learns an auxiliary action whose torque (computed through a low-level PD controller) refines the decoder's output. This combination covers both well-learned behaviors and new or unexpected situations.

4. Distill perception for deployment. A student encoder taking real depth images and 2D LiDAR data was trained to imitate a teacher encoder that had access to privileged height maps, so the deployed policy can run on the robot's actual sensors. Custom-made mechanical vibration absorbers were installed on the robot body to improve sensor stability under shock and vibration.

The final policy was deployed on the KAIST HOUND robot with zero-shot sim-to-real transfer, using only onboard LiDAR, a depth camera, and onboard computation.

Why This Matters

Research impact. The paper shows that physically grounded, torque-annotated motion data generated from a heavily simplified dynamics model on flat terrain can serve as a strong and reusable prior for complex 3D, high-speed, perceptive locomotion. It also provides evidence that a single unified policy can beat fixed-gait and hierarchical baselines on spectrums of speed and terrain, and it documents the roles of decoder torque versus auxiliary torque in producing behavior beyond the training distribution.

Real-world applications:

• Search-and-rescue in rubble, collapsed structures, and disaster zones where terrain is irregular and the robot must move quickly.
• Inspection and monitoring of forests, mountains, and other natural outdoor environments where paths are unstructured.
• Urban delivery and patrol on routes that mix asphalt, stairs, grass, and ramps.
• Field robotics or defense operations requiring rapid traversal of obstacle-laden ground, implied by the involvement of the Agency for Defense Development.

Industry relevance. The framework's efficiency story matters commercially: 180,000 trajectories totaling 15.5 hours of motion generated in 8 minutes, no online trajectory optimization during training, no separate RL stage for reference tracking, and no motion capture infrastructure at deployment. That combination lowers the practical cost of producing a single onboard policy that covers many terrains and speeds.

Future Directions

• Closing the gap between 2D training data and 3D deployment. The decoder was trained only on 2D flat-terrain data, which the paper states lacks coverage of full 3D dynamics and terrain complexity. How much of the remaining behavior must come from the learned auxiliary action versus richer training data is an open question.
• Filling gaps in the pretraining dataset. The pretraining data contained no torque data for the hip abduction-adduction (HAA) motor, leaving the policy to compensate with auxiliary torque during rotation. Extending data generation to cover such joints and behaviors could reduce that reliance.
• Richer terrain and sensing conditions. The paper reports the perception analysis separately; how the depth and LiDAR combination behaves under weather, lighting, dust, or reflective surfaces is not reported.
• Scaling gait and skill diversity. The work deliberately selected trot and bound as the two primary torque-level primitives after evaluating five gaits. Whether additional gaits can be integrated without the inconsistency observed for pace, gallop, and pronk remains open.

Target Audience

This paper is most valuable to robotics researchers and graduate students working on legged locomotion, reinforcement learning for control, and sim-to-real transfer, especially those interested in motion priors and reusable skill representations. It is also relevant to engineers at robotics companies building quadruped platforms for inspection, delivery, or field operations, and to program managers evaluating how close quadruped systems are to reliable high-speed operation in unstructured outdoor environments. Readers without a background in RL or trajectory optimization will find the methods section demanding.

Authors’ abstract

Enabling quadrupedal robots to traverse complex terrains-from rugged outdoor environments to urban landscapes-requires seamless integration of multiple motor skills, smooth transitions between gaits, and high-speed perceptive locomotion using only onboard sensors. We present APT-RL (Action Pretrained Transformer-based Reinforcement Learning), a unified framework that enables multi-skill locomotion to achieve high-speed traversal in complex environments through autonomous skill transitions utilizing only onboard perception and computation. Our approach generates large-scale, feature-rich 2D motion datasets through trajectory optimization with simplified dynamics. These datasets enable training of diverse, reusable locomotion skills that transfer effectively to a real quadruped robot operating on complex uneven terrains. The resulting high-quality skills serve as strong priors for efficient learning of complex downstream tasks and extend naturally to 3D environments, enabling smooth, high-speed multi-skill locomotion in deployed policy. Real-world experiments demonstrate the framework's capabilities: the robot performs agile maneuvers through complex indoor obstacles and outdoor wild environments, including dynamic drop-down maneuvers that reach instantaneous peak speeds of up to 6 meters per second. A single onboard policy enables robust traversal of diverse obstacles, including stairs, hurdles, stepping stones, gaps, and fallen branches, demonstrating the versatility and effectiveness of our approach.