Beyond Egocentric Limits: Multi-View Depth-Based Learning for Robust Quadrupedal Locomotion

AI summary

Overview

• Research area: Legged robotics, deep reinforcement learning, multi-view perception and sensor fusion for quadrupedal locomotion.
• Technical level: Advanced (assumes familiarity with reinforcement learning, teacher-student distillation, and depth-based perception).
• Scope: The paper proposes and evaluates, entirely in simulation, a quadruped locomotion framework that fuses egocentric and exocentric depth streams, trained with domain randomization to tolerate signal dropout and remote-camera displacement.

What This Paper Is About

Most learned parkour-style locomotion controllers for legged robots see the world through a single first-person (egocentric) camera, so their performance collapses when that view is occluded or ambiguous. This paper asks whether adding a second, external (exocentric) viewpoint, such as a camera carried by a hovering drone, gives a quadruped enough extra environmental context to cross gaps, descend steps, and traverse other discontinuous terrain more reliably. The goal is a single control policy that uses both views when available and degrades gracefully when the remote view is lost or misaligned.

Key Contributions

1. A two-phase reinforcement learning framework that extends the teacher-student distillation approach of prior parkour work to integrate both egocentric and exocentric depth observations.
2. A domain-randomized training process that adds stochastic visual dropout and camera perturbations (orientation, 3D position, and latency) to emulate real-world sensing uncertainty, including aerial-ground cooperative sensing.
3. A demonstration that multi-viewpoint visual policies outperform the single-viewpoint baseline on complex parkour-like terrain.
4. A systematic study of viewpoint misalignment, training six policies with different remote-camera displacement radii and cross-evaluating them under six test radii, which maps out the operational bounds of camera alignment.

Main Findings

• Multi-view fusion improves performance: The combined vision policy (trained without dropout) reached an average episode reward of 18.97 ± 5.60, a traversal success rate of 90.24 percent, and a mean x-displacement of 0.93 ± 0.19, compared with 13.66 ± 6.24, 61.02 percent, and 0.73 ± 0.31 for the onboard-only baseline of prior work.
• Dropout-aware training is what makes the policy robust: The combined vision model trained with random dropout retained a traversal success rate of 78.77 percent and mean x-displacement of 0.89 ± 0.23 when the remote signal was dropped at validation, versus 28.23 percent and 0.43 ± 0.25 for the model trained without dropout, and versus 41.60 percent for a remote-only policy.
• Graceful fallback under full disconnection: With the remote camera permanently zeroed, the dropout-trained combined policy achieved 63.86 percent success and 0.74 ± 0.32 mean x-displacement, close to the 61.02 percent and 0.73 ± 0.31 of the egocentric baseline; the non-dropout-trained combined policy fell to 23.19 percent.
• Overreliance without dropout exposure: Qualitative sequences show the policy trained without dropout moving erratically and failing when remote vision is lost during step descent, while the dropout-trained policy maintains a stable gait by shifting to the egocentric view.
• Moderate viewpoint misalignment is tolerable: A policy trained with no positional variation (training radius 0.0 m) dropped from almost 90 percent traversal success at perfect alignment to below 30 percent with only a 0.1 m offset. Training radii of 0.4 m or larger added regularization but slightly reduced peak performance and increased variability, while the best trade-off was around 0.3 m, where success stayed above 80 percent and mean x-displacement stayed near its maximum.
• Qualitative gap-crossing improvement: In a gap-crossing sequence, the egocentric baseline misjudged the gap distance and performed an early, insufficiently long jump, whereas the combined vision policy estimated the gap size from both views and executed a well-timed leap.

Methodology in Plain English

The authors train two policies in sequence inside the Isaac Gym simulator using a Unitree Go2 quadruped. In the first phase, a "teacher" policy is optimized with Proximal Policy Optimization using privileged information that a real robot would not have: full terrain geometry represented as a low-resolution elevation map under the robot, plus ground-truth environment parameters such as friction, added mass, center-of-mass offset, and actuator strength. The physics simulation runs at 200 Hz, the control policy at 50 Hz, and depth sensing at 10 Hz.

In the second phase, a "student" policy is distilled from the teacher by DAGGER-style imitation. The student sees only proprioception (base orientation, velocities, joint positions and velocities, and foot contacts) plus two synchronized depth streams modeled on an Intel RealSense D435i: an onboard egocentric camera and a remote exocentric camera. Depth images are 106 x 60 pixels with an 87 degree horizontal field of view, clipped at 3 m, cropped and resized to 87 x 58 pixels, and min-max normalized. Separate convolutional backbones encode each stream, which are combined with proprioception, passed through a gated recurrent unit, and reduced to a latent representation by a multilayer perceptron that also estimates the heading toward the current and next goal. A binary flag tells the policy whether the remote camera is available.

To make the policy robust, the training applies continuous domain randomization: friction from 0.6 to 2.0, added base mass from 0.0 to 3.0 kg, center-of-mass offsets from -0.2 to 0.2 m, actuator strength scaling from 0.8 to 1.2, external velocity pushes of 0.0 to 0.5 m/s applied every 8 s, and a 20 ms delay on visual streams. The onboard camera pitch is randomized within plus or minus 5 degrees, and the remote camera's roll, pitch, and yaw are also randomized. Remote camera position is perturbed within a sphere using a uniform spherical displacement model, with the radius varied across experiments. Finally, the remote stream suffers stochastic dropout: with probability 0.1 the input becomes a zero tensor for at least a duration drawn from a normal distribution with mean 2.0 s and standard deviation 0.1 s, after which it reconnects with probability 0.1.

Evaluation averages results over 256 parallel runs on 25 procedurally generated terrains of four types (hurdles, steps, gaps, and parkours combining hurdles, steps, gaps, and slopes) for 1500 iterations, using five metrics: average episode reward, traversal success rate, mean x-displacement, mean edge violation, and average dropout time.

Why This Matters

The work shows that heterogeneous visual feedback can raise both the agility and the fault tolerance of learned locomotion controllers, and that explicitly simulating sensor loss during training is what produces that fault tolerance rather than simply adding a second camera. It also provides concrete bounds on how much remote-camera misalignment a learned policy can absorb, which is useful for anyone assuming a perfectly positioned external observer.

Real-world applications:

• Aerial-ground robot teams where a drone provides an overhead view to a ground quadruped for inspection, search and rescue, or disaster response.
• Legged robots operating in occluded or cluttered environments where a single forward-facing camera cannot see hazards such as gaps or step edges.
• Infrastructure inspection and construction-site monitoring, where an external observer can supply the geometric context the robot's own view lacks.
• Deployment scenarios with unreliable communication links, where graceful degradation to onboard sensing is required for safety.

Industry relevance: the results speak to operators of legged robot fleets who want to coordinate drones and quadrupeds, and to teams building perception stacks that must remain safe when an external data stream fails. The full implementation is released publicly at https://anonymous.4open.science/r/multiview-parkour-6FB8, which lowers the barrier for replication.

Future Directions

• Transferring the proposed system from simulation to real hardware, which the paper explicitly identifies as not yet done.
• Investigating temporal synchronization across asynchronous viewpoints, since the two depth streams may not be aligned in time in a real deployment.
• Extending the framework to heterogeneous multi-robot teams rather than a single quadruped paired with a single remote camera.
• Refining the trade-off between larger positional randomization radii (0.4 m and above), which added regularization but slightly reduced peak performance, and the 0.3 m setting that performed best in this study.

Target Audience

This paper is most useful to robotics researchers and graduate students working on learned legged locomotion, visuomotor control, and multi-view or collaborative perception. It also suits engineers building aerial-ground cooperative systems, and reinforcement learning practitioners interested in teacher-student distillation and domain randomization for sim-to-real transfer. Readers without a background in reinforcement learning or depth-based perception will find the framework's training procedure and results table difficult to interpret without additional reading.

Authors’ abstract

Recent progress in legged locomotion has allowed highly dynamic and parkour-like behaviors for robots, similar to their biological counterparts. Yet, these methods mostly rely on egocentric (first-person) perception, limiting their performance, especially when the viewpoint of the robot is occluded. A promising solution would be to enhance the robot's environmental awareness by using complementary viewpoints, such as multiple actors exchanging perceptual information. Inspired by this idea, this work proposes a multi-view depth-based locomotion framework that combines egocentric and exocentric observations to provide richer environmental context during agile locomotion. Using a teacher-student distillation approach, the student policy learns to fuse proprioception with dual depth streams while remaining robust to real-world sensing imperfections. To further improve robustness, we introduce extensive domain randomization, including stochastic remote-camera dropouts and 3D positional perturbations that emulate aerial-ground cooperative sensing. Simulation results show that multi-viewpoints policies outperform single-viewpoint baseline in gap crossing, step descent, and other dynamic maneuvers, while maintaining stability when the exocentric camera is partially or completely unavailable. Additional experiments show that moderate viewpoint misalignment is well tolerated when incorporated during training. This study demonstrates that heterogeneous visual feedback improves robustness and agility in quadrupedal locomotion. Furthermore, to support reproducibility, the implementation accompanying this work is publicly available at https://anonymous.4open.science/r/multiview-parkour-6FB8