Toward Reliable Sim-to-Real Predictability for MoE-based Robust Quadrupedal Locomotion

AI summary

Overview

• Research area: Robotics — reinforcement learning for quadrupedal locomotion, sim-to-real transfer, and policy evaluation.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning, Partially Observable Markov Decision Processes (POMDPs), teacher-student distillation, and sim-to-real transfer.
• Scope: The paper proposes a unified framework combining a Mixture-of-Experts (MoE) locomotion policy with a predictive sim-to-sim assessment suite called RoboGauge, aimed at selecting and deploying robust proprioception-only quadrupedal controllers on the Unitree Go2.

What This Paper Is About

Training quadruped robots in simulation often produces policies with high simulated rewards that nonetheless fail on real hardware, because the sim-to-real gap and reward overfitting cause policies to latch onto the specifics of the simulated robot. Because validating policies directly on physical robots is risky and slow, the authors build a framework that (1) improves multi-terrain representation through a Mixture-of-Experts policy driven only by proprioception, and (2) predicts real-world transferability through a sim-to-sim scoring suite. The goal is to select reliable policies before hardware deployment, without extensive physical trials.

Key Contributions

1. RoboGauge, a predictive assessment framework. A sim-to-sim methodology that quantifies sim-to-real transferability using multi-dimensional proprioception-based metrics, described as reducing the risk of hardware damage during direct physical deployment.
2. A Mixture-of-Experts policy module. The MoE module is integrated into the policy (specifically the student encoder within the Concurrent Teacher-Student, or CTS, framework) to address deficiencies in multi-terrain representation, demonstrated on a physical Unitree Go2.
3. High-speed locomotion with an emergent gait. The framework enables the robot to reach 4 m/s on flat terrain (reported as 4.01 m/s in the detailed experiments) while developing an emergent narrow-width gait associated with improved stability.
4. Training-stability and command-design improvements. Dynamic velocity tracking precision adjustment, a command curriculum, extreme command sampling, and dynamic command sampling collectively accelerate convergence and raise the peak RoboGauge score by 11%.

Main Findings

• RoboGauge tracks reality better than training metrics. Comparing metric errors against real-world ground truth gathered with a 12-camera NOKOV Mars18H motion capture system at 90Hz, the MuJoCo-based RoboGauge (Ours) produced average errors of 0.0558 (tracking), 0.0117 (safety), and 0.0120 (quality), versus IsaacGym (training) averages of 0.0883, 0.0333, and 0.0380.
• The proposed policy leads on RoboGauge. Final scores: Ours 0.6713 (tracking 0.6669, safety 0.7857, quality 0.7392, level 7.85); CTS 0.5786 (0.5755, 0.7066, 0.6624, 6.83); HIM 0.5379 (0.5453, 0.6476, 0.6050, 6.19); DreamWaQ 0.5054 (0.5105, 0.6149, 0.5730, 5.74).
• Training terrain level alone is misleading. The paper reports that its model does not necessarily reach the highest terrain levels during training compared to baselines, yet the RoboGauge scores correctly reflect underlying performance — stable RoboGauge scores despite fluctuating terrain levels show that training levels fail to accurately represent model performance.
• The MoE design choice matters. Ablations show MoE (Ours) scoring 0.6713, ahead of MoE-NG at 0.6519, AC-MoE at 0.6509, and MCP at 0.6399. Modifying the action network (AC-MoE, MCP) was prone to loss divergence, attributed to expert combination acting directly in the action space.
• Latent space separates terrains and commands. Principal Component Analysis of the student encoder hidden states during 5 s of forward locomotion showed that the MoE architecture achieves superior discrimination of encoding features across terrains and motion commands.
• Physical robustness. On real-hardware survival tests: lateral impulse (80–100 N) — Ours 18/20, CTS 11/20, HIM 8/20, DreamWaQ 7/20, Built-in RL 5/20; tile stairs (15.5 cm, μ = 0.38) — Ours 85/85, Built-in RL 85/85, HIM 24/85, CTS 18/85, DreamWaQ 12/85; 30 cm obstacle (μ = 0.85) — Ours 17/20 with all other methods at 0/20.
• Velocity tracking and speed. Wooden stairs (10 cm rise, 15 cm drop): 2.98 s duration, 1.31 m/s average speed, 0.15 m/s tracking error. On a 30 degree wooden slope: 3.33 s duration, 1.53 m/s average speed, a 1.7 s traversal-time reduction versus the built-in RL baseline. On flat ground (μ = 0.6) within an 8 m indoor runway: peak 4.01 m/s reached within 2.16 s with 0.20 m/s error.
• Disturbance rejection and recovery. The robot stayed stable under continuous lateral pulls of approximately 25–40 N and sudden impulses of 85–100 N, and recovered from a 60 cm drop (reported as 0.6 m in Fig. 8).
• 100% field-test success. In diverse outdoor environments including sand and ice, slopes, and uneven terrain, the robot completed all trials with a 100% success rate and zero unexpected terminations.

Methodology in Plain English

The authors trained locomotion policies entirely in simulation using the IsaacGym environment, running 8192 agents in parallel on seven terrain types (flat, wave, slope, rough slope, stairs up, stairs down, obstacle). Slope inclinations range from 5.7° to 29.6°, rough slopes add random height fluctuations of 5 cm, stair heights range from 5 cm to 25.7 cm with a 31 cm tread width, and obstacles are random cubic structures 5 cm to 27.5 cm tall and 1 m to 2 m wide. The platform is the Unitree Go2, a 12-degree-of-freedom quadruped, with PD gains kp = 20.0 and kd = 0.5, a 50 Hz control frequency, and a 200 Hz simulation frequency.

The robot only uses proprioception — IMU and joint encoders — avoiding cameras, LiDAR, or foot contact sensors that can fail in smoke, poor lighting, or violent shaking. Training uses privileged state information, then distills it into a student policy. The key architectural change is replacing the student encoder with a MoE network: K parallel expert subnetworks combined by a gating network whose softmax weights determine each expert's contribution, plus a load-balancing loss that discourages the gate from activating only one expert. Domain randomization spans friction [0.5, 1.5], payload mass [-1, 1] kg, link mass and actuator properties scaled relative to nominal values, base center of mass [-3, 3] cm on each axis, restitution [0.0, 0.5], actuator offset [-0.035, 0.035] rad, and control latency [0, 20] ms.

RoboGauge, built on MuJoCo, then evaluates trained policies back in simulation through three hierarchical stages: a BasePipeline for single-environment evaluation, a Multi/Level Pipeline for parallelized difficulty and randomization testing, and a Stress Pipeline that synthesizes a unified robustness score. Scoring aggregates normalized metrics using a weighted geometric mean (which penalizes a single failing dimension), applies a Worst-Case Mean over the lowest 50% of motion-goal scores, and uses binary search to find the highest passable difficulty level (passing requires over 80% success across five stochastic seeds). The final terrain quality score combines difficulty level and execution quality with the constraint that the quality weight exceeds the level weight.

Why This Matters

The work addresses a persistent practical bottleneck: researchers cannot cheaply or safely tell whether a simulated policy will work on real hardware. By providing a quantitative proxy for transferability, the framework changes model selection from a gamble into a measurement.

Real-world applications:

• Search and rescue or disaster response, where robots must cross snow, sand, stairs, slopes, and obstacles of roughly 30 cm while relying only on onboard proprioception.
• Industrial inspection and logistics, where consistent velocity tracking and disturbance rejection matter in cluttered facilities with varying floor friction.
• Field robotics in unstructured outdoor terrain, since the model handled sand and ice with a reported 100% success rate and no unexpected terminations.
• High-speed agile platforms, where the narrow-width gait may inform controller design for legged systems operating near their speed limits.

Industry relevance: sim-to-real evaluation suites could reduce the number of prototype iterations and hardware-damaging test runs required before deployment, lowering development cost for legged robot vendors. The reported ability to survive 80–100 N lateral impulses and to be the only tested method to clear a 30 cm obstacle suggests direct value for ruggedized commercial platforms.

Future Directions

• Extending RoboGauge to broader morphologies, specifically humanoid robots.
• Integrating exteroceptive perception with the MoE representation to improve crossing of complex structural obstacles.
• Investigating why action-network MoE variants (AC-MoE, MCP) diverged during training, since the paper attributes the instability to expert combination in the action space but does not resolve it.
• Reconciling the differing metric and randomization counts reported in different sections (the introduction describes 6 metrics with 3 objectives and 4 domain randomizations, while the scoring methodology formalizes K = 8 metrics and M = 9 domain randomizations), and clarifying how training-time and evaluation-time proxies should be aligned.

Target Audience

Robotics researchers and engineers working on reinforcement learning for legged locomotion, sim-to-real transfer, and policy evaluation methodologies. It is also relevant to practitioners deploying quadruped platforms commercially who need evidence-based criteria for selecting policies before committing to physical trials, and to students studying modular neural architectures such as Mixture-of-Experts in control settings.

Authors’ abstract

Reinforcement learning has shown strong promise for quadrupedal agile locomotion, even with proprioception-only sensing. In practice, however, sim-to-real gap and reward overfitting in complex terrains can produce policies that fail to transfer, while physical validation remains risky and inefficient. To address these challenges, we introduce a unified framework encompassing a Mixture-of-Experts (MoE) locomotion policy for robust multi-terrain representation with RoboGauge, a predictive assessment suite that quantifies sim-to-real transferability. The MoE policy employs a gated set of specialist experts to decompose latent terrain and command modeling, achieving superior deployment robustness and generalization via proprioception alone. RoboGauge further provides multi-dimensional proprioception-based metrics via sim-to-sim tests over terrains, difficulty levels, and domain randomizations, enabling reliable MoE policy selection without extensive physical trials. Experiments on a Unitree Go2 demonstrate robust locomotion on unseen challenging terrains, including snow, sand, stairs, slopes, and 30 cm obstacles. In dedicated high-speed tests, the robot reaches 4 m/s and exhibits an emergent narrow-width gait associated with improved stability at high velocity.