Isaac Sim-to-Real: Reinforcement Learning based Locomotion for Quadrupeds

AI summary

Overview

Research area: Legged robotics — reinforcement learning (RL) for quadrupedal locomotion and zero-shot simulation-to-reality (sim-to-real) transfer.

Technical level: Advanced. The paper assumes familiarity with Markov Decision Processes, policy optimization (PPO), actor-critic networks, domain randomization, and actuator modeling.

Scope: The paper presents an end-to-end RL locomotion framework trained entirely in Nvidia's Isaac Sim / Isaac Lab that transfers without fine-tuning to a Unitree Go1 quadruped, and benchmarks it against the robot's factory-integrated controller on velocity tracking and disturbance rejection.

What This Paper Is About

Policies trained in physics simulators often fail when moved onto real hardware, a mismatch known as the sim-to-real gap. This work builds a reinforcement learning framework in Nvidia's Isaac Sim and Isaac Lab that produces a whole-body locomotion controller for the Unitree Go1, and achieves a "zero-shot" transfer — meaning the policy goes from simulation to the physical robot without any real-world fine-tuning. The goal is to match or exceed the robot's stock model-based controller while being more robust to pushes and collisions.

Key Contributions

1. An end-to-end RL-based locomotion policy with zero-shot transfer to the Unitree Go1 physical quadruped.
2. A set of domain randomization and actuator modeling techniques (including an "actuator-net" learned from recorded joint data) used specifically to close the Isaac Sim sim-to-real gap.
3. Validation on physical hardware and a direct comparison against Unitree's integrated controller, reporting comparable velocity tracking while reaching higher velocities.
4. Reporting of disturbance rejection capability, measured by kicking the robot's trunk three times with increasing force and observing recovery.

Main Findings

• Zero-shot transfer succeeded: The policy produced in Isaac Sim ran on the Unitree Go1 with no real-world training, reaching linear velocities of 2.0 m/s and angular velocities of 1.8 rad/s.
• Velocity tracking comparable to the stock controller: Against a sinusoidal command over thirty-five seconds, the learned policy showed a small phase lag between desired and measured velocities, and Unitree's model-based approach produced similar results. The paper states the learning-based approach matches or produces similar results to a model-based approach; it does not report a numeric error metric such as RMSE.
• The integrated controller struggled at aggressive commands: Instabilities in the model-based approach appeared at higher frequencies and amplitudes (large accelerations and velocities), causing feet to slip, so the maximum and minimum commanded velocities had to be reduced to mitigate the issue.
• Disturbance rejection: The policy recovered from all three kicks, only stumbling and never falling completely to the ground.
• Actuator modeling was decisive: Training the policy before integrating an actuator-net showed significant challenges with the sim-to-real gap.
• Terrain capability: The policy walked over gravel, sand, incline and decline plains on hardware, and up a steep incline pallet (shown in Fig. 1). Simulation training used ramps, random-height boxes, and a bumpy floor.
• Loop rate mattered on hardware: Performance increased as the policy loop rate rose from the trained 50 Hz to 100 Hz. The system achieved up to 400 Hz without GPU acceleration.
• Rich domain randomization: 4096 environments were simulated simultaneously on an NVIDIA RTX 3090 GPU.

Methodology in Plain English

The robot's locomotion problem is framed as a Markov Decision Process: the policy sees a compact set of sensor readings and outputs joint position commands, and is rewarded for tracking commanded velocities smoothly and naturally.

What the policy sees (48 values): base linear velocity, angular velocity, projected gravity, the user's velocity command, joint positions, joint velocities, and the previous action. Sensor values have injected noise (for example 0.02 on linear velocity, 0.01 on angular velocity, 0.05 on projected gravity and joint positions, 0.1 on joint velocity; command and previous action get no noise). Unlike some prior work, these states are not scaled.

What the policy does (12 values): the target positions of the robot's twelve actuated joints, scaled by 0.2 and offset by the nominal standing configuration so the robot starts training near a stable pose. These commands are passed to the learned actuator-net, which converts them into torques.

How it is trained: Proximal policy optimization (PPO) with an actor-critic design. Both actor and critic are multilayer perceptron networks with layers of 512, 256, and 128 neurons and exponential linear unit (ELU) activations. Hyperparameters include an adaptive learning rate, entropy coefficient 0.0025, target KL-divergence 0.01, discount factor (γ) 0.99, advantage estimator discount factor (λ) 0.95, and learning rate 0.001. Training runs for 5,000 iterations and takes roughly 3 hours, with a physics timestep of 0.002 s and a policy timestep of 0.02 s.

Shaping behavior through rewards: Reward terms include feet air time (weight 5.0), linear velocity error (5.0), angular velocity error (5.0), gait synchrony (10.0), and foot clearance (5.0). Penalties discourage unsmooth actions (-1.5), uneven air time (-1.0), base motion in roll and pitch (-2.0), tilted base orientation (-5.0), foot slippage (-1.0), foot force (-0.00002), joint torque (-0.0005), joint acceleration (-0.0002), joint velocity (-0.015), joint deviation from nominal (-0.75), joint position limit violations (-100.0), joint velocity limit violations (-10.0), thigh contact (-10.0), and calf contact (-1.0). The authors note they used higher weights than other work for action smoothness and joint deviation, which improved performance at higher command velocities, and added a foot-force penalty for softer contact and lower hardware damage risk.

Closing the sim-to-real gap: Foot friction, foot restitution, body mass, initial joint positions, and initial velocities were randomized within ±[5, 20]% of their initial values. Table I lists the full randomization set, including foot friction drawn fresh from (0.4, 1.1), center-of-mass offset added in (-0.025, 0.025), link mass scaled by (0.8, 1.3), body velocity added in (-0.25, 0.25), body position added in (-0.5, 0.5), body orientation added in (-0.02, 0.02), joint positions scaled by (-0.3, 0.3), and joint velocities scaled by (-2.5, 2.5). A terrain curriculum gradually increased height and roughness as tracking improved. Random pushes set the base linear velocity to a value in [-1.0, 1.0] m/s every 10 to 15 seconds. Command velocities were sampled from [-2, 3] m/s in x, [-1.5, 1.5] m/s in y, and [-1.5, 1.5] rad/s angular, resampled every 0.3 to 10.0 seconds.

Actuator modeling: Joint data recorded during policy deployment trained a multilayer perceptron actuator-net with three layers of 64 neurons using the soft-sign activation, with kp = 20 and kd = 0.5.

Deployment: The policy was compiled to ONNX and run in a Docker container with the needed Python dependencies on an NVIDIA Jetson Nano. Linear velocity was estimated with a Kalman filter implemented entirely in C++ and compiled with Pybind11 into a Python library, which cut latency enough to reach loop rates of up to 400 Hz. The authors report that the policy plus state collection and estimation can reach up to 400 Hz without GPU acceleration.

Experiments: Hardware tests compared the policy against the Unitree Go1's "default" gait (one of three built-in gaits; the others target running and stair climbing), using a sinusoidal velocity command recorded for thirty-five seconds, plus a three-kick disturbance test with increasing force. During the kick test, raw data was recorded at 25 Hz with the policy running at 100 Hz. The Unitree Go1 is described as an 18-degree-of-freedom quadruped with 12 actuated joints and a trunk expressing a full 6 degrees of freedom.

Why This Matters

Research impact: The work adds evidence that Isaac Sim and Isaac Lab, together with the PhysX 5.4 solver, can produce policies that transfer to hardware without fine-tuning, and that careful actuator modeling (a learned actuator-net) plus domain randomization can be sufficient to cross the sim-to-real gap. Showing a learned policy matching a commercial model-based controller while recovering from large disturbances is a meaningful data point for the learning-based locomotion community, particularly for frameworks that avoid teacher-student pre-trained components.

Real-world applications:

• Inspection and data collection in uneven outdoor terrain such as gravel, sand, and sloped ground, where the policy was validated.
• Search-and-rescue or disaster response, where sudden collisions and pushes are expected and recovery matters (the disturbance rejection test directly models this).
• Warehouse and logistics robots that must execute rapid velocity commands and maneuver in tight spaces.
• Research platforms where teams want to reproduce locomotion results without large-scale hardware fleets, since training used simulation only.

Industry relevance: The framework is built on Nvidia tooling (Isaac Sim, Isaac Lab, PhysX 5.4, RTX 3090 training GPU) and deployed on commodity edge hardware (NVIDIA Jetson Nano), and it targets the Unitree Go1, a widely used commercial quadruped. That combination makes the pipeline directly relevant to robotics companies evaluating whether simulation-trained controllers can replace or augment hand-tuned locomotion stacks.

Future Directions

• Whether the framework generalizes beyond the Unitree Go1 to other quadruped platforms, and to other robot morphologies, is not addressed.
• Whether real-world fine-tuning or online adaptation would push performance beyond the reported velocity tracking (which the paper notes remains comparable to, rather than clearly better than, the integrated controller in tracking) remains an open question.
• The paper does not report a future work section, so systematic studies of which domain randomization terms or reward weights matter most are left open.
• Quantitative characterization of the sim-to-real gap itself — for example a numeric error metric on velocity tracking — is not reported and would strengthen comparisons.

Target Audience

Robotics researchers and graduate students working on reinforcement learning for legged locomotion; engineers evaluating Nvidia Isaac Sim and Isaac Lab for sim-to-real pipelines; and practitioners at companies deploying quadrupeds who need to know how a simulation-trained policy compares with a vendor's stock controller on speed, tracking, and push recovery. Readers without background in RL, Markov Decision Processes, and domain randomization will find the training and reward details dense.

Authors’ abstract

Learning-based approaches to locomotion have risen in popularity in recent years, showing the capability for complex legged locomotion and whole-body control. Reinforcement learning (RL), the primary learning-based approach for locomotion, often utilizes a high-performance simulation tool, providing a controlled and efficient training and development environment. However, policies that perform well in simulation frequently encounter unexpected challenges when deployed on a physical system, known as the sim-to-real gap. This work presents a robust RL locomotion framework capable of whole-body control. The proposed RL framework utilizes Nvidia's new set of simulation tools, Isaac Sim, and its companion RL framework, Isaac Lab, for training, achieving a zero-shot sim-to-real policy. The performance of our policy is validated on physical hardware using the Unitree Go1, with experimental results showing similar velocity tracking performance to the quadruped's integrated controller, with a greater ability to recover from large disturbances, and achieve linear velocities of 2.0 m/s and angular velocities of 1.8 rad/s.