Crossing the Sim2Real Gap Between Simulation and Ground Testing to Space Deployment of Autonomous Free-flyer Control

AI summary

Overview

Research area: Robotics — reinforcement learning (RL) for spacecraft control, specifically Sim2Real transfer for free-flying robots.

Technical level: Intermediate. The paper uses RL terminology (PPO, actor-critic, Generalized Advantage Estimation, curriculum learning, domain randomization), but explains each element in accessible prose alongside the experiments.

Scope in one sentence: The paper reports training a PPO reinforcement learning policy in NVIDIA's Omniverse physics simulator to replace the NASA Astrobee free-flyer's classical attitude and translation control, then validating that policy through NASA Ames' Gazebo simulator, air-bearing ground testing at the Granite Lab, and finally a first-of-its-kind on-orbit test aboard the International Space Station.

What This Paper Is About

Reinforcement learning can learn control policies from experience, but policies trained in simulation often degrade when moved onto real hardware — the "simulation-to-reality" (Sim2Real) gap. That gap is especially risky in space, where the environment is hard to model and testing opportunities are scarce.

This paper asks whether an RL policy trained entirely on the ground can control a real free-flying robot in the microgravity of the ISS. The authors train a deep neural network to replace the Astrobee robot's classical Guidance Navigation and Control (GNC) controller, then push it through a progressive validation ladder from simulation, to a NASA simulator, to a granite-table ground facility, to actual flight on the ISS.

Key Contributions

1. First on-orbit demonstration of RL-based autonomous control of a free-flying robot. The authors state this is the first RL policy to control a free-flyer's motion in microgravity, aboard the ISS using the NASA Astrobee.

2. A complete Sim2Real training and verification pipeline. The workflow proceeds from high-fidelity training in the NVIDIA Omniverse Isaac Lab physics simulator, to validation inside NASA Ames' Gazebo/ROS Noetic Astrobee software stack, to ground hardware testing at the NASA Ames Granite Lab, to ISS flight.

3. A curriculum-trained PPO policy that fits on flight hardware. The actor and critic each have 2 hidden layers of 64 neurons, with a policy that must occupy less than 1 MB of memory to run on Astrobee's Wandboard Dual board for low-level control.

4. Experimental evidence that curriculum training over mass variation yields hardware robustness. The "iss tested" policy performed the undock maneuver on the Granite Lab both with and without the robotic arm payload attached, and a simulated comparison was made against the classical GNC controller.

Main Findings

• Curriculum-trained policies succeed far more often than non-curriculum policies in simulation. Across 10,000 environments and 10 seeds in the Omniverse simulator, the curriculum-trained "iss tested" policy reached the goal pose in 91.99% (no mass variation), 91.89% (±2 kg), and 89.19% (±9.0877 kg) of environments. The "no curriculum" policy reached only 9.77%, 9.74%, and 8.9% respectively.

• The "more mass var" policy was the strongest in simulation. It reached 95.4% (no mass variation), 94.27% (±2 kg), and 92.08% (±9.0877 kg), slightly outperforming the "iss tested" policy. The authors attribute this to its training on greater mass variation from the start. This policy was evaluated only in simulation, not on hardware.

• Success was judged by the Astrobee GNC tolerance. An environment was counted a success only if the robot reached the goal position and orientation within 0.1 m translation error and 20 degrees rotational error. Results were consistent across seeds, with standard deviations under 1%.

• Non-curriculum policies were insensitive to mass variation. The "no curriculum" policy's success rate stayed near 9% whether mass varied by 0, ±2 kg, or ±9.0877 kg — its low performance was not caused by mass uncertainty.

• In the Gazebo simulator, the non-curriculum policy both overshot and undershot the 0.5 m undock target, while the two curriculum policies reached the goal. Their trajectories were less smooth than the classical GNC controller's, but under a large (~7 kg) mass variation they performed similarly to the GNC controller, which itself showed oscillatory motion at that variation.

• Curriculum policies showed mass-variation robustness comparable to the classical controller. Table III reports mean squared error between baseline-mass and varied-mass undock trajectories; the paper states the curriculum-trained policies' MSE values indicate similar robustness to mass variation as the GNC baseline.

• Ground hardware testing validated mass robustness. At the NASA Ames Granite Lab, the "iss tested" policy successfully completed undocking maneuvers both with and without the arm mass payload, providing 3 degree-of-freedom validation before flight.

• The ISS flight test succeeded within tolerance. On orbit, the policy controlled the Astrobee's 6 degree-of-freedom free-flying motion for undock, rotate, translation, and redocking maneuvers. In the undock comparison against the Gazebo simulation, the hardware undershot the 0.5 m target but remained within the acceptable margin of error for a successful undock.

Methodology in Plain English

The researchers built a simulated zero-G environment in NVIDIA's Omniverse Isaac Lab physics simulator and trained the Astrobee there using Proximal Policy Optimization (PPO), an actor-critic deep RL algorithm with a clipped objective and Generalized Advantage Estimation. The actor network outputs six values — three for force and three for torque applied at the robot's center of mass — while the critic outputs a single value estimate. The policy's observations are the robot's linear and angular velocities plus its position and orientation error relative to the goal pose, all in 6 degrees of freedom.

The reward function is simple: positive for being close to the goal pose, negative for excessive linear and angular velocity. Position and orientation error are mapped through a tanh function so the reward approaches one as the robot nears the goal, with a threshold distance corresponding to the Astrobee GNC's own success criterion.

Training used 10,000 parallel environments and over 10,000 training iterations. Three policies were produced. The "no curriculum" policy trained on goals within a 1.0 m radius sphere of the start with randomized orientations. The other two used a 22-level curriculum that progressively widened goal position variation (from 0 up to ±1.5), goal orientation variation (from 0 up to ±0.5 on the quaternion components), and mass variation (from 0 up to ±1.5 kg) around a baseline mass of 9.0877 kg. An environment advanced to the next level only by holding reward above a threshold for 550 or 1000 time steps, each step lasting 0.016 seconds. The "more mass var" policy instead varied mass by ±9.0877 kg from level 1 onward.

Validation then proceeded in stages: the same Omniverse simulator, then NASA Ames' Gazebo/ROS Noetic Astrobee simulation (which runs the flight hardware's own code, allowing direct comparison with the classical GNC controller on an undock maneuver of 0.5 m in the x direction), then the Granite Lab air-bearing table, then the ISS itself.

Why This Matters

The work shows an end-to-end pipeline — simulation training, simulator validation, ground hardware validation, flight — actually closing the Sim2Real gap for a space robot, rather than only arguing that it could be closed. It also shows that a policy small enough to run on constrained flight hardware can still benefit from GPU-accelerated, massively parallel training on the ground.

Real-world applications:

• In-Space Servicing, Assembly, and Manufacturing (ISAM): autonomous free-flyers could inspect, service, assemble, or manufacture in orbit without continuous ground control.
• Autonomous Assembly, Integration and Testing (AI&T): the APIARY experiment this work belongs to aims toward autonomous robotic assembly and testing for future space missions.
• Intravehicular robot assistants: Astrobee operates inside the ISS's pressurized environment, supporting inspection and servicing tasks with its cameras, sensors, and 2-DOF perching arm.
• Rapid on-orbit adaptation: because policies can be trained terrestrially and uploaded, missions could retrain controllers to meet changing requirements rather than relying only on fixed classical control laws.

Industry relevance: the results support the commercial and government push toward more autonomous space robotics, and validate GPU-accelerated simulation (NVIDIA Omniverse Isaac Lab) as a practical training substrate for flight-deployable control policies.

Future Directions

• Testing the remaining policies on orbit. The "no curriculum" and "more mass var" policies were evaluated only in simulation. The authors state they believe these would have been successfully validated on ISS hardware had there been time.
• More complex policies for in-space tasks. The authors propose building on this training and testing method to generate policies for more demanding mission tasks beyond pose-to-pose maneuvering.
• Extending the AI&T process. The paper frames this as an initial validation of a training pipeline for autonomous Assembly, Integration and Testing, with broader ISAM applications to follow.
• Open question — robustness beyond mass. Mass variation was chosen for curriculum randomization because it was unknown whether the Astrobee would fly with or without a payload. Robustness to other unmodeled physical variations between simulation and the ISS is not established by this work.

Target Audience

Robotics and RL researchers working on Sim2Real transfer; spacecraft guidance, navigation and control engineers; space systems integrators and ISAM program managers; and graduate students in aerospace engineering or robotics who want a concrete case study of moving a learned controller from simulation to flight hardware. Readers seeking detailed RL hyperparameters, training curves, reward-scale values, or quantitative flight-test error statistics beyond the reported tolerance margins will not find them in this paper.

Authors’ abstract

Reinforcement learning (RL) offers transformative potential for robotic control in space. We present the first on-orbit demonstration of RL-based autonomous control of a free-flying robot, the NASA Astrobee, aboard the International Space Station (ISS). Using NVIDIA's Omniverse physics simulator and curriculum learning, we trained a deep neural network to replace Astrobee's standard attitude and translation control, enabling it to navigate in microgravity. Our results validate a novel training pipeline that bridges the simulation-to-reality (Sim2Real) gap, utilizing a GPU-accelerated, scientific-grade simulation environment for efficient Monte Carlo RL training. This successful deployment demonstrates the feasibility of training RL policies terrestrially and transferring them to space-based applications. This paves the way for future work in In-Space Servicing, Assembly, and Manufacturing (ISAM), enabling rapid on-orbit adaptation to dynamic mission requirements.