Reward-Free Continual Adaptation for Resilient Space Robots

AI summary

Overview

Research area: Robotics, specifically continual reinforcement learning and model-based RL for space robotics.

Technical level: Advanced. The paper assumes familiarity with latent-state world models, the DreamerV3 architecture, Recurrent State-Space Models (RSSM), actor-critic training, KL-divergence losses, and domain randomization.

Scope: A simulation-only study showing that a DreamerV3-style world model, pre-trained across randomized simulations and deployed with a frozen encoder and reward predictor, can adapt a control policy to severe hardware failures using only unsupervised rollouts and imagined trajectories.

What This Paper Is About

Space robots cannot rely on precise reward signals during deployment, because there are no external tracking systems and the environment is too complex to measure outcomes like excavated regolith volume reliably. Continual reinforcement learning could let a robot keep adapting after hardware degrades, but it normally requires those unavailable rewards. The authors test whether a world model pre-trained in simulation already encodes a usable reward landscape in its latent space, so that online adaptation can proceed by updating only the transition dynamics and never observing a new reward.

Key Contributions

1. A reward-free continual adaptation framework: The work freezes the observation encoder, decoder, reward predictor, and continuity predictor of a pre-trained DreamerV3 agent, and updates only the sequence model and forward dynamics during deployment.

2. An empirical demonstration across three space robotics domains: Planetary traversal, orbital navigation, and precision screwdriving assembly, each paired with an unmodeled morphological failure that shifts the transition dynamics.

3. A strict simulation/deployment separation protocol: Pre-training uses 20 million environment steps, 32 policy updates per step, and 512 parallel environment workers; adaptation is restricted to a single simulated agent environment for exactly 60 minutes.

4. An honest account of the method's limits: The paper reports that the approach recovers performance initially but decays over time, attributing this to a capacity limit in the RSSM when the reward head stays frozen while dynamics keep updating.

Main Findings

• Zero-shot transfer fails catastrophically: Pre-trained policies collapse across all three domains once the unmodeled dynamics shift appears, confirming that behavior learned purely in imagination depends on the original transition dynamics.

• The tasks remain solvable after degradation: An agent retrained from scratch on the new dynamics reaches high performance, serving as an asymptotic upper bound, but the paper characterizes this route as extremely sample inefficient.

• Privileged rewards adapt fastest: An adaptive agent with access to privileged reward information recovers rapidly and approaches the retrained upper bound.

• Reward-free adaptation works, but only partly: The proposed agent shows promising initial recovery in all three domains, demonstrating that the latent reward landscape can guide adaptation without external rewards, yet it consistently underperforms the privileged baseline.

• Late-stage volatility and decay: After the initial gain, the agent becomes unstable and loses performance, most notably in the orbital navigation and assembly tasks. The authors suggest that updating transition dynamics on degraded morphologies causes the RSSM representation to drift away from the original latent space.

• Adaptation window differs by control frequency: The fixed 60-minute budget translates to 90K steps for planetary traversal, 36K steps for orbital navigation, and 180K steps for screwdriving assembly. Each task's adaptation phase was repeated across three random seeds with identical pre-trained models.

Methodology in Plain English

The researchers start from DreamerV3, a model-based reinforcement learning agent that compresses high-dimensional observations into a compact latent state. The world model has six pieces: a sequence model that carries memory forward, a forward dynamics model that predicts the next latent state, an encoder that turns observations into latent states, a decoder that reconstructs observations, a reward predictor, and a continuity predictor.

During pre-training in simulation, all of these are trained together while an actor-critic policy learns entirely inside imagined rollouts. Domain randomization over physical parameters such as the gravity vector, inertial properties, friction coefficients, and random external disturbances pushes the encoder to learn a broadly generalizable mapping and the reward head to reflect an objective that stays invariant to perturbations.

At deployment, the robot's hardware has degraded, so the real transition dynamics no longer match what the policy expects. Rather than asking for rewards that cannot be measured, the authors freeze the encoder, decoder, and reward predictor, and update only the sequence model and forward dynamics using a KL divergence loss between the posterior and prior distributions on the new proprioceptive transitions. To avoid catastrophic forgetting they lower the world model learning rate by an order of magnitude, from 4×10⁻⁵ to 4×10⁻⁶, and inject small Gaussian exploration noise drawn from N(0.0, 0.02) into the normalized output actions so the model encounters novel transitions. The actor-critic is then re-trained in imagination using the updated dynamics at a ratio of 2048 policy updates per environment step. For fixed-horizon tasks the continuity predictor could be frozen or updated; the authors freeze it because termination conditions did not change.

Why This Matters

The paper targets a real bottleneck: on a physical spacecraft or rover, the information needed to compute a reward is often simply unavailable, so continual learning has been hard to justify for flight hardware. The authors' proposal reframes the problem as preserving a reward landscape learned in simulation rather than measuring one in the field.

Impact on research: It connects offline world-model pre-training to online adaptation under unobservable rewards, and it reports negative results (decay, volatility) alongside the positive ones. That failure mode is useful for anyone studying representation drift in latent dynamics models.

Real-world applications:

• Planetary rovers that lose wheel or steering actuators on long traverses, such as the wheel failure that occurred on the Mars Curiosity rover.
• Orbital servicing and navigation spacecraft that lose thrusters and must keep tracking a moving target pose.
• In-space assembly with manipulators whose tools become physically misaligned, such as a bent screwdriver mounting flange.
• Terrestrial analogue facilities and other extreme, hard-to-instrument environments, where the same reward-free logic applies.

Industry relevance: Space agencies and commercial space robotics developers looking for autonomy that degrades gracefully rather than failing outright are the clearest beneficiaries. The release of source code through the Space Robotics Bench also makes the approach reproducible for groups already working in NVIDIA Isaac Lab.

Future Directions

• Bounding dynamics updates: The authors propose investigating localized latent-space adapters that strictly limit how far transition dynamics can move, to prevent the late-stage decay observed here.
• Closing the sim-to-real gap: The study is simulation-only and explicitly bypasses the peculiarities of transferring to physical hardware.
• Efficient in-situ learning: The online adaptation phase demands computation that heavily exceeds the power constraints of space-grade embedded compute modules, so new efficiency breakthroughs are needed.
• Handling altered episode structure: The continuity predictor was frozen because termination conditions were unchanged in these experiments; the paper suggests updating it may be necessary if degradation significantly alters episode structure.

Target Audience

This paper suits researchers and graduate students in reinforcement learning and robotics who already understand model-based RL and latent world models. It is also relevant to space robotics engineers and autonomy architects evaluating whether continual learning can be deployed under flight constraints, and to practitioners using DreamerV3, NVIDIA Isaac Lab, or the Space Robotics Bench who want a concrete recipe for freezing heads and updating only dynamics during deployment.

Authors’ abstract

Space robots operate in extreme environments where hardware degradation can critically compromise traditional control strategies. While continual reinforcement learning offers a promising mechanism for online adaptation, it inherently requires access to a reward signal during deployment. However, precise reward computation in space is often infeasible due to the lack of external tracking systems and the overall complexity of the environment. To address the challenge of unobservable rewards, we introduce a reward-free continual learning framework that leverages latent-state world models. By pre-training a model-based agent across diverse simulations, the world model learns a robust predictor of the reward structure within its latent space. Upon deployment to an environment with severe hardware degradation, we freeze the observation encoder and reward predictor to update only the transition dynamics of the world model through unsupervised rollouts. By training the policy entirely on imagined trajectories generated by this updated world model, the agent adapts to altered dynamics without receiving new rewards. We demonstrate our approach across simulated planetary traversal, orbital navigation, and precision assembly tasks subjected to severe morphological failures.