ForgetMimic: Motion Unlearning for Reinforcement Learning Humanoid Control

AI summary

Overview

Research area: Robotics / reinforcement learning for physical humanoid control, at the intersection of machine unlearning, policy fine-tuning, and sim-to-real robot deployment.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning, proximal policy optimization (PPO), reward shaping, motion tracking objectives, and humanoid whole-body control.

Scope: The paper proposes and evaluates the first motion-level unlearning method for RL-driven humanoid control, tested in simulation (mjlab/MuJoCo) and on Unitree G1 and H2 hardware across 12 motions.

What This Paper Is About

Humanoid robots are increasingly trained with a single RL policy that masters many motions at once, but there is no established way to delete one specific skill from an already-trained policy. ForgetMimic addresses this by fine-tuning a pre-trained policy with an "anti-reward" so that designated target motions (for example, Fight and FightAndSports1) become unusable, while the remaining motions keep working. The motivation is both safety (removing dangerous, poisoned, outdated, or suboptimal behaviors) and legal compliance (the right to be forgotten under regulations such as the GDPR).

Key Contributions

1. First motion-level unlearning method for physical humanoid control. The authors state that ForgetMimic is the first unlearning method designed specifically for RL-driven humanoid control in the real physical world, rather than for tabular, game-based, or offline RL agents.
2. Anti-reward fine-tuning (ARFT). A mechanism that injects a selective penalty term on target motions during continued PPO training, proportional to tracking quality, so that better tracking of a target motion produces a larger penalty.
3. Identification and resolution of two conflicting training mechanisms. The paper identifies Reference State Initialization (RSI) and Assistive Wrench (AW) as mechanisms that create positive feedback loops which counteract unlearning, and shows that disabling both is necessary for effective forgetting.
4. Empirical evaluation across two embodiments and 12 motions. Experiments cover the Unitree G1 and H2 robots and motions spanning task-oriented, locomotion-oriented, performance-oriented, and combat-oriented categories, with comparisons against four baselines and ablations.

Main Findings

• Selective forgetting works at λ = 8.0 and 16.0. In the validity analysis reported in Figure 2, target-motion success rate drops from 100% to 0% and episode length collapses from 500 steps to approximately 3–4 steps, meaning the policy terminates almost immediately on encountering the target motion.
• Non-target motions are largely preserved. In the same Figure 2 analysis, mean non-target tracking reward decreases by less than 2% (from 0.87 to 0.85–0.86) and non-target success rates remain above 85%.
• Higher penalty weights produce stronger forgetting. The degradation ratio (target tracking divided by non-target tracking) falls from 0.997 for the base model to 0.851 at λ = 8.0 and 0.646 at λ = 16.0 in the Figure 2 analysis. In the separate Table 2 sensitivity analysis (30,000 fine-tuning iterations), the ratios are 0.756 at λ = 8.0 and 0.628 at λ = 16.0.
• There is a sharp transition at λ = W. For λ below 8.0 (tested at 0.5, 1.0, and 4.0), degradation ratios are 0.994, 0.987, and 0.972 respectively and target success rates stay at 1%, meaning the penalty is too weak to overcome learned behavior. At λ = W = 8.0, target success rate falls to 0%, episode length drops from about 500 to about 3.7 steps, and the degradation ratio falls to 0.756, while non-target mean tracking reward stays above 0.85.
• Removing RSI and AW matters. With only adaptive RSI active (ARFT with RSI), target tracking is 0.7330 ± 0.0102 with 73% non-target success rate; with only AW active (ARFT with AW), target tracking is 0.7054 ± 0.0021 but non-target success rate falls to 43% and non-target episode length to 249.8. With both disabled, ForgetMimic at λ = 8 achieves 0.6653 ± 0.0187 target tracking and 72% non-target success rate.
• Baselines degrade non-target motions severely. Ye et al. (2025) achieves the lowest target tracking reward (0.5027 ± 0.0046) and a target episode length of 3.0 steps, but non-target tracking drops from 0.8435 to 0.6906, episode length plummets from 400.6 to 48.2 steps, and success rate falls from 74% to 6%. Random reward and Gong et al. (2025) leave the policy partially capable on target motions (168.3 and 138.8 steps respectively) and drop non-target success rates to 12% and 23%.
• Retraining from scratch is a costly reference point, not a strict upper bound. Retraining without the target motions yields target tracking of 0.6086 ± 0.0738 with 0% success and 4.9 steps, but requires 100× the computation, i.e. 100,000 training iterations or roughly 40 GPU hours.
• No catastrophic forgetting during long unlearning runs. Across 22 checkpoints and 99,999 fine-tuning iterations, target motions move from roughly 0.86 tracking reward to about 0.52–0.61, while all 10 non-target motions stay in the 0.84–0.88 range with standard deviations below 0.04. Final-iteration non-target mean tracking (0.868) is comparable to the non-unlearning baseline (0.855).
• The method generalizes across target motion types. Applying the same procedure with Fight & FightAndSports1, Dance, and Run as targets in turn shows that unlearning is not tied to one specific target motion.

Methodology in Plain English

The starting point is a policy already trained with PPO to track a library of 12 reference motions. To make the policy forget some of them, the authors keep training with PPO but change the reward: for target motions, the policy receives a penalty equal to a weight λ multiplied by the normalized tracking reward, subtracted from the normal training reward. For every other motion, the reward is left exactly as before, so those skills continue to be optimized. The authors show algebraically that the target-motion reward becomes (1 − λ/W) times the original tracking reward plus the regularizers, where W is the total effective tracking weight; when λ exceeds W, the target reward becomes negative. They also examine the policy gradient: with an unchanged reward, advantages remain positive and the original update direction is preserved, but for target motions the advantage shrinks or turns negative, so the policy learns to avoid reproducing them.

The second half of the method addresses two mechanisms that would otherwise undo the forgetting. Reference State Initialization increases the probability of starting an episode on motions the policy performs poorly on, so penalizing a target motion makes it more likely to be practiced. Assistive Wrench applies corrective forces and torques to the robot's torso when the policy struggles, physically helping it complete the motion, which directly counteracts the penalty. Both mechanisms fail to distinguish "poor performance because the policy is weak" from "poor performance because the anti-reward is penalizing success," so the authors disable both during unlearning. The resulting procedure is summarized in Algorithm 1: override RSI, disable the assistive wrench, sample motions and start frames uniformly, and apply the modified reward.

Evaluation uses MuJoCo through mjlab with the implicitfast integrator at a 5 ms simulation timestep, a decimation factor of 4 for a 50 Hz control frequency, and episodes of at most 10 s (500 control steps), running on a single NVIDIA 4090 GPU under Ubuntu 24.04. Both actor and critic are MLPs with ELU activations and observation normalization; the actor has hidden sizes [1024, 512, 256, 128] and outputs a Gaussian with learnable scalar standard deviation (σ₀ = 1.0), and the critic has hidden sizes [1024, 1024, 512, 256] and receives privileged observations. Each model is evaluated on all 12 motions with 20 episodes per clip (240 episodes total) and action noise disabled.

Why This Matters

Impact on research. This work opens a new problem setting: motion-level unlearning for online RL policies in physical robot control. It shows that environment-level or trajectory-level unlearning techniques do not transfer cleanly, because humanoid control trains many motions inside a single environment through a shared actor-critic network, so negative rewards propagate to unrelated skills. It also highlights that robot-specific training machinery (RSI and AW) must be considered part of any unlearning pipeline, and it supplies a benchmark-style setup with four baselines, two embodiments, and 12 motions.

Real-world applications:

• Removing combat-oriented or otherwise dangerous motions from a service robot's latent repertoire before deployment.
• Complying with right-to-be-forgotten requests for copyright-protected or privacy-sensitive motion-capture data used in training.
• Purging poisoned, outdated, or suboptimal behaviors introduced by real-world policy updates without retraining the whole policy.
• Avoiding the cost of full retraining, which the paper quantifies as 100× the computation and roughly 40 GPU hours in their setting.

Industry relevance. Humanoid robot developers that train generalist multi-skill policies face a practical governance problem: a single deployed policy can silently retain skills that operators, customers, or regulators do not want present. ForgetMimic offers a cheaper targeted alternative to retraining, and it demonstrates transfer to real Unitree G1 hardware (with H2 results reported in Appendix A.7).

Future Directions

• Theoretical guarantees for continuous control. The paper notes that exact RL unlearning has only been proven for discrete, finite state and action spaces, while robot state and action spaces are continuous, so formal guarantees remain an open problem.
• Scaling beyond 12 motions. It is an open question how the anti-reward approach behaves on the much larger heterogeneous motion libraries used by generalist policies such as BeyondMimic, SONIC, OmniXtreme, and ZEST.
• Robustness and reversibility of forgetting. The paper does not report whether forgotten motions could be recovered by later fine-tuning, or how resistant the unlearned policy is to attempts at relearning, which matters for security claims.
• Reducing computational cost further. Since disabling RSI and AW removes the adaptive sampling that speeds up learning of hard motions, future work could design unlearning-aware variants of these mechanisms rather than turning them off entirely.

Target Audience

Researchers and engineers working on reinforcement learning for legged and humanoid robots, machine unlearning, and safe or privacy-compliant deployment of learned policies. The paper is also relevant to practitioners who maintain multi-skill humanoid controllers and need targeted removal of specific behaviors, and to readers interested in how robot-specific training mechanisms interfere with policy editing. A working knowledge of PPO, reward shaping, and motion-tracking objectives is assumed.

Authors’ abstract

Humanoid control, leveraging human demonstrations, has achieved diverse, agile, and natural locomotion behaviors through reinforcement learning (RL). While this paradigm has yielded remarkable performance in physical humanoid control, how to eliminate specific motions from learned policies remains insufficiently explored. Addressing this issue is motivated by pressing safety and privacy concerns: the removal of malicious, poisoned, or suboptimal motions, as well as copyright-protected motions subject to the right to be forgotten under regulations such as the GDPR, is of critical importance. To this end, we propose {ForgetMimic}, the first motion-level unlearning method designed specifically for physical-world humanoid control. The core idea of ForgetMimic is as follows: given a policy $π_θ$ trained on $N$ motions, our method degrades performance on a target subset of $K$ motions while preserving the effectiveness of the remaining $N-K$ motions. Furthermore, we identify and resolve two key training mechanisms in robot control that lead to unlearning failure. We conduct extensive experiments on the Unitree G1 and H2 humanoid robots across 12 motions, including Dance, Fight, Flip, and others. Experimental results demonstrate that ForgetMimic effectively eliminates memory of designated motions while maintaining the normal operation of all other motions.