Gain Tuning Is Not What You Need: Reward Gain Adaptation for Constrained Locomotion Learning

AI summary

Overview

Research area: Robot locomotion learning, reinforcement learning (RL), and safe/constrained reinforcement learning for legged robots.

Technical level: Advanced. The paper assumes familiarity with reward shaping, constrained RL (PDO, CPO, IPO, CRPO), control barrier functions, and policy optimization, though the central idea is a simple adaptive rule.

Scope: The paper proposes Reward-Oriented Gains via Embodied Regulation (ROGER), a method that adapts reward-weighting gains online during learning instead of tuning them offline, and evaluates it on a simulated 60-kg Unitree B1 quadruped, a physical 60-kg quadruped, and MuJoCo hopper benchmarks.

What This Paper Is About

Existing locomotion-learning methods require the practitioner to pre-select reward-weighting gains offline (sometimes as many as 16 terms with 11 distinct properly selected values), and even the state-of-the-art constrained RL techniques still violate safety constraints during training, such as a quadruped tipping past an orientation limit. ROGER instead adjusts the balance between the primary reward gain and the penalty gains automatically from the robot's own embodied interaction with the environment: the primary gain shrinks and the penalty gains grow as the robot approaches a constraint threshold, and reverses when the robot is safe. The goal is to eliminate most gain tuning while keeping constraint satisfaction near zero throughout learning, including in real-world learning from scratch.

Key Contributions

1. ROGER, an online reward-gain adaptation rule. The gains are computed at every timestep from the ratio of estimated constraint penalties to user-defined constraint thresholds, with λ₀ₜ = 1 − Δₜ and λᵢₜ = r_λᵢₜ Δₜ, where Δₜ is the clipped sum of squared penalty-to-threshold ratios and Σλ = 1. No additional non-intuitive hyperparameters are introduced beyond the constraint thresholds τᵢ and the uncertainty level k_σ.

2. Demonstration of near-zero constraint violation on a heavy quadruped. On a 60-kg Unitree B1, ROGER produced one violation in 50,000 timesteps (attributed to exploration, with zero violations on reruns without exploration; no falls), comparable to CRPO (zero violations in 25,000 timesteps) and carefully tuned fixed-gain penalties, but with higher final primary reward.

3. Higher performance than state-of-the-art baselines. ROGER reached a final primary reward of 0.6 m/s, about 50% more than the fixed-gain penalty and CRPO (both ≈0.3 m/s, p ≪ 0.01, t-test), and statistically matched the reward achieved by training on the primary reward alone (p = 0.54, t-test) while avoiding the latter's violations.

4. Real-world locomotion learning from scratch in about one hour without falls, over five trials on a physical 60-kg quadruped, plus a MuJoCo hopper study showing 6.6 m hopping distance, 0.33 Nm average torque, and 2° average orientation deviation.

Main Findings

• Failures of prior techniques on the quadruped. Training with the primary reward alone produced over 30° orientation deviations and an estimated violation probability of 0.85. CBF-based fixed-weighting approaches (quadratic qCBF and logarithmic lnCBF) reduced the estimated violation probability to 0.60–0.16, with most deviations below 20°, still significantly worse than the fixed-gain penalty (p ≪ 0.01, two-proportion test).

• A properly tuned fixed-gain penalty works but is fragile. It kept orientation deviation below 10° with an estimated violation probability of 2×10⁻¹², or three violations in 25,000 timesteps. The paper notes that in other work, adjusting a single hyperparameter may require four additional training repetitions, and improper values cause unnatural motion or poor performance.

• Adaptive-weighting results. PDO reached ≈20° orientation deviations with an estimated violation probability of 0.02. OL-AUX showed similar 20° deviations with an estimated violation probability of 0.0006–1.6%, lower than PDO but still significantly higher than CRPO and ROGER. CRPO had zero violations in 25,000 timesteps; ROGER had one in 50,000 timesteps.

• ROGER learned a higher-performance policy than the safe baselines. Primary reward was 0.1 m/s after 50 learning episodes, 0.3 m/s after 250 episodes, and 0.6 m/s at 500 episodes — 50% greater than the fixed-gain penalty and CRPO, while matching the constraint-violating primary-reward-only run.

• Gains vary within a gait, not just across training. In the simulated quadruped, λ₀ₜ and the roll penalty gain λ₁ₜ oscillated with the locomotion rhythm, with two periodic repetitions per 50 timesteps (two gait cycles), meaning constraints are enforced more strongly at specific phases of the gait.

• Physical quadruped results. All five trials completed without falls across 300 episodes. By episode 50 (≈10 minutes) the robot reached 0.1 m/s with roll and pitch below 3°; by episode 250 (≈40 minutes) average forward speed reached 0.3 m/s, matching the simulation. Roll and pitch stayed below the 10° threshold throughout all 300 episodes despite continuous exploration.

• A specific recovery event. Around episode 220 (≈2:36 in the video), the robot began oscillating in roll and pitch; ROGER reduced the primary gain and increased penalty gains, and the robot spent the next five episodes adjusting, briefly losing speed before returning to a stable gait after 3:00 minutes.

• Robustness to challenging terrain and loading. Under dynamic loading (a 1.5-kg bottle at the front, two 1.5-kg bottles at the rear, and a football sack — nearly 10% of the robot's weight), the robot maintained balance. On a slippery surface (machine lubricant on a whiteboard, static friction coefficient ≈0.25) it learned to slide forward while preserving posture. On gravel (diameters 2–7 cm) it leaned forward and kicked top-layer gravel aside. On a step field (heights 4–10 cm) it developed a hopping-like gait, initialized from a trotting-in-place gait with increased foot lift. All behaviors emerged without exteroceptive terrain sensing.

• Hopper benchmark. With default OpenAI Gymnasium reward weighting, CRPO, and ROGER compared over more than 20 trials of one million timesteps each (2048 timesteps per episode), ROGER hopped 6.6 m, a 20% improvement over the default (p < 0.01, t-test) and similar to CRPO. Torque usage averaged 0.33 Nm, a 57% reduction from the default and 58% from CRPO (p < 0.01, t-test). Orientation deviation averaged 2°, a 59% decrease from the default and 53% from CRPO. Constraint violation percentages were lower than both baselines throughout learning and in the last 10 episodes.

• Hopper learning dynamics. During the first 50,000 timesteps, the hopper prioritized reducing torque and orientation penalties before shifting to maximizing the primary reward, creating a slight delay in primary reward maximization. With stricter thresholds (τ₁, τ₂) = (0.5, 5°), reward optimization was delayed further, but the hopper maintained forward hopping without moving backward.

• Ablation behavior in the hopper. ROGER_τ (primary reward plus torque penalty only) produced torque usage mirroring the three-channel ROGER, with orientation deviation similar to the default reward function; ROGER_β (primary reward plus orientation penalty only) produced orientation deviation matching three-channel ROGER, with torque usage similar to the default. The paper states the results may therefore be environment-specific.

• Assumption violations. ROGER is designed around zero-penalty optimality, gentle system dynamics, and gentle learning dynamics. It remained effective when the torque constraint has a non-zero optimum, and outperformed baselines on the MuJoCo hopper, though fall prevention is not guaranteed there. An experiment with excessive learning rates on the quadruped showed that sensitive learning dynamics can lead to failures.

• Computational cost. ROGER adds 0.46 ± 0.09 ms of computation time on an Intel i7 CPU with an Nvidia GTX1050 GPU, approximately 0.03% of data collection/exploration time.

Methodology in Plain English

The researchers start from the standard constrained-RL reward function, in which the total reward is the weighted primary reward minus weighted penalty terms. Instead of fixing those weights, they compute them fresh at every timestep from how close the robot currently is to each constraint threshold: each penalty is divided by its threshold, squared, and summed to give Δₜ (capped at 1.0). The primary gain becomes 1 − Δₜ and each penalty gain is Δₜ distributed in proportion to that penalty's share of the total squared ratio. Penalties are statistically estimated using the average minus k_σ times the standard deviation (k_σ = 3, covering 99.9% of exploration uncertainty), so the gains respond to typical behavior rather than a single noisy sample. The rewards and penalties are kept in separate channels before being combined, which is what allows the weighting to change across both episodes and timesteps.

For the quadruped, the team used the Sequential Motion Executor-Adaptive Gradient-weighting Online Learning (SME-AGOL) framework: a central pattern generator layer, a triangular basis layer, and an output layer mapping to eight joint position commands, with hip abduction/adduction joints fixed for future turning control. Mapping weights start at zero and are updated by the AGOL rule using trajectories of 8 episodes × 70 timesteps (about three gait cycles per episode) and returns averaged over a 20-timestep horizon. ROGER is layered on top as an add-on that adapts the reward weighting gains. The reward is forward velocity, with roll and pitch as penalties; thresholds are 0.2 rad (≈10°). Comparisons covered two fixed-weighting methods (fixed-gain penalty, fixed-gain CBF) and three adaptive methods (PDO, CRPO, OL-AUX) over 500 episodes per condition with more than 10 repetitions, with hyperparameters chosen by grid search.

For the hopper, a three-hidden-layer neural network with 256 hyperbolic tangent neurons per layer was trained with standard PPO under the default OpenAI Gymnasium setup, with the reward decomposed into forward velocity plus a healthy reward, an absolute torque penalty, and an absolute body orientation penalty (thresholds τ₁ = 1.0 for torque and τ₂ ≈ 10° for orientation). The physical quadruped experiment trained 300 episodes from a standing posture with all weights at zero, using an Intel RealSense T265 camera for reward and penalty estimation, a wheeled 15-kg support structure with slack ropes that did not assist balance, and three-second pauses between episodes for repositioning; the whole trial, including resets, lasted almost one hour.

Why This Matters

Impact on research. The work reframes reward-weight tuning as an online, state-dependent control problem rather than an offline hyperparameter search. It reports that ROGER matches carefully tuned fixed weights and CRPO on constraint satisfaction while exceeding them on task performance, and it provides partial stability results near the constraint thresholds and at convergence, with the expected primary reward guaranteed to increase (detailed in the appendix, which is truncated in the provided content). It also shows the approach can be applied to a DNN-PPO setup and outperformed the default reward function in the reported comparison.

Real-world applications (from the paper's experiments and framing):

• Quadruped inspection or monitoring robots operating on uneven industrial sites, gravel, and step fields without terrain sensing.
• Robots carrying payloads, since the dynamic-loading test added nearly 10% of the robot's weight and balance was maintained.
• Robots operating on slippery or low-friction surfaces such as lubricant-contaminated floors.
• On-robot, in-the-field learning where a fall is destructive and simulation-to-real transfer or fall-recovery resets are impractical.

Industry relevance. The paper argues that gain selection is currently time-consuming and risky, and that automating it lowers the barrier to deploying physical robots that learn in the real world. Because ROGER's hyperparameters are intuitive physical quantities (acceptable constraint thresholds and an uncertainty level), the process is less dependent on specialist RL tuning, and the reported 0.46 ± 0.09 ms overhead suggests the method is cheap enough to run alongside real-time control.

Future Directions

• Guaranteeing fall prevention under assumption violations. On the MuJoCo hopper, ROGER satisfied constraints in expectation but falls could not be prevented because the system dynamics are sensitive; making the guarantee hold in that regime is an open problem. The paper also reports failures when learning rates are excessive on the quadruped.
• Extending to more and higher-dimensional constraints. The framing references prior work using up to 16 reward terms with 11 distinct values, and the paper compares against CR-MOPO and other multi-constraint methods, suggesting multi-objective constraint handling as a natural extension.
• Enabling turning and richer behaviors. The hip abduction/adduction joints were fixed in all experiments and reserved for future turning control, so full 3D maneuvering remains to be demonstrated.
• Reducing the initial penalty-first phase. In the hopper, the first 50,000 timesteps were devoted to reducing constraints before performance increased, and stricter thresholds (0.5, 5°) delayed reward optimization further; shortening this delay without sacrificing safety is a practical question. The authors also note that results may be environment-specific because gains adapt through embodied interaction.
• Adding exteroceptive sensing. All the challenging-terrain behaviors emerged without terrain sensing, so integrating perception is an unexplored next step.

Target Audience

Robotics and machine learning researchers working on reinforcement learning for legged locomotion, safe or constrained RL, and sim-to-real transfer; engineers developing quadruped or other dynamic legged platforms who need training procedures that do not risk hardware; and practitioners who want to reduce reward-shaping and hyperparameter-tuning effort. Readers should be comfortable with RL terminology such as reward shaping, policy optimization, and constraint penalties; the paper's central mechanism, however, is understandable without deep RL background because it reduces to a ratio of current penalties to acceptable thresholds.

Authors’ abstract

Existing robot locomotion learning techniques rely heavily on the offline selection of proper reward weighting gains and cannot guarantee constraint satisfaction (i.e., constraint violation) during training. Thus, this work aims to address both issues by proposing Reward-Oriented Gains via Embodied Regulation (ROGER), which adapts reward-weighting gains online based on penalties received throughout the embodied interaction process. The ratio between the positive reward (primary reward) and negative reward (penalty) gains is automatically reduced as the learning approaches the constraint thresholds to avoid violation. Conversely, the ratio is increased when learning is in safe states to prioritize performance. With a 60-kg quadruped robot, ROGER achieved near-zero constraint violation throughout multiple learning trials. It also achieved up to 50% more primary reward than the equivalent state-of-the-art techniques. In MuJoCo continuous locomotion benchmarks, including a single-leg hopper, ROGER exhibited comparable or up to 100% higher performance and 60% less torque usage and orientation deviation compared to those trained with the default reward function. Finally, real-world locomotion learning of a physical quadruped robot was achieved from scratch within one hour without any falls. Therefore, this work contributes to constraint-satisfying real-world continual robot locomotion learning and simplifies reward weighting gain tuning, potentially facilitating the development of physical robots and those that learn in the real world.