Efficient Test-Time Adaptation For Robot Policies

AI summary

Overview

• Research area: Robotics / reinforcement learning — specifically test-time adaptation (TTA) for legged locomotion policies under deployment-time distribution shift.
• Technical level: Advanced. The paper assumes familiarity with actor–critic RL, distributional critics, entropy regularization, and prioritized experience replay.
• Scope: The paper proposes TEMPO, a lightweight on-device adaptation method for pretrained robot locomotion policies, and evaluates it in MuJoCo Playground simulation (Go1, Go2, G1) plus a physical Unitree Go2.

What This Paper Is About

Reinforcement-learning locomotion policies that work well in training can fail at deployment when conditions change — rough or slippery ground, added payload, or degraded sensing — and retraining on the robot is usually infeasible given on-device latency and memory limits. The paper asks whether a pretrained policy can adapt itself online, during execution, using no rewards and no labels, purely by leaning on the critic network it was trained with. It introduces TEMPO, which stabilizes this critic-driven self-supervised adaptation with an entropy regularizer on the critic's predictive distribution and a score-based memory that favors "hard" observations.

Key Contributions

1. A lightweight on-device TTA framework for RL control. Only the actor parameters are updated at test time while the critic is frozen; a compact memory of recent observations supports periodic, self-supervised updates designed to fit control-loop latency and compute budgets.
2. Entropy-regularized critic-driven adaptation. An objective that maximizes the critic's estimated action value while promoting higher-entropy critic outputs, intended to mitigate overconfident value extrapolation and preserve local action coverage. The procedure is described as architecture-agnostic and supervision-free.
3. Score-based memory with importance sampling. Each observation receives a score inversely proportional to the critic's predicted value (Eq. 3); harder samples are retained and importance sampling focuses adaptation compute on more informative samples under memory constraints.
4. Comprehensive validation in simulation and on-device. Consistent gains are reported on Go1/Go2/G1 embodiments across four deployment shifts, and the same trends are confirmed on a physical Unitree Go2 via fully on-device adaptation with a {3, 2} kg calf payload.

Main Findings

• Baseline TTA can recover part of the drop. The simple critic-guided baseline, QMax, which maximizes the frozen critic's predicted value on a FIFO buffer of recent observations, recovers performance across a wide range of shift severities but "often saturates under harder shifts."
• TEMPO is best across shift families and severities. In Figure 2 (Go2, episodic return top row, average forward linear velocity bottom row), TEMPO is reported as consistently best across slippery terrain, calf mass, rough terrain, and torso mass, with the clearest advantages in the most challenging regimes. The gains are mirrored in forward velocity, which the authors interpret as better stabilized behavior rather than reward-only artifacts. Results are averaged across six seeds.
• Off-the-shelf TTA and prior RL adaptation baselines fail. The paper states that standard TTA baselines such as Tent and RL-based alternatives such as OPEX are ineffective in this setting (Figure 1, middle), failing under a rocky-terrain shift. QMax is described as "pretty effective" in recovering a significant portion of the performance drop.
• Reported simulation gains. The abstract reports "up to 80% improvement in a single-device setting"; the introduction states "up to 50% performance gains very efficiently in a single-device setting."
• Real-world result on Unitree Go2. With weight plates mounted on both front left and rear left calves via a custom 3D-printed rigid mount, adaptation was performed once on-device for 4 minutes. Across the tested commands, the mean score improved from 89.8 to 98.7. Each evaluation consisted of 5 trials spanning a 3 second window (150 steps each), with rewards r_t = exp(−‖v^cmd_xy − v^loc_xy‖²) capped at a maximum possible value of 150. Per-command scores (No adaptation → TEMPO): (v_x=0.5, w_z=0.0) 93.2 ± 1.3 → 95.2 ± 0.6; (v_x=−0.5, w_z=0.0) 98.7 ± 1.6 → 113.5 ± 0.7; (v_x=−0.7, w_z=0.0) 88.1 ± 1.1 → 96.5 ± 1.0; (v_x=0.7, w_z=0.6) 88.4 ± 3.3 → 101.6 ± 1.4; (v_x=0.7, w_z=−0.6) 80.5 ± 1.0 → 86.9 ± 1.2. The largest improvement is reported for walking backwards at medium speed (v_x = −0.5), a 15% increase from 98.7 to 113.5.
• Mixed corruptions. Across all 8 mixed conditions (calf mass 2 kg or torso mass 5 kg combined with slippery or rough terrain at two severities), TEMPO is reported as best. Averaged over the table, reward improves from 11.18 (No Adapt) and 12.48 (QMax) to 14.09 — a gain of +2.91 (+26.1%) over No Adapt and +1.61 (+12.9%) over QMax. For the harder torso-payload compositions, the average rises from 8.23 to 12.33 (+4.10, +49.8%) versus No Adapt, and from 10.00 to 12.33 versus QMax (+2.33, +23.3%). The largest single-case improvement is torso mass with rocky terrain at the highest severity: TEMPO reaches 14.6 versus 10.5 for QMax (+4.1, +39.0%) and 7.8 without adaptation (+6.8, +87.2%).
• Continual TTA. Under a non-stationary stream of progressively increasing shift severity, TEMPO recovers faster after the shift intensifies and reaches a higher performance plateau than QMax; QMax improves initially but saturates earlier.
• No observable forgetting on the source domain. When the source domain (flat terrain without shift) is appended at the end of the continual sequence, both TEMPO and QMax recover source-domain performance "instantly," matching the original policy without transient degradation or extra adaptation.
• Ablation. Starting from QMax, adding entropy regularization yields consistent gains of +0.5 to +1.6 return across six settings; replacing FIFO with score-based memory gives larger improvements of +1.0 to +2.3; combining both (TEMPO) gives the best result in every column, improving over QMax by +1.4 to +2.5 and over No Adapt by +2.6 to +4.9. The authors conclude that memory design can play a crucial role in adaptation performance.
• TEMPO complements RMA. Native RMA adaptation improves over a frozen RMA baseline, and RMA + TEMPO yields the best performance across the evaluated shift severities, indicating TEMPO adds to rather than replaces RMA's built-in adaptation.

Methodology in Plain English

The setup is a standard actor–critic pair trained on flat terrain without domain randomization, then deployed in a single fixed shifted environment. Four shift types are used, parameterized by a severity scalar δ: rough terrain (height variation δ), calf mass perturbation (δ kg on one calf), torso mass perturbation (δ kg on the torso), and slippery terrain (ground friction set to 1 − δ). Unless stated otherwise, each test episode contains a single shift type.

The key move is that instead of discarding the critic at deployment, the method keeps it — frozen — and treats its predicted action value as a stand-in for the performance signal that would normally come from rewards. Only the actor is updated.

• QMax baseline: maintain a bounded FIFO buffer of the most recent observations (append the newest, evict the oldest when the buffer exceeds capacity N). Every K steps, sample a minibatch and take a gradient step that increases the critic's value of the actions the policy would take on those observations. The critic stays frozen.
• TEMPO adds entropy: assuming the pretraining produced a distributional critic (a distribution over a discretized value support whose expectation is the action value), the loss adds a term that rewards high entropy in that distribution, weighted by λ. This keeps updates from chasing overly peaky, overconfident value estimates.
• TEMPO replaces FIFO with a scored memory: each observation gets a score inversely proportional to its predicted value, so low-value ("hard") observations score highly. New observations are inserted only if they beat the lowest score currently stored, evicting the worst. Batches are then drawn without replacement with probability proportional to shifted scores, so limited adaptation compute concentrates on the states most associated with degraded behavior.

Implementation details: Adam with learning rate 5 × 10⁻⁶; all learnable parameters of the actor adapted while the critic(s) are frozen; replay buffer size |B| = 16k; batch size B = 2k; adaptation period K = 128 environment steps with a single optimizer step each time; TEMPO entropy weight λ = 0.5. With 49-dimensional observations and |B| = 16k, the FIFO buffer footprint is about 3 MiB and score-based variants about 3.05 MiB. The default configuration for experiments is Fast-TD3 architecture, Go2 environment, flat terrain, no domain randomization.

Evaluated embodiments are Go1-joystick, Go2-joystick, and G1-joystick from MuJoCo Playground, with three pretrained policy families: Fast-TD3, SimbaV2, and RMA. The observation vector combines proprioceptive and command signals: IMU measurements (gyro and a gravity-aligned vector), base linear velocity, joint angles relative to a default pose, joint velocities, and the previous action.

Why This Matters

Impact on research. The paper argues that deployment-time adaptation is a distinct problem from domain randomization and architecture add-ons: it neither requires enumerating variability in advance nor increases system complexity, and it fits strict on-device budgets. It also broadens TTA — largely developed on perception benchmarks under synthetic corruptions — into closed-loop control, where the test stream is highly correlated, on-policy, and must preserve stability. The finding that a frozen, shifted critic still supplies a useful learning signal, and that entropy regularization and prioritized memory each add measurable value, gives the TTA and RL communities a reusable recipe.

Real-world applications (as motivated or demonstrated by the paper):

• Legged robots walking on rocky or low-friction surfaces where contact dynamics differ from training.
• Robots carrying localized payloads, such as a weight attached to a calf or torso after deployment.
• Recovery from emergent situations such as a mass stuck on a limb, where the shift is abrupt.
• Bridging the sim-to-real gap, where a policy trained in simulation is deployed on physical hardware under conditions not modeled in training.

Industry relevance. The work comes from Qualcomm AI Research, and the framing is explicitly on-device: a single-device setting, tight latency and memory budgets, fully on-device adaptation carried out once for 4 minutes on a physical Unitree Go2, and a memory footprint measured in a few MiB. That positions the method for embedded robot controllers rather than cloud retraining pipelines. The demonstration that TEMPO stacks on top of RMA — an existing adaptation-aware backbone — suggests it can be added to deployed stacks rather than requiring a new architecture.

Future Directions

• Extending beyond locomotion. The evaluation covers quadruped (Go1, Go2) and humanoid (G1) locomotion in MuJoCo Playground; whether the same critic-guided scheme transfers to manipulation or other embodied tasks is untested here.
• Richer and longer-horizon non-stationarity. Continual TTA is evaluated with progressively increasing shift severity; other forms of drift (e.g., slowly varying payloads or hardware wear) are not reported.
• Broader real-world validation. The hardware study uses a single physical Unitree Go2 with a calf payload and 5 trials per command over a 3 second window; larger-scale, longer-duration, and multi-shift physical evaluations are not reported.
• Tuning and robustness of the new hyperparameters. Entropy weight λ = 0.5, buffer size 16k, batch size 2k, and adaptation period K = 128 are fixed in the main protocol; the paper points to Appendix B for robustness analyses of TEMPO, but sensitivity of these settings is not the focus of the main text.

Target Audience

Researchers and engineers working on robot learning, especially those deploying RL locomotion policies on real hardware under compute and memory constraints. It is also relevant to the test-time adaptation and continual learning communities looking for a closed-loop, on-policy application domain, and to practitioners who want a lightweight adaptation layer that can sit on top of an existing actor–critic policy such as RMA without retraining.

Authors’ abstract

Reinforcement-learning policies for robot locomotion can fail at deployment when operating conditions shift (e.g., rough or slippery terrain, payload changes, sensing degradation), and retraining or manual recalibration is often infeasible under on-device latency and memory constraints. We study test-time adaptation (TTA) for robotic control: adapting a pretrained policy online and in a self-supervised manner during execution, without access to rewards. Our starting point is a lightweight baseline that updates only the actor using a frozen pretrained critic as a deployment-time performance surrogate. Building on this, we introduce TEMPO, a test-time entropy-regularized, memory-guided policy optimization method that stabilizes critic-driven updates under severe shift. TEMPO combines (i) an entropy-promoting regularizer on the critic's predictive distribution to mitigate overconfident extrapolation, and (ii) a score-based memory, inspired by prioritized replay, that prioritizes low-value ("hard") observations and applies importance sampling to focus limited computation on the most informative deployment conditions. Across Go1, Go2, and G1 in MuJoCo Playground under four representative shifts and multiple architectures, TEMPO yields consistent gains, reaching up to 80% improvement in a single-device setting. We further validate on a physical Unitree Go2 with an added calf payload, where fully on-device adaptation improves performance over the non-adapted baseline across real-world episodes.