ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models

AI summary

Overview

Research area: Continual reinforcement learning for robot foundation models (robot learning, off-policy RL post-training, manipulation).

Technical level: Advanced. The paper assumes familiarity with actor–critic RL, off-policy replay, vision-language-action (VLA) and world action models (WAMs), and continual-learning metrics such as forward and backward transfer.

Scope: The paper introduces ACCRUE, a framework that carries a lightweight actor and critic across sequentially arriving robot tasks so that each new assembly task is learned with fewer real-robot interactions while prior task behavior is retained.

What This Paper Is About

Robot foundation models generalize broadly but still fail on tasks and conditions outside their pretraining data, and reinforcement learning (RL) post-training can close that gap. Prior RL post-training work mostly treats this as a one-time procedure for a single task, so a deployed robot must pay the full interaction cost again for every new task instead of reusing what earlier RL stages taught it. ACCRUE's goal is continual post-training that reduces the interaction needed for each new task without sacrificing behaviors acquired from previous tasks.

Key Contributions

1. A continual off-policy RL framework (ACCRUE) that inherits a lightweight actor and critic across tasks, transferring a behavioral initialization through the actor and a value initialization through the critic, so each stage amortizes prior RL experience.

2. An asymmetric update rule for limited replay. The inherited critic is adapted only on current-task transitions, and the actor is trained on current-task data with the critic while prior behavior is preserved by distilling the actor toward cached outputs of the preceding-stage actor on the last N episodes retained from each completed task. Prior-task replay is deliberately excluded from critic updates and from the actor's RL objective.

3. A stable input interface for carryover: a fixed off-the-shelf visual encoder (DINOv3) plus robot proprioception forms the actor–critic state, so stored transitions keep the same representation even as the base policy is repeatedly fine-tuned on accumulated demonstrations. Both networks are conditioned on instruction-conditioned reference action chunks from the base policy, with reference action dropout applied during training.

4. Validation across simulation and real hardware with different base policy families (GR00T N1.7 and π0.5), different off-policy algorithms (FastSAC and FastTD3), and different interaction budgets, on contact-rich assembly tasks.

Main Findings

• Simulation improvement over the strongest off-policy continual baseline: Across FORGE and AutoMate, ACCRUE improves continual AUC by up to 21.0% and reduces negative backward transfer by up to 70.0%.

• FORGE (4 task orderings, 4 seeds): ACCRUE reaches NBT 0.065, FWT 0.823, AUC 0.869, final accuracy 0.917, and final forgetting 0.068. Among matched shared actor–critic methods, the comparison values are Sequential (0.427, 0.819, 0.718, 0.685, 0.409), EWC (0.217, 0.593, 0.582, 0.603, 0.224), RETAIN (0.514, 0.802, 0.682, 0.644, 0.474), and ER (0.518, 0.833, 0.697, 0.626, 0.516). Reference methods: π_base final accuracy 0.687, Single-task RL FWT 0.741, Simple Recipe (0.038, 0.586, 0.583, 0.594, 0.061), and the privileged Isolated reference FWT 0.816, AUC 0.868, final accuracy 0.919.

• AutoMate (4 asset sequences, 4 seeds): ACCRUE reaches NBT 0.047, FWT 0.848, AUC 0.858, final accuracy 0.863, and final forgetting 0.054. Comparison values are Sequential (0.178, 0.824, 0.792, 0.765, 0.175), EWC (0.075, 0.704, 0.704, 0.699, 0.067), RETAIN (0.191, 0.822, 0.784, 0.750, 0.192), and ER (0.167, 0.834, 0.799, 0.779, 0.176). Reference methods: π_base 0.757, Single-task RL FWT 0.807, Simple Recipe (0.016, 0.730, 0.726, 0.730, 0.019), Isolated FWT 0.826, AUC 0.858, final accuracy 0.879.

• ACCRUE exceeds Single-task RL in forward transfer under the same stage-wise interaction budget on both benchmarks, and on AutoMate it approaches the privileged Isolated reference while using a single shared actor–critic pair.

• Real-robot time savings: On the DROID platform, inheriting the actor and critic reduces robot RL interaction time for subsequent tasks by 31.0% to 68.5% compared with training a new actor–critic for each task, and retention stays essentially complete — every earlier task succeeds on at least 18 of 20 evaluation trials, 19.4 on average, and approximately 97% average success on prior tasks across the two multi-stage sequences.

• Both robot effects exceed the simulation effects, which the authors attribute to the far smaller interaction budget on hardware: a better starting point is worth more when a stage is measured in tens of minutes of robot time.

• Critic inheritance supports forward transfer: Resetting the critic at each stage produced modestly more forgetting and lower task-learning efficiency on FORGE sequence S1 (NBT 0.042, FWT 0.841, AUC 0.890, final accuracy 0.942, final forgetting 0.054), versus ACCRUE at (0.028, 0.854, 0.906, 0.961, 0.037).

• Restricting critic updates to current-task data helps: updating the critic with prior-task replay combined with current-task replay raised forgetting and lowered continual AUC and final success (0.037, 0.842, 0.888, 0.925, 0.067) compared with ACCRUE.

• Fixed visual features matter: swapping the fixed DINOv3 representation for features from the stage-updated GR00T VLM worsened retention and forward transfer (0.071, 0.814, 0.846, 0.883, 0.113). The authors note this comparison also changes the encoder itself and therefore does not isolate representation drift.

• Reference-action conditioning matters: removing the reference input, π_θ(s, ã) → π_θ(s), lowered retention and transfer (0.035, 0.807, 0.869, 0.936, 0.050), and removing the reference-deviation penalty (β = 0.0) had little effect on backward retention but substantially weakened forward transfer and task learning (0.029, 0.763, 0.821, 0.942, 0.037).

• Retained-episode budget is non-monotonic: On FORGE sequence S1, N = 100 retained episodes per completed task gave the best joint retention–learning trade-off (NBT 0.040, FWT 0.854, AUC 0.896, final accuracy 0.925, final forgetting 0.062). N = 10 (0.107, 0.853, 0.854, 0.847, 0.192) and N = 20 (0.093, 0.844, 0.859, 0.869, 0.167) were insufficient, while N = 200 degraded performance (0.043, 0.842, 0.879, 0.903, 0.083). The 100–200 episode range corresponds to approximately 3% of the total replay buffer.

• Hyperparameters were tuned per setting: λ = 1.0 for the more diverse real-world NIST sequence and λ = 0.1 for the more closely related real-world AutoMate assemblies; β = 0.1 in simulation and 10.0 on hardware.

Methodology in Plain English

The robot faces a sequence of K tasks, one per stage, with known task boundaries and identities. At each stage a small set of expert demonstrations (1–50 trajectories in the experiments) arrives, and the base robot foundation model is fine-tuned on all demonstrations seen so far, then frozen during that stage's RL.

On top of the frozen base model, a small actor and critic learn in action-chunk space, using state built from a frozen external visual encoder plus proprioception, and conditioned on the action chunk the base policy proposes. Because this visual encoder never changes, transitions stored at earlier stages remain usable later, even though the base policy itself keeps being updated.

At each new stage, the actor and critic start from the previous stage's checkpoint rather than from random weights. The critic is updated only with transitions gathered on the current task, deliberately ignoring the small prior-task memory, because that memory is too small to give reliable value estimates for older tasks. Instead of using the critic to protect old behavior, the method keeps the last N training episodes of each finished task, uses the previous stage's actor to compute target actions on those episodes once, caches them, and then trains the current actor to match those cached targets while also learning the new task. This separates "learn the new task" (critic-driven RL on current-task data) from "do not forget the old tasks" (distillation on retained episodes). In simulation the setup uses GR00T N1.7, FastSAC, chunks of length H = 16, 64 parallel environments, a 500K environment-step budget per stage with evaluation every 50K steps, and γ = 0.997. On the real DROID platform it uses π0.5 with LoRA fine-tuning, FastTD3, chunks of length H = 10, and an asynchronous learner and actor process.

Why This Matters

Impact on research. The paper argues that continual post-training for robot foundation models should be treated as a sequence of RL stages rather than isolated single-task runs, and it shows that value knowledge acquired through interaction — not just policy parameters or demonstrations — can be carried forward. It also argues that the natural instinct to replay all prior data through the critic is counterproductive when the retained memory is small, which reframes how forgetting prevention should be split between critic updates and actor distillation.

Real-world applications:

• Manufacturing and assembly lines where new parts and product variants arrive over time and the robot must adapt its contact behavior while keeping skills for previously encountered assemblies.
• Electronics and precision insertion work, where contact-rich behaviors such as peg insertion, gear meshing, and nut threading must be retained as new components are introduced.
• Warehouse and logistics robots that must pick up new object families without losing competence on the existing catalog.
• Deployed robot fleets that need to be improved repeatedly after shipping, using a handful of demonstrations plus on-robot interaction rather than full retraining.

Industry relevance. The reported 31.0%–68.5% reduction in robot RL interaction time for subsequent tasks is framed as the difference between a training stage that fits inside a single session and one that does not. The method only needs access to the base policy's input–output interface, uses a frozen off-the-shelf encoder, and works with different base policy families and different off-policy RL algorithms, which lowers the barrier to adopting it on top of an existing foundation-model stack.

Future Directions

• Removing the assumption of known task boundaries and identities. The current setup is task-incremental, with boundaries and identities known and each stage's language instruction supplied; handling unknown boundaries or task inference is left open.
• Scaling past short sequences. Experiments cover three-stage (FORGE, real NIST) and five-stage (AutoMate simulation and real) sequences; behavior at longer horizons and with more accumulated replay is not reported.
• Reducing dependence on demonstrations. Each stage still requires 1–50 expert trajectories plus LoRA or full fine-tuning of the base policy; the effect of fewer demonstrations is examined only in the ablation section, and the truncated Table 6 on demonstration retention is not fully reported here.
• Better value knowledge under constrained replay. The non-monotonic N result and the finding that prior-task critic updates hurt suggest that reliably reusing retained prior data for value estimation, or automatically setting the distillation weight λ, remains an open question. A full baseline sweep on hardware is also noted as infeasible at the scale tested, which leaves open comparisons against a broader set of continual baselines on real robots.

Target Audience

Robotics and reinforcement learning researchers working on continual learning, RL post-training, and robot foundation models; engineers at robotics or manufacturing companies deploying manipulation policies that must be updated repeatedly after deployment; and graduate students familiar with actor–critic methods who want a concrete example of how off-policy replay, distillation, and foundation-model fine-tuning can be combined into a persistent learner across tasks.

Authors’ abstract

Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors. Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks. We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL. The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes. Fixed visual features keep the inherited learner and its replay compatible with base-policy updates. Across contact-rich assembly sequences in simulation and on real-robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-robot training time by 31.0\% to 68.5\%.