DRAP: Dynamics-Risk Adaptive Plasticity for Continual Learning in Physical AI

AI summary

Overview

Research area: Machine learning — continual learning for vision-language-action (VLA) robot policies (physical AI / embodied control).

Technical level: Advanced. The paper combines a Lyapunov-inspired local dynamics analysis, tensor-level gradient statistics, and per-tensor AdamW learning-rate control inside a 7B-parameter OpenVLA policy.

Scope in one sentence: The paper proposes DRAP, a replay-free continual-learning method that weights the historical importance of model tensors by a locally estimated "trajectory-amplification risk" and uses that importance to scale each tensor's future AdamW learning rate, evaluated on a sequential LIBERO Spatial → Object → Goal protocol with OpenVLA.

What This Paper Is About

When a robot policy is fine-tuned on a new task, it tends to overwrite the behavior it learned for earlier tasks — the classic catastrophic-forgetting problem. In physical AI this is worse than in ordinary supervised learning, because a small action error pushes the robot into states it was never trained on, which produces larger errors, which can compound until the whole trajectory fails. The paper's goal is to keep a single VLA checkpoint able to perform earlier manipulation suites while still learning new ones, without storing or replaying any previous demonstrations, by making the tensors that mattered most at physically sensitive points of past trajectories less willing to change.

Key Contributions

1. A policy-independent dynamics-risk estimator. For sliding windows of expert state-action transitions within a single episode, the method fits a ridge-regularized local linear model of the form s̃_{j+1} = A_t s̃_j + B_t ã_j + c_t + ϵ_j, and uses the largest singular value σ_max(A_t) as a raw one-step perturbation-amplification score. This score is normalized against the median raw score over the suite's samples and clipped to the bounds [0.25, 4] (described in the paper as arbitrarily set for this experiment). It is explicitly framed as a relative sensitivity indicator, not a stability certificate.

2. Risk-weighted, sample-level tensor importance. After each task, the method runs a deterministic subset of N_k demonstration samples at batch size one (so gradients from different samples cannot cancel) and averages r_i² · ||∇_T ℓ_i(θ)||_F² / n_T over samples, where r_i is the precomputed risk weight and n_T is the number of scalars in tensor T. This yields one raw importance scalar per adaptable tensor.

3. Plasticity control through the optimizer, not a penalty. Cumulative importance is maintained as I_T^(k) = γ I_T^(k−1) + Ĩ_T,new^(k) (default γ = 1), normalized by a median-based operator across tensors, and converted into a multiplier s_T = (1 + β Ī_T)^(−1/p) that sets each tensor's effective AdamW learning rate η_T = η · s_T. No reference checkpoint is stored and no displacement penalty is applied, so protection is active from the first update of the next stage — the paper contrasts this directly with EWC in Table I.

4. A sequential, replay-free LIBERO evaluation with two ablations. DRAP is instantiated on a fixed KappaTune-selected subnetwork of OpenVLA (tensors chosen once by low regularized condition number within the vision, language, and action-head groups) and compared against standard sequential fine-tuning, KappaTune+EWC, DRAP with uniform risk (r_i = 1), and DRAP without KappaTune.

Main Findings

• Standard fine-tuning is strongly recency-dominated. After training on Goal, success on Spatial falls from 0.77 to 0.00 and on Object from 0.83 to 0.07; average success drops to 0.26 and average forgetting reaches 0.77.

• EWC gives only partial retention. KappaTune+EWC holds Object at 0.30 after Goal training but leaves Spatial at 0.03, for average success 0.32 and average forgetting 0.63. The paper reads this as Fisher-weighted anchoring helping short-term retention but being insufficient across the full sequence.

• KappaTune+DRAP gives the best stability–plasticity balance. It reaches average success 0.77 and Spatial forgetting 0.02 after Object training, and average success 0.55 with average forgetting 0.31 after Goal training — against 0.26 / 0.77 for standard fine-tuning and 0.32 / 0.63 for EWC. Its per-suite numbers after Goal are Spatial 0.34, Object 0.61, Goal 0.71.

• New-suite acquisition is not sacrificed. Goal-suite success after Goal training is 0.71 for KappaTune+DRAP, the same value reported for standard fine-tuning, and Object acquisition is 0.80 versus 0.83 for standard fine-tuning.

• KappaTune and DRAP are complementary. DRAP without KappaTune is competitive through the Object stage (0.78 Spatial, 0.71 Spatial retention, 0.82 Object) but collapses after Goal adaptation, falling to Spatial 0.11 and Object 0.08, close to the standard baseline, with average forgetting 0.71 versus 0.31 for the full method.

• The dynamics-risk signal adds a measurable but secondary gain. Uniform-risk DRAP (r_i = 1 for every sample, all other machinery retained) already reaches average success 0.72 / forgetting 0.14 after Object and 0.51 / 0.36 after Goal, so most of the benefit comes from historical importance and tensor-specific learning rates; risk weighting adds further retention (Spatial 0.34 vs 0.29, Object 0.61 vs 0.57 after Goal) without impairing acquisition.

• Stated limitation. ρ_t = σ_max(A_t) captures only the direct state-sensitivity term and omits the policy-feedback term B_t J_π,t in the first-order closed-loop Jacobian A_t + B_t J_π,t; even the full Jacobian would be a local approximation that ignores higher-order nonlinearities, contact-mode transitions, observation aliasing, and local-model estimation error.

• Benchmark context, not a like-for-like comparison. The paper notes that OpenVLA-OFT reports 97.1% average success with four independently fine-tuned policies and 96.8% with one policy jointly trained on all four suites — settings that evaluate suite-specific or joint acquisition rather than sequential replay-free learning.

Methodology in Plain English

The authors start from a physical intuition: some moments in a demonstration are "knife-edge" moments, where a tiny deviation sends the robot further off course, and parameters that produce correct actions at those moments deserve extra protection later.

To find those moments without asking the policy, they look only at the logged expert data. For each short window of a trajectory, they fit a simple linear model predicting the next robot state from the current state and the expert action. The largest singular value of the state-to-state part of that model says how much a small state error gets magnified in one step. Dividing by the median such value across the suite and clipping the result puts every sample on a comparable scale.

Training then proceeds suite by suite. After finishing a suite, they run through a subset of its samples one at a time, compute the gradient of the action-token loss for each adaptable tensor, square it, multiply by the squared risk weight for that sample, and average. Dividing by the tensor's parameter count stops big tensors from looking important merely by being big, and dividing by the median across tensors makes the scale comparable from task to task. This is added into a running memory per tensor.

Before the next suite, each tensor's accumulated score is turned into a shrinkage factor between 0 and 1 and used to set that tensor's AdamW learning rate. Highly important tensors take smaller optimizer steps from the very first update; unimportant ones stay flexible. Because the shrinkage is applied inside the optimizer rather than as an extra loss term, there is no stored copy of the old weights to compare against and nothing needs to be replayed.

The model itself is an OpenVLA policy in which only a fixed budget of low-condition-number tensors per architectural group (vision, language, action head) is trainable, chosen once by KappaTune; everything else stays frozen. Evaluation is closed-loop: after each suite, the single checkpoint is tested on every suite seen so far, pooling 6 Gym evaluation runs of 50 episodes each to reduce rollout noise.

Why This Matters

Impact on research. The paper connects continual learning to the control-theoretic question of where a trajectory is fragile, rather than treating every training sample as equally worth protecting. It also demonstrates a design pattern — importance-driven per-tensor learning rates rather than a quadratic penalty — that avoids storing reference weights, and it separates two distinct protective roles: architectural restriction (KappaTune) and within-subnetwork plasticity allocation (DRAP). Its negative result, that removing KappaTune destroys the benefit late in the sequence, is as informative as the positive one.

Real-world applications (settings implied by the paper's framing of sequential manipulation skill learning):

• Home or service robots that must pick up new household manipulation skills after deployment without losing older ones.
• Warehouse and logistics manipulators that are updated for new object categories while continuing to handle existing inventory.
• Manufacturing cells where a robot is retrained for new part geometries without revisiting prior demonstrations.
• Any deployed embodied system where storing or regenerating past sensor data is costly, restricted, or privacy-sensitive — the replay-free property is the operative constraint here.

Industry relevance. Teams fine-tuning large VLA policies face exactly this problem: full fine-tuning is destructive, replay buffers are operationally awkward for physical robots, and a policy that fails mid-trajectory is a safety and cost issue rather than a benchmark regression. A method that requires storing only a per-tensor importance vector plus a subnetwork mask, and that plugs into standard AdamW parameter groups, is comparatively cheap to deploy in an existing training stack.

Future Directions

• Extending the risk estimator to include the control pathway (the B_t J_π,t term) rather than the state-transition term alone, which the paper identifies as the main gap in the current score.
• Using multi-step perturbation propagation and contact-aware or learned local dynamics instead of a one-step linear estimate, to better capture contact-mode transitions and nonlinearities the current model ignores.
• Moving from offline, per-suite risk computation to online estimation, so risk scores need not be precomputed over a completed suite's demonstrations.
• Adding a CLASSP-inspired update threshold on top of the learning-rate scaling, and testing DRAP over longer continual-learning sequences than the three-suite Spatial → Object → Goal order used here.

Target Audience

Researchers and graduate students working on continual learning, robot learning, or VLA policy training; engineers responsible for fine-tuning large pretrained policies on deployed robotic systems; and readers interested in how control-theoretic ideas such as local linearization and Lyapunov-style analysis can be repurposed as practical training signals. Familiarity with AdamW, singular values, and standard continual-learning baselines (EWC, synaptic importance methods) will make the method section substantially easier to follow.

Authors’ abstract

Continual learning in physical AI differs from conventional continual learning because small per-step policy errors can accumulate over a trajectory. Each deviation may move the agent outside the training distribution, increase subsequent prediction errors, and ultimately cause closed-loop task failure. This work introduces Dynamics-Risk Adaptive Plasticity (DRAP), a replay-free method that estimates the historical importance of adaptable model tensors from sample-level action gradients weighted by local trajectory-amplification risk. A ridge-regularized local linear model is fitted to expert state-action transitions, and the largest singular value of its state-transition matrix provides a policy-independent, Lyapunov-inspired estimate of one-step perturbation amplification. After each task, risk-weighted squared gradients are accumulated in a tensor-level importance memory. During subsequent tasks, DRAP uses normalized cumulative importance to scale the effective AdamW learning rate of each tensor, immediately reducing the plasticity of components that supported previous behavior at dynamically sensitive trajectory points. We instantiate DRAP on a fixed KappaTune-selected subnetwork of OpenVLA and evaluate replay-free sequential adaptation across the LIBERO Spatial, Object, and Goal suites. Compared with standard AdamW fine-tuning and EWC, the results show substantially improved retention of previously learned behaviors while preserving strong new-suite acquisition.