Continual Knowledge Adaptation for Reinforcement Learning

AI summary

Overview

Research area: Continual Reinforcement Learning (CRL), specifically knowledge transfer and catastrophic forgetting mitigation across sequential tasks.

Technical level: Intermediate. The core idea (adding task-specific offset vectors to a frozen base policy) is intuitive, but full appreciation requires familiarity with RL terminology (MDPs, policies, PPO/SAC) and continual learning evaluation metrics.

Scope: The paper introduces CKA-RL, a method that stores task knowledge as additive vectors, adaptively weights them when learning new tasks, and merges redundant vectors to bound memory growth.

What This Paper Is About

Reinforcement learning agents typically assume a fixed environment, but real-world settings change constantly, forcing agents to learn a stream of tasks in sequence. Standard continual RL methods either overwrite earlier knowledge (catastrophic forgetting) or accumulate per-task models that inflate memory and inference cost. This paper proposes a compact alternative: keep one frozen base policy, represent each task as a small offset vector, and blend those vectors dynamically when facing a new task.

Key Contributions

1. CKA-RL framework: A continual reinforcement learning method that accumulates historical task knowledge as additive vectors and reuses it to accelerate new task learning while reducing performance loss on old tasks.

2. Continual Knowledge Adaptation strategy: A mechanism that maintains a pool of task-specific knowledge vectors and learns softmax-normalized adaptation weights that determine how much each historical vector contributes when constructing the policy for a new task. The base parameter stays frozen; only the new task vector and the weighting parameters are optimized.

3. Adaptive Knowledge Merging: A scalability mechanism that computes pairwise cosine similarity across stored vectors and, when the pool exceeds a capacity limit, merges the two most similar vectors by averaging them. This bounds memory and parameter growth without discarding essential task knowledge.

4. Empirical validation across three benchmarks: Comparisons against nine state-of-the-art CRL methods (Baseline, FT-1, FT-N, ProgNet, PackNet, MaskNet, CReLUs, CompoNet, CbpNet) on Meta-World, Freeway, and SpaceInvaders, with reported improvements of 4.20% in average performance and 8.02% in forward transfer over the best prior method.

Main Findings

• Overall performance superiority: Averaged across three environments, CKA-RL reaches 0.7498 performance versus CompoNet's 0.7196 (+4.20%) and 0.5049 forward transfer versus CompoNet's 0.4674 (+8.02%).

• Consistent gains per environment: Meta-World shows a 6.27% performance improvement over CbpNet (0.4368 to 0.4642); SpaceInvaders improves 1.02% over CompoNet (0.9828 to 0.9928); Freeway improves 1.12% over CReLUs (0.7835 to 0.7923).

• Strong forward transfer: In Meta-World, CKA-RL improves forward transfer by 41.82% over CompoNet; in SpaceInvaders by 11.29% (0.6963 to 0.7749); in Freeway by 1.73% over CReLUs.

• Better plasticity than dedicated plasticity methods: Against CbpNet in SpaceInvaders, CKA-RL gains 18.30% in average performance (0.8392 to 0.9928) and 59.97% in forward transfer (0.4844 to 0.7749).

• Ablation confirms each component matters: On Freeway, naive averaging of knowledge vectors actually hurts (0.7437 versus baseline 0.7532), while Continual Knowledge Adaptation alone raises performance to 0.7821 and forward transfer to 0.7321. Adding Adaptive Knowledge Merging pushes these to 0.7923 and 0.7429 while cutting memory from N vectors to 5.

• Pool size trade-off: Performance peaks at K_max = 5 for Freeway and SpaceInvaders, and K_max = 8 for Meta-World. Too small a pool discards useful knowledge; too large a pool introduces redundant or conflicting vectors that slow adaptation.

• Constant memory and inference cost: Merging keeps parameter memory nearly flat beyond the fifth task, whereas competing methods grow linearly. Inference latency stays essentially constant at roughly 0.0012s per input, while CompoNet and ProgNet slow down as tasks accumulate.

• Strong final consolidated policy: Evaluated with the final policy on all previously seen tasks, CKA-RL scores 0.3966 average performance, edging CbpNet (0.3933), FT-N (0.3733), and CReLUs (0.3666).

• Faster convergence in practice: Reward curves show CKA-RL starting from a higher initial reward and converging more quickly than baselines, indicating effective reuse of historical vectors.

Methodology in Plain English

The researchers start by training an agent on the first task and freezing those weights as a shared "base" policy. Every subsequent task gets its own small offset vector added on top of the base. When a new task arrives, rather than using all previous task vectors equally, the agent learns weights (through a softmax over learnable parameters) that decide how much each past vector should contribute. The new task's own vector starts at zero, so the policy initially relies on adapted historical knowledge and gradually develops its own specialized offset. The base stays frozen throughout, which protects general-purpose features from drift.

To prevent the vector pool from growing without bound, the method measures cosine similarity between all stored vectors once the pool exceeds a cap. The two most similar vectors get averaged into one, freeing a slot. This assumes similar vectors encode compatible knowledge, so averaging is a safe consolidation. The result is a system whose memory and inference cost stay roughly flat as more tasks are learned, unlike modular architectures that add parameters or modules per task.

Why This Matters

Impact on research: The paper reframes continual RL as a problem of knowledge composition rather than parameter isolation or replay. It connects model-editing concepts from supervised fine-tuning (task arithmetic, additive offsets) to reinforcement learning, showing that the same linear-combination properties hold for policies. This opens a path toward continual learning systems whose cost does not scale with task count—an issue that has limited deployment of methods like PackNet and CompoNet.

Real-world applications:

• Robotics: A manipulator that must learn new assembly or pick-and-place skills without forgetting earlier ones, running on fixed onboard memory.
• Autonomous driving: Vehicles adapting to new cities, weather conditions, or traffic rules while retaining prior driving competence.
• Game-playing and simulation agents: NPCs or training environments that continuously adopt new scenarios without retraining from scratch.
• Industrial control and process optimization: Systems tracking shifting operating conditions (e.g., equipment degradation, changing inputs) where constant inference latency is required.

Industry relevance: The near-constant memory footprint and stable inference latency directly address deployment constraints on edge hardware. A method that avoids per-task parameter growth is easier to certify, monitor, and ship than one whose cost grows with the number of environments encountered.

Future Directions

• Scaling to heterogeneous tasks: The current formulation assumes shared observation and action spaces across tasks; extending to tasks with differing action dimensions or sensor modalities remains open.
• Smarter merging criteria: Averaging the most similar vectors is simple, but it is unclear whether task importance, recency, or performance impact should influence which vectors are merged or how they are combined.
• Better conflict handling: When tasks encode contradictory behaviors, merging similar vectors may not be enough. Explicit mechanisms for detecting and resolving conflicts—rather than relying on softmax weighting to down-weight them—could improve robustness.
• Automatic pool sizing: K_max is a hyper-parameter tuned per benchmark. Learning it adaptively from task similarity or performance signals would make the method more practical.
• Theory-to-practice gap: The paper includes a mathematical analysis in its appendix; empirical testing on longer task sequences and more diverse domains would clarify how well the assumptions behind linear vector combination hold in practice.

Target Audience

This paper is most useful for reinforcement learning researchers and graduate students working on continual learning, transfer learning, or multi-task agents. Practitioners building deployed RL systems with memory or latency constraints will also benefit, particularly those familiar with PPO or SAC who want a drop-in mechanism for sequential task training. Readers without a background in RL will find the high-level idea accessible but may need supplementary reading to follow the experimental setup and metrics.

Authors’ abstract

Reinforcement Learning enables agents to learn optimal behaviors through interactions with environments. However, real-world environments are typically non-stationary, requiring agents to continuously adapt to new tasks and changing conditions. Although Continual Reinforcement Learning facilitates learning across multiple tasks, existing methods often suffer from catastrophic forgetting and inefficient knowledge utilization. To address these challenges, we propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowledge. Specifically, we introduce a Continual Knowledge Adaptation strategy, which involves maintaining a task-specific knowledge vector pool and dynamically using historical knowledge to adapt the agent to new tasks. This process mitigates catastrophic forgetting and enables efficient knowledge transfer across tasks by preserving and adapting critical model parameters. Additionally, we propose an Adaptive Knowledge Merging mechanism that combines similar knowledge vectors to address scalability challenges, reducing memory requirements while ensuring the retention of essential knowledge. Experiments on three benchmarks demonstrate that the proposed CKA-RL outperforms state-of-the-art methods, achieving an improvement of 4.20% in overall performance and 8.02% in forward transfer. The source code is available at https://github.com/Fhujinwu/CKA-RL.