Permanent and Transient Representations for Continual Reinforcement Learning

AI summary

Overview

Research area: Continual Reinforcement Learning (CRL), specifically the stability–plasticity trade-off and complementary learning systems.

Technical level: Advanced. The paper combines two-timescale stochastic approximation theory (linear function approximation convergence proofs) with a non-parametric, MinHash-based memory architecture, and evaluates it on deep RL benchmarks.

Scope (1 sentence): The paper extends the permanent–transient value function decomposition to use separate feature representations for each component, proves convergence for the linear case with distinct feature spaces, and introduces a hashed non-parametric transient approximator that scales the approach to image-based tasks and Craftax-Classic.

What This Paper Is About

Continual RL agents must keep adapting to changing situations without forgetting what they already know, which is the stability–plasticity trade-off. Prior work by Anand & Precup (2023) split the value function into a slow "permanent" component and a fast "transient" component, but used the same features in both, which limits how complementary the two systems can be. This paper argues that the permanent system should be parametric (for broad generalization) while the transient system should be non-parametric (for situation-specific, rapidly reversible corrections), and tests that idea theoretically and empirically.

Key Contributions

1. Separate feature spaces in the PT framework. The permanent–transient value function decomposition is extended so the two approximators can use different features, with convergence guarantees under linear function approximation derived via a two-timescale proof technique.

2. A new non-parametric transient approximator. A hashing-based, slot-table architecture (inspired by tile coding and CMACs) that operates directly on raw observations, enables rapid online corrections, and controls generalization through a tunable weight ρ — described as mirroring the Dentate Gyrus's role in fast, specific encoding without catastrophic interference.

3. Theory showing distinct features do not cause divergence. The main theorem establishes that permanent parameters θ* and transient parameters w*_τ converge to a unique fixed point under Assumptions 5.1–5.4, using distinct feature matrices Φ and Z. A corollary shows that when Z = Φ in a single-task setting, θ* equals the TD fixed point and w*_τ = 0 — i.e., the transient component collapses to zero, motivating separate features.

4. Large-scale empirical validation. The approach is tested on small illustrative prediction/control tasks, an image-based CIFAR-10 MDP, and Craftax-Classic at both 18M and 250M step scales, compared against parameter-level stability–plasticity methods (EWC, continual backprop) and other decompositions (NEC, PT-NN, PT-IHT, T-Only).

Main Findings

• Separate features help in prediction. In the 1D chain task, the "separate" PT-TD variant (permanent uses first-order Fourier features only) converged to near-zero RMSVE on other tasks after one task, while the "full" variant oscillated because its permanent weights absorbed task-specific variation. TD learning with only one feature order had large errors, confirming both orders are needed for accurate value estimation.

• Separate features help in control. In the 2D grid prediction task, PT-TD-Separate had the lowest overall RMSVE, with first-order features giving sufficient expressivity at lower variance for the permanent component. In the 7-state chain control task, PTQ-Separate was the only method to learn meaningful behaviour within the first 50 episodes and re-adjusted within five episodes after each task change; the reset Q-learning variant performed worst.

• The transient approximator drives fast adaptation. On the image task, the non-parametric transient component enabled precise, low-variance corrections; PT-NN and T-Only failed (instability and insufficient capacity respectively), PT-IHT degraded from hash collisions and over-generalization, and NEC performed poorly due to dependence on learned neural features.

• Mechanistic interventions did not produce rapid adaptation. Augmenting PQN with EWC or CB did not change performance on the image task, and the indifference between EWC and CB on Craftax suggests these methods preserve stability and plasticity but do not enable the fast adaptation CRL requires.

• Beats task-boundary-aware methods in adaptation. In the adaptation experiment (train 250k steps on one task, evaluate over the next 50k), the approach adapted faster than CompoNet, Reset(Final), and Reset(Full), despite those methods using task-boundary information and more parameters.

• Craftax-Classic scaling. On the 250M-step benchmark with 1024 parallel environments, the method surpassed a return of 10 in under 50M steps (twice as fast as PQN(32)) and reached a return of 15 within 150M steps, whereas PQN(32) required over 200M steps. It also learned successfully in the two-stream setting and outperformed all baselines.

• Theory confirms stability with distinct features. The two-timescale analysis shows the permanent estimates appear stationary while transient values update, and vice versa; substituting the transient fixed point into the permanent updates satisfies the conditions for convergence.

Methodology in Plain English

The researchers start from a simple decomposition: the value of a state (or state–action pair) is the sum of a slow, general estimate and a fast, situation-specific correction, Q^(PT) = Q^(T)_w + Q^(P)_θ. The permanent part is updated slowly (every k timesteps or at task boundaries) toward the full estimate; the transient part is updated every step on the TD error and is decayed or reset after each permanent update to preserve plasticity.

Their change is to give each part its own features. The permanent part uses parametric features — Fourier bases in the toy problems, neural networks in the large-scale ones — chosen to capture stationary structure. The transient part uses a new non-parametric memory with four stages: (1) tokenize the observation into a set of discrete symbols (object type at each grid position, or quantized grayscale pixels at 64 levels for images); (2) hash the set with several MinHash functions, which preserves Jaccard similarity so similar observations get similar signatures; (3) split the signature into subsequences that index into bins of a transient table, each bin holding S slots with least-recently-used eviction and a stored tag; (4) estimate the value as a convex combination of the focal slot (weight ρ) and its neighbours (weight (1−ρ)/(S−1)), summed across all tables. The TD error is distributed across tables and slots in proportion to their contributions.

For theory, they use Borkar's two-timescale stochastic approximation framework with standard assumptions (step-size conditions, full-rank feature matrices, i.i.d. task sampling with observable boundaries, and rapidly mixing Markov chains), proving the transient updates converge with the permanent values held fixed, then substituting that fixed point into the permanent updates.

Experiments run from controlled linear-approximation prediction and control problems (with known task boundaries, i.e. semi-CRL) up to fully hidden-boundary image tasks and Craftax-Classic, where boundaries are not given to the agent.

Why This Matters

Impact on research. The paper supplies the first convergence guarantees for the permanent–transient decomposition with distinct feature spaces, upgrading earlier tabular results and showing that using two feature sets does not cause the divergence common in linear RL. It also offers a concrete architectural alternative to parameter-level fixes (regularization, specialized optimizers), arguing that the stability–plasticity trade-off is better handled at the system level. The non-parametric memory design revives tile-coding-style sparse distributed memory in a form that scales, by pairing hashing with deep networks rather than using it standalone.

Real-world applications (illustrative):

• Adaptive robotics or vehicle routing, where a stable map or model is corrected temporarily (e.g., a blocked road) and reverts when the situation clears.
• Personalized recommendation or control systems that must track shifting user context without discarding long-term preferences.
• Game-playing and simulation agents operating in open-ended worlds with many achievements and changing reward structure, such as Craftax.
• Streaming or non-stationary industrial control, where fast local correction must coexist with a slow baseline model.

Industry relevance. Practitioners deploying RL in non-stationary settings get a drop-in decomposition that works alongside standard value-based learners (the implementation builds on the PQN library and is publicly released), needs no task-boundary labels, and adds a bounded-size memory rather than growing networks. The finding that EWC and continual backprop did not change performance is practically important: it suggests these widely used continual-learning add-ons may not deliver the fast adaptation deployments actually need.

Future Directions

• Better tokenization. The authors note their grayscale quantization to 64 levels is a stopgap and suggest incorporating advanced tokenization strategies from foundation-model work.
• Automated hyperparameter tuning. Table size, number of tables, number of hashes, and ρ are currently tuned coordinate-wise (Craftax) or over all combinations (image task); meta-learning is proposed as a way to automate this.
• Selective consolidation. Determining when and what to transfer from the transient to the permanent component remains an open mechanism.
• Extensions. Applying the framework to policy-gradient algorithms, combining mechanistic approaches with the permanent component, and using the framework in streaming RL are named as immediate next steps.

Target Audience

Researchers and graduate students in reinforcement learning and continual learning who are interested in the stability–plasticity problem, complementary learning systems, or memory-augmented architectures; theoretically minded readers working on two-timescale convergence; and applied RL engineers who need agents that adapt online in non-stationary, image-based, or open-ended environments. Familiarity with TD learning, linear function approximation, and stochastic approximation is assumed.

Authors’ abstract

Continual Reinforcement Learning (CRL) agents struggle to adapt to new situations while retaining prior knowledge, leading to a stability-plasticity trade-off. One approach to solving this problem is by using complementary learning systems: one for slow, long-term learning and another for quick, transient adaptation. Anand & Precup (2023) instantiated this idea using permanent and transient value functions, but used the same features in both systems. Intuitively, long-term learning should rely more on parametric representations for broad generalization, while transient learning would be best served by non-parametric representations for situation-specific adaptation. In this paper, we explore this idea both theoretically and empirically. By introducing a novel non-parametric approximation for estimating the transient value function in complex tasks, we demonstrate that our method enables online learning and outperforms competitive baselines on image-based tasks and Craftax-Classic, underscoring the effectiveness of our system-level decomposition.