NeuroSynth: A Biologically Inspired Continual Reinforcement Learning Architecture for Mitigating Catastrophic Forgetting

AI summary

Overview

Research area: Continual reinforcement learning (RL) and brain-inspired artificial intelligence — specifically, methods for reducing catastrophic forgetting in agents trained on a sequence of tasks.

Technical level: Intermediate. The paper is written accessibly and defines its terms, but assumes familiarity with standard RL concepts (PPO, replay buffers, policy distillation, KL divergence, Fisher Information) and with the continual-learning problem setting.

Scope: The paper proposes NeuroSynth, a dual-pathway architecture that separates fast task acquisition from slow consolidation, and benchmarks it against PPO and EWC on three sequential grid-world navigation tasks in a custom NeuroMaze-CL environment.

What This Paper Is About

Neural networks trained on a new task tend to overwrite the parameters that supported earlier tasks, a failure mode called catastrophic forgetting. Biological brains largely avoid this by using complementary systems — a fast-encoding hippocampus and a slow-consolidating neocortex — so the paper asks whether copying that division of labor into a reinforcement learning architecture can preserve old task knowledge without sacrificing the ability to learn new tasks. The goal is a working algorithm, NeuroSynth, tested under a deliberately harsh "non-revisitation" protocol where a task is never trained on again once its phase ends.

Key Contributions

1. A functionally specified dual-pathway architecture. NeuroSynth splits memory into a hippocampal-inspired "plan" pathway that rapidly acquires the first task and is then frozen, and a cortical-inspired "habit" pathway that stays plastic for later tasks — an architectural separation of acquisition and consolidation rather than a purely parameter-protection approach.

2. A combined consolidation mechanism. The frozen planner is maintained as an exponential moving average (EMA) teacher network, and the habit pathway is stabilized through replay-based rehearsal (with reservoir sampling) plus Kullback-Leibler divergence policy distillation with temperature scaling applied specifically on replayed first-task states.

3. A purpose-built benchmark and protocol. The authors introduce NeuroMaze-CL, an 8×8 partially observable grid-world built on Gymnasium, and evaluate all methods under a sequential non-revisitation schedule with shared deterministic seed schedules.

4. A head-to-head comparison across six seeds. PPO, EWC, and NeuroSynth are compared with means, standard deviations, two-sample significance tests, and Cohen's d effect sizes reported for the retention and final-task results.

Main Findings

• Task A retention after full sequential training: NeuroSynth reached a mean 18.00% success rate versus 0.33% for PPO (p = 0.014929, Cohen's d = 1.49), a statistically significant difference with a large effect size.

• Task B retention after Task C training: NeuroSynth retained 35.33% versus 0.00% for PPO (p = 0.002376, Cohen's d = 2.31), a significant difference with a very large effect size.

• Final Task C performance versus EWC: NeuroSynth achieved 9.00% versus 2.00% for EWC (p = 0.226643, Cohen's d = 0.56). This is described as a moderate but not statistically significant advantage.

• PPO showed severe forgetting: The paper reports that PPO's final Task A retention approached zero after Task C training, indicating near-complete overwriting of earlier representations, and its Task B performance fell to 0.00%.

• EWC traded plasticity for stability: EWC preserved earlier-task representations more effectively than PPO, but the paper attributes its comparatively weaker Task C acquisition to Fisher-based parameter constraints reducing the network's ability to reorganize representations in later phases.

• A moderate stability–plasticity balance: The authors conclude that NeuroSynth reduced forgetting relative to PPO while retaining better adaptability than EWC, though the Task C comparison against EWC did not reach significance.

Methodology in Plain English

The authors built a small navigation world, NeuroMaze-CL, on an 8×8 grid. The agent does not see the whole maze: it receives a 7×7 egocentric RGB window centered on itself, converted into normalized float32 tensors of size 3×7×7. This partial observability forces the agent to build internal representations rather than react to a full map.

Three tasks were trained in a fixed order — Task A, then Task B, then Task C — with the maze structure held constant and only the goal location changing. Reaching the correct goal gave a terminal reward of +1.0. Decoy goals ended the episode with 0.0, and conflict decoys used during Task C gave −1.0 to increase interference pressure against previously learned goals. Non-terminal steps carried a −0.01 penalty plus potential-based reward shaping proportional to reductions in Manhattan distance to the active goal. Episodes were capped at 150 steps, and the action space was four discrete cardinal moves. Crucially, once a task phase ended it was never revisited.

Training budgets were 100,000 steps for Task A, 100,000 for Task B, and 200,000 for Task C — the longer final phase was chosen to intensify interference and forgetting pressure. No parameter resets occurred between tasks. Evaluation used deterministic seed schedules shared across all three methods, with 50 episodes per task per evaluation phase, run over six independent random seeds.

The three compared systems were:

• PPO: a standard actor-critic setup using clipped surrogate objectives, generalized advantage estimation, entropy regularization, and gradient clipping — deliberately containing no continual-learning or replay machinery, serving as the forgetting reference condition.
• EWC: a double DQN value-based agent whose updates are constrained by Fisher Information regularization, with Fisher matrices computed from temporal-difference loss gradients sampled from task-specific replay buffers and penalties averaged across prior tasks.
• NeuroSynth: the plan pathway learns Task A via temporal-difference learning and is then frozen, with an EMA copy kept as a teacher. The habit pathway remains plastic during Tasks B and C, learning from replay batches that mix current-task and replayed prior-task experiences, while KL divergence distillation on replayed Task A states pulls the habit policy toward the frozen planner's behavior.

Everything was implemented in Python 3.10 with PyTorch, using Stable-Baselines3 components for PPO and the Gymnasium interface for the environment, with NumPy for numerical processing and Matplotlib for figures. Experiments ran on an NVIDIA RTX 4060 GPU. Reproducibility was enforced through deterministic seed initialization, deterministic evaluation schedules, synchronized environment resets, and deterministic PyTorch configurations where supported. Code, the benchmark, training and evaluation scripts, and statistical analysis scripts are publicly available on GitHub.

Why This Matters

The paper argues that most RL architectures assume a stationary task, which limits deployment in settings where objectives change over time — robotics, autonomous navigation, and medical AI are named explicitly. It also positions architectural separation of memory systems as an alternative to the two dominant families of continual-learning methods (parameter regularization like EWC, and progressive network expansion), which either restricts plasticity or requires the model to keep growing.

Real-world applications named or implied in the paper:

• Robotics and robotic manipulation: systems that must acquire new motor behaviors while preserving previously learned ones.
• Autonomous navigation: agents operating in dynamic environments where goals or routes change over time.
• Medical artificial intelligence: the paper notes a shortened, applications-focused version of this work evaluating NeuroSynth in an ICU sepsis setting.
• Embodied control and drone coordination: listed as future large-scale evaluation targets where lifelong adaptation matters.

Industry relevance: any deployment where retraining from scratch is impractical — an agent that must pick up a new objective on the fly without losing competence at its original job. The specific mechanism (frozen teacher + replay + distillation onto a plastic student) is also a pattern familiar from industrial model-update pipelines, where a stable production policy must not be corrupted by continued training.

Future Directions

• Scaling beyond grid worlds: evaluating NeuroSynth in large-scale continuous-control and robotic environments, since the current experiments used discrete actions and low-dimensional observations. The paper notes this does not fully represent problems with high-dimensional sensory inputs, stochastic dynamics, or long-horizon planning.

• Sparse-reward and procedurally generated settings: testing under sparse rewards, where useful trajectories are rare and forgetting may become more severe, and in procedurally generated environments to see whether the architecture generalizes across changing task structures rather than memorizing fixed layouts.

• Adaptive instead of static consolidation: the planner pathway is permanently frozen after Task A, which helps retention but may reduce long-term flexibility. The paper proposes dynamic consolidation schedules, selective reactivation of frozen pathways, or meta-learned replay policies.

• Component-level ablations and broader baselines: the study did not run ablations isolating the individual contributions of replay, pathway separation, and distillation, and it compared only against PPO and EWC. The authors call for comparisons against replay-based continual RL systems, hybrid methods, and transformer-based memory architectures, with larger statistical populations and many more sequential tasks.

Target Audience

Readers who will get the most from this paper are students and researchers working at the intersection of reinforcement learning and neuroscience-inspired AI — particularly those interested in continual learning, catastrophic forgetting, and complementary learning systems theory. It is also well suited to readers who want a concrete, reproducible example of turning a biological theory (hippocampal-cortical consolidation) into an implementable algorithm with an empirical evaluation. Given its accessible framing, clear baseline descriptions, and public code release, it should be readable by advanced undergraduates or motivated high-school researchers, while still offering practitioners a useful architectural pattern for stabilizing sequential training. Readers seeking large-scale benchmarks, component ablations, or high-dimensional/continuous-control evidence should note that this paper does not provide them.

Authors’ abstract

Artificial Intelligence (AI) systems often perform well on isolated tasks but struggle under continual learning conditions, where training on new tasks can overwrite previously acquired knowledge, a failure mode known as catastrophic forgetting. Biological learning systems reduce this interference through complementary memory processes involving rapid hippocampal encoding and slower cortical consolidation. This study introduces NeuroSynth, a brain-inspired continual reinforcement learning architecture designed to mitigate catastrophic forgetting through a dual-pathway consolidation mechanism. NeuroSynth separates rapid task acquisition from long-term retention using distinct "plan" and "habit" pathways combined with replay and knowledge distillation. NeuroSynth was evaluated against Proximal Policy Optimization (PPO) and Elastic Weight Consolidation (EWC) across three sequential navigation tasks with changing goal locations in a non-revisitation continual learning setting. Across six independent seeds, NeuroSynth preserved substantially more early-task knowledge than PPO after sequential training, achieving 18.00% Task A success rate compared to 0.33% for PPO (p = 0.014929, Cohen's d = 1.49) and 35.33% Task B success rate compared to 0.00% for PPO (p = 0.002376, Cohen's d = 2.31). NeuroSynth also demonstrated higher final Task C performance than EWC, achieving 9.00% compared to 2.00% (p = 0.226643, Cohen's d = 0.56), indicating a moderate but not statistically significant advantage. These findings suggest that biologically inspired consolidation mechanisms may improve the stability-plasticity balance in continual reinforcement learning systems.