Continual Learning and Catastrophic Forgetting

A model trained in sequence can improve on new data while losing ground on old. New gradients reuse parameters that already supported other behaviors, so ordinary optimization may overwrite useful solutions. When the loss of earlier capability is severe, it is called catastrophic forgetting.

What makes the setting hard depends on what is available at each stage. Some systems may store old examples. Others face strict retention limits. Task identity may be known at inference, or the model may have to tell all old and new classes apart without it. Changes to the input distribution and to the label space create further variants that should not be treated as interchangeable. Continual learning therefore needs a stream protocol, not merely a new optimizer. The protocol fixes task order, memory allowance, revisitation, evaluation times, and what the model knows at prediction time.

One method attacked the interference head on, by slowing learning exactly where it would do damage. Elastic weight consolidation “remembers old tasks by selectively slowing down learning on the weights important for those tasks”. Kirkpatrick and colleagues tested it on permuted MNIST and on ten Atari games “chosen randomly from those that are played at human level or above by DQN”.

The rest of this lesson runs on measurements rather than on that description. Every number below comes from a published continual-learning paper or from a named regulator. The subject turns on quantities: how much was lost, how much memory it took to keep, and what the loss depends on.

Three families protect old knowledge, and most working systems mix them. Regularization keeps every parameter and constrains the important ones — the elastic weight consolidation route, which “remembers old tasks by selectively slowing down learning on the weights important for those tasks”. Replay keeps data. Isolation or expansion keeps parameters, reserving adapters, experts or subnetworks per task. Each family can be priced.

Isolation has a published price tag. PackNet fits several tasks into one network by pruning it between them, and Mallya and Lazebnik measured what the extra bookkeeping costs. On an ImageNet-trained VGG-16 of 537 MB, the per-task masks came to about 17 MB for one added task and about 34 MB for three. Four tasks in total, for a 1/16 increase in size. Storage is the cheap part.

The expensive part is capacity, and it drains along the stream. After the initial 50% pruning, about 67M of the 134M conv1_1–fc7 parameters were free. Task II consumed 16.75M, leaving 50.25M. Task III consumed about 13M, leaving about 37M. Nothing was lost to forgetting there. The network simply had less room for each task than for the one before, which is a different failure with a different fix.

How hard the problem is depends on what the model is told at prediction time, and that difference has been measured on one unchanged network. Fine-tuned naively, with no defense at all, it reached 84.32% (±0.99) task-incremental, 60.13% (±1.66) domain-incremental and 19.89% (±0.02) class-incremental on Split MNIST, over 20 seeds per cell. On Split CIFAR-100 the same three columns read 61.43% (±0.36), 18.42% (±0.33) and 7.71% (±0.18). Van de Ven and colleagues published the comparison in Nature Machine Intelligence in 2022. The data is the same and the training is the same. Only what the model is told at prediction time changes, and that alone moves accuracy by more than sixty points on Split MNIST.

Defenses do not close the gap uniformly. Elastic weight consolidation scored 99.06% (±0.15) on Split MNIST task-incrementally and 20.64% (±0.52) class-incrementally on the same data. That is indistinguishable, in practice, from the 19.89% of doing nothing at all. The authors state the pattern in one sentence: “With task-incremental learning, almost all tested methods performed well compared to the ‘none’ and ‘joint’ baselines, with domain-incremental learning the relative performances of many methods dropped considerably and with class-incremental learning they decreased even further.”

So the three settings below are not three flavors of one benchmark. A method that works in the first column may be worth nothing in the third. Reporting the first while deploying into the third is the most common way to overstate a continual learner.

A continual learner has to be judged across the whole task sequence, and the vocabulary for doing that was fixed by the GEM paper in 2017. Lopez-Paz and Ranzato fill a T×T matrix R whose entry is the test accuracy on task j after the model has finished training on task i. Three numbers come off it: ACC, backward transfer (BWT, the mean of R_T,i − R_i,i) and forward transfer (FWT, the mean of R_i−1,i − b̄_i). One line of the paper connects the matrix to this lesson's subject: “Large negative backward transfer is also known as (catastrophic) forgetting.”

On MNIST Rotations at five passes per task, a single predictor trained sequentially fell to ACC 0.43 with BWT −0.40. GEM held ACC 0.89 with BWT −0.02 — the same 0.89 that the same architecture reached when it was trained on shuffled i.i.d. data instead of a stream. One final average would have reported 0.43 and 0.89. The matrix explains that the difference is almost entirely retention.

A second family normalizes rather than differences. Kemker and colleagues built three benchmark protocols — data permutation, incremental class learning, multi-modal learning — and three normalized area-under-the-curve metrics, Ωbase, Ωnew and Ωall, each divided by αideal, the accuracy of the same network trained offline. That ideal was 62.1% on CUB-200 and 46.1% on AudioSet. That is what a perfect score on those sets actually means.

The protocol should make forgetting visible at every stage. Every step below has a published instance to copy rather than an instruction to interpret.

Defining the stream can be as concrete as CORe50: 164,866 128×128 RGB-D images, being 11 recording sessions (8 indoor, 3 outdoor) × 50 domestic objects in 10 categories × about 300 frames, recorded at 20 fps with a Kinect 2.0, with sessions 3, 7 and 10 held out for test. Lomonaco and Maltoni also fixed three stream protocols by name — New Instances (NI), New Classes (NC), and New Instances and Classes (NIC) — the second defined this way: “New Classes (NC): new training patterns belonging to different classes become available in subsequent batches. In this case the model should be able to deal with the new classes without losing accuracy on the previous ones.” That NI/NC/NIC split is an independent naming of the same distinction the settings section teaches.

Step two, setting the memory and compute budgets, is where the replay sweep from 200 to 5,120 examples belongs. It is also where a deployment under the FDA guidance writes down its planned modifications and impact assessment in advance — the predetermined change control plan described in the governance callout above.

Steps three and four — evaluate after every task, then compare against naive sequential training — only discriminate if the benchmark is hard enough to separate methods. Kemker and colleagues documented the failure directly. Their incremental class learning started from half the classes (5 for MNIST, 100 for CUB-200, 50 for AudioSet), with one new class per session, and the ranking obtained on MNIST did not survive the move to real-world data: “In nearly every case, Ωall is greater for MNIST than on CUB-200 or AudioSet, demonstrating the need for alternative incremental learning benchmarks.”

Step six, order sensitivity, is not a precaution against a hypothetical. Mallya and Lazebnik ran all six orderings of three tasks with three runs each, and reported the effect of training order: “For example, the top-1 error increases from 16.00% to 18.34% to 19.91% for the Stanford Cars dataset as we delay its addition to the network.” Same dataset, same method, same budget. Three different results, produced by nothing but position in the queue. A single-order experiment cannot tell you which of those three numbers it happened to draw.

Final average accuracy hides when and where knowledge was lost. The object that does not hide it is the full matrix R: test accuracy on task j after the model has finished task i, filled in as the stream runs. Current-task quality is its diagonal, R_i,i. Retained quality is its final row. Backward transfer is the gap between the two, the mean of R_T,i − R_i,i, and forward transfer reads across the row above the diagonal, as the mean of R_i−1,i − b̄_i.

Two numbers show why the whole matrix is worth keeping. A single predictor trained sequentially on MNIST Rotations reported ACC 0.43 and BWT −0.40. GEM reported 0.89 and −0.02, matching the 0.89 of the same architecture trained on shuffled i.i.d. data, which is the ceiling. Reported alone, the accuracies say one method is better. With BWT beside them, they say the weaker one learned each task and then lost it.

Two more panels sit outside the matrix. Memory and compute have to be tracked as first-class results, since the same method scores 0.487 or 0.654 depending only on the buffer it was given, and about 17 MB of masks per task on a 537 MB backbone is a real line item. Unknown-task behavior has to be tested explicitly too. The drop from 84.32% to 19.89% on Split MNIST was caused by nothing except removing the task identifier at prediction time.