Self-Preservation in Language Models

AI summary

Overview

Research area: Continual learning and catastrophic forgetting in large language models, specifically whether a model's own knowledge can be used to decide what to rehearse so that prior capabilities survive new instruction tuning.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, replay-based continual learning, and paired accuracy analysis.

Scope (one sentence): The paper introduces "Task Banks" — collections of task descriptions and optional examples — and uses them to test whether model-generated replay, bank composition, and model-driven target selection can protect held-out capabilities during continual instruction tuning of Qwen models from 0.6B to 4B parameters.

What This Paper Is About

When a pretrained language model is fine-tuned on a new task, it often loses capabilities it previously had — a phenomenon called catastrophic forgetting. Replay-based continual learning mitigates this by rehearsing old data, but the original training data are usually unavailable, so any supplied set of preservation tasks covers only part of what the model knows. The paper asks a "self-preservation" question: can a model use its own knowledge both to generate replay data and to choose which preservation targets to rehearse, so that protection extends beyond the tasks it was explicitly given?

Key Contributions

1. Task Banks as a formal interface for preservation targets. The authors organize supplied targets into banks ℬ = {b₁, …, bₘ}, where each entry contains a name, a description of the behavior and input/output format, and optional demonstrations. The bank separates what to replay from the examples generated for it, and allows rehearsing one entry, a selected subset, or the full bank.

2. Task-Bank Self-Replay. A frozen, pre-update model generates both questions and answers for selected bank entries, and these are mixed with new-task examples during supervised fine-tuning. This is contrasted with SFT (incoming data only), Single-Task Replay (one supplied task), and Replay (stored real examples and labels).

3. A held-out evaluation protocol that measures protection beyond the bank. ARC-Challenge, WinoGrande, PIQA, and SVAMP are excluded from bank construction and target selection — their examples, labels, descriptions, and scores — so their results test whether protection generalizes beyond the replay inventory. The authors also report Retention (correct → correct), the fraction of initially correct answers that remain correct, alongside task-level accuracy changes.

4. Bank-composition and target-selection comparisons. Combining two partial banks is compared against either alone, and random selection is compared against selection using the bank only and selection also informed by the incoming task.

Main Findings

• Aggregate accuracy can conceal lost behavior. In the paired analysis averaged over four held-out tasks and two seeds (Qwen3-4B learning MedQA with 256 input rows, one epoch, 1e-6, two seeds), 3.56% of examples become incorrect and 3.61% become correct, leaving only a +0.05 pp net change. The near-flat average hides substantial movement in both directions.

• Broad replay protects held-out tasks better than single-target replay. In the C-STANCE comparison with Qwen3-4B (one epoch, 1e-5, five seeds), mean held-out accuracy changes are −4.34 pp for SFT and −3.99 pp for Single-Task Replay, versus +0.86 pp for Replay and +1.07 pp for Task-Bank Self-Replay. Incoming gains stay close across methods, from +27.74 to +27.86 pp.

• Combining partial banks improves mean held-out Retention at all three tested sizes. With Bank A (MMLU, GSM8K) and Bank B (HellaSwag, BoolQ), the combined bank reaches 88.70%, 93.28%, and 96.01% Retention at 0.6B, 1.7B, and 4B — exceeding the better partial bank by 4.42, 3.32, and 2.99 pp. Incoming gains are 0.45, 0.60, and 0.28 pp below SFT. The best single bank varies by held-out task; the combined bank has the best mean.

• Incoming-task information improves target selection, but only at 4B. When choosing two of four bank entries, incoming-aware selection reaches 95.69% mean held-out Retention at 4B, versus 92.94% for random and 93.42% for bank-only selection — a 2.75 pp advantage over random, with an incoming gain 0.04 pp above random. It lands within 0.32 pp of the full bank while using two of its four targets. The Retention difference from random is −0.76 pp at 0.6B and approximately zero at 1.7B.

• Replay trades acquisition for preservation under a fixed example budget. With 59 of 256 input rows allocated to replay (Qwen3-4B, MedQA, 1e-6, one epoch, 16 updates, two seeds), Task-Bank Self-Replay improves the mean seed-wise worst held-out change from −1.56 to −0.68 pp, while MedQA gain falls from +1.17 to +0.49 pp. Real-data Replay sits in between, at −0.88 pp worst held-out change and +0.98 pp incoming gain.

• The incoming task shapes which capabilities degrade. Holding model and protocol fixed at 1e-6, Qwen3.5-4B gains +4.7 pp on MedQA and +7.8 pp on C-STANCE, but prior-task changes differ: SVAMP accuracy rises by about six points after the MedQA update and about one point after the C-STANCE update. The effect is heterogeneous, not a uniform shift.

• A learning rate that looks adequate can produce failed adaptation. In the 36-run sweep (three models × two update tasks × three learning rates × two seeds, 256 rows, one epoch), the 1e-5 conditions reduce incoming-task accuracy across all six model–task combinations. In the separate five-rate Qwen3-1.7B C-STANCE study (1,024 examples, 64 updates), 1e-7 gives +12.01 pp incoming gain and 97.02% mean held-out Retention, while 3e-6 gives +9.67 pp and 82.22%, and 1e-5 gives −2.05 pp incoming and 32.88% Retention.

Methodology in Plain English

The setup has three separate roles: a selector picks which task entries to rehearse, a frozen reference model generates replay questions and answers for those entries, and a learner is fine-tuned on incoming-task examples mixed with the replay. The reference model is never updated, and the roles stay distinct even when they share a checkpoint.

To test protection beyond the supplied inventory, the authors split evaluation tasks into three tiers: update tasks (MedQA, C-STANCE), replay-eligible bank tasks (MMLU, HellaSwag, GSM8K, BoolQ), and held-out tasks (ARC-C, WinoGrande, PIQA, SVAMP) whose data and scores never touch replay construction or selection. Main protection experiments use one epoch, a learning rate of 1e-5, and five seeds; MedQA comparisons use 1,024 incoming examples. A separate controlled sweep varies model, incoming task, and learning rate over {1e-7, 1e-6, 1e-5} with full-parameter AdamW.

Evaluation compares post-update accuracy to the pre-update model on the same items (capped at 512 examples per capability, with at least 400 required after filtering), and — crucially — tracks individual answer transitions rather than only averages, computing Retention as the share of initially correct answers that stay correct. This paired view separates genuine preservation from newly corrected answers masking lost ones. A fixed-allocation study replaces some incoming rows with replay rows to see what preservation costs in new-task learning, with the caveat that token exposure and generation cost are not matched even when nominal rows and optimizer steps are.

Why This Matters

Impact on research. The paper reframes preservation as a coverage problem rather than purely a volume problem: generating more examples for the same targets adds depth but not breadth, while combining distinct partial banks adds breadth without needing the model to discover new tasks. It also supplies a measurement argument — a +0.05 pp net change coexisting with 3.56% correct→incorrect transitions — for why aggregate accuracy is an insufficient forgetting metric, and it shows a learning-rate effect where a rate that appears reasonable (1e-5) actually destroys incoming-task accuracy, which can be mistaken for a retention–acquisition trade-off.

Real-world applications (potential, not demonstrated in the paper):

• Updating a deployed assistant with new domain knowledge (e.g., a medical QA update) without silently losing reasoning or commonsense performance on unrelated user requests.
• Safety-relevant specialization, where a model is tuned for a regulated domain and operators need evidence that prior behavior survives the update.
• Model maintenance pipelines that must keep adapting a checkpoint over successive releases without access to the original pretraining corpus.
• Auditing fine-tuned vendor models for hidden capability regressions that headline benchmark averages would mask.

Industry relevance. The approach requires no stored pretraining data — only compact task descriptions and a frozen copy of the pre-update model — which matches practical constraints where training corpora are unavailable, restricted, or too large to retain. The finding that target selection helps at 4B but not at 0.6B or 1.7B also gives practitioners a concrete caution: selection quality appears to scale with model size in the tested configurations, so small-model deployments should not assume the same benefit.

Future Directions

• Maintaining the inventory across successive updates. The authors explicitly leave open how to keep a useful bank over repeated updates, noting that proposed entries would need demonstrated competence, distinct behavioral coverage, and incremental preservation value. Their exploratory single-arm discovery pilot produced negative accuracy changes, and a two-update, one-seed pilot showed mixed accuracy changes.

• Separating the roles that are currently confounded. Learner, selector, and replay-generator strength are not varied independently, so it is unclear whether the 4B selection advantage comes from selector quality, learned capability self-knowledge, or scale in general.

• Testing other update rules. The paper studies SFT because it is a demanding setting; on-policy reinforcement learning and on-policy distillation are reported elsewhere to forget less, but their different learning signals may change which preservation targets are useful.

• Establishing a real acquisition–retention frontier. The fixed-allocation comparison shows replay reduces worst held-out loss while also reducing new-task gains, and the authors state it does not establish a better frontier; token exposure and generation cost also remain unmatched.

Target Audience

Researchers and engineers working on continual learning, fine-tuning, and model maintenance for language models — particularly those who must adapt models without access to original training data. It is also relevant to evaluation researchers interested in paired, answer-level measurement of forgetting, and to practitioners who need to reason about how much protection a limited inventory of preservation tasks can actually buy. Readers without background in replay-based continual learning will find the setup accessible but the experimental detail dense.

Authors’ abstract

Continual learning (CL) seeks to adapt pretrained models while preserving prior capabilities. Yet the tasks supplied for preservation cover only part of what a model knows, while forgetting can extend beyond them. Recent work shows that language models can supplement this limited evidence by generating useful replay without access to their original training data. This raises a question about self-preservation: can a model help choose preservation targets that protect its own capabilities more broadly? To study this question, we organize task descriptions and their examples into Task Banks, whose entries can be selected and combined to guide replay generation by a frozen model. Continual instruction tuning with Qwen models of varying sizes yields three findings. First, the new task and learning rate shape which prior capabilities degrade, while tracking previously correct answers reveals losses that average accuracy can conceal. Second, generated replay reduces forgetting on tasks outside the bank. Combining partial banks strengthens this protection, improving mean retention of previously correct answers over either bank alone across the tested sizes. Third, the model can also help choose replay targets: selection informed by the new task improves mean held-out retention over random selection at 4B, with similar new-task gains, although smaller models show no corresponding advantage. Under a fixed example budget, however, replacing new-task examples with replay improves preservation at the cost of smaller new-task gains. Together, these findings show that models can support broader self-preservation by generating replay data and, in some settings, choosing which tasks to replay.