FIT to Forget: Robust Continual Unlearning for Large Language Models

AI summary

Overview

• Research area: Machine unlearning for large language models, specifically the under-explored setting of continual (sequential) unlearning, plus a new evaluation benchmark.
• Technical level: Advanced. The paper assumes familiarity with gradient-ascent unlearning, preference optimization objectives (NPO), LoRA adapters, Shapley-value attribution, and standard LLM evaluation suites (MMLU, CommonsenseQA, GSM8K).
• Scope: The paper proposes FIT, a three-module framework layered on top of existing single-shot unlearning primitives, and PCH, a benchmark of Personal, Copyrighted, and Harmful content with two symmetric metrics (Forget Degree and Retain Utility), evaluated on five LLMs up to 14B parameters and up to 300 sequential deletion requests.

What This Paper Is About

Almost all LLM unlearning research assumes a single-shot setting: all the data to be forgotten arrives at once and is removed in one operation. In practice, deletion requests arrive one after another over a model's lifetime. The paper shows that naively re-applying single-shot methods to each incoming request causes a rapid, cumulative collapse in both forgetting quality and general model utility — a form of catastrophic forgetting the authors attribute to redundant requests, unstable sequential updates, and long-term parameter drift. Their goal is a framework that processes long streams of deletion requests while keeping the model useful and resistant to attempts to recover the erased knowledge afterward.

Key Contributions

1. FIT, a continual unlearning framework combining three mechanisms — embedding-based redundancy Filtering, Importance-aware adaptive algorithm selection, and Targeted layer attribution — designed to resist both catastrophic forgetting and post-unlearning recovery.
2. PCH, a unified continual unlearning benchmark covering Personal information, Copyright, and Harmful content (200 samples per category, 600 total, split randomly into forget and retain subsets), generated by GPT-4o with structured prompts and manually verified for category consistency and distributional properties.
3. Two symmetric metrics, Forget Degree (F.D.) and Retain Utility (R.U.), each a geometric mean of Probability, ROUGE-L, and token-level Accuracy applied identically to forget and retain sets, normalized against a retain model — giving a scale-invariant, interpretable trade-off score.
4. The first continual LLM unlearning evaluation at 300 sequential requests and on models up to 14B parameters, far exceeding the typical fewer-than-10 requests and 7B/8B baselines of prior work.

Main Findings

• Single-shot unlearning does not transfer to sequential requests: The paper reports that while single-shot removal has minimal impact, continual unlearning produces a rapid cumulative decline in both forget and retain accuracy after only 25 sequential requests; the Figure 1 experiment on Llama-3-8B with GA is described as covering single unlearning and 100 sequential requests.
• FIT gives the best forgetting-utility trade-off on Llama-3-8B: At 300 requests, FIT reaches F.D. 0.73 and R.U. 0.71, versus 0.64/0.63 for O³ and 0.60/0.62 for ALKN — stated in the abstract as up to +0.09 F.D. and +0.08 R.U. over ALKN and O³.
• Aggressive baselines collapse: On Llama-3-8B at 300 requests, GA, GA+KL, RLabel, and PISCES all report F.D. and R.U. of 0.00, and GA, GA+KL, RLabel, and PISCES report GSM8K of 0.00.
• Conservative baselines under-forget: On Qwen3-14B at 300 requests, O³ reports F.D. 0.86 and R.U. 0.75 while ALKN reports 0.74/0.71 and NPO+KL reports 0.69/0.60; the paper argues higher R.U. with lower F.D. indicates incomplete forgetting.
• Downstream capability is preserved: After 300 requests on Llama-3-8B, FIT scores MMLU 65.56, CSQA 70.01, and GSM8K 57.02, against retain-model scores of 65.97, 70.02, and 58.83. On Qwen3-14B, FIT scores MMLU 79.15, CSQA 77.31, GSM8K 74.28 against retain-model values of 79.99, 81.33, 84.84.
• Strongest resilience to recovery attacks: Under relearning on mixed data and under int4 quantization on Llama-3-8B, FIT achieves the highest F.D. among compared methods, while remaining competitive under retain-only and unrelated-data relearning.
• Adaptive algorithm selection is the most load-bearing component: Removing it from FIT drops GSM8K to 14.12 after 300 requests on Llama-3-8B and cuts F.D./R.U. to 0.14/0.07; removing filtering yields a milder but consistent decline, and removing targeted layer selection also weakens performance.
• K = 25% is the chosen update ratio: Ablations across update ratios (Figure 6) identify 25% as the optimal threshold for the forgetting-utility trade-off, and layer-attribution analysis across Llama-2-7b-chat-hf, Llama-3-8B, Llama-3-8B-Instruct, and Yi-6B shows unlearning relevance concentrates in compact, request-specific modular regions rather than spreading uniformly.
• PCH is out-of-distribution to pretraining: A model fine-tuned only on the retain set starts at low accuracy and improves steadily, confirming that PCH examples are unseen by the base model.
• Cross-benchmark results on MUSE and TOFU are not reported in the provided content, which is truncated at the start of Section 5.3; efficiency (GPU memory) and results on the remaining LLMs are likewise deferred to Appendix I and are not included in the supplied text.

Methodology in Plain English

FIT does not replace existing unlearning algorithms; it decides how to apply them across a stream of requests.

Step 1 — Filter redundant requests. Each incoming unlearning sample is split into fixed-size chunks. Each chunk gets a SimCSE embedding, and the system computes its maximum cosine similarity against all previously forgotten content. Chunks below a threshold τ are treated as new and queued for unlearning. Chunks above τ are not discarded immediately; the system runs a second "loss-difference" test, measuring how much the model's cross-entropy loss changes when that chunk is removed from the request. If the change is large, the chunk is kept anyway, because it carries unique information (for example, a different person's name in an otherwise identical sentence).

Step 2 — Pick the right algorithm per request. For each filtered request the framework computes IMP, the L2 norm of the loss gradient with respect to the request's input embedding. This is a cheap proxy for how strongly the model has memorized that data. IMP is bucketed into three tiers, which route the request to one of six standard single-shot primitives: low-importance requests get aggressive methods such as RLabel, medium-importance requests get NPO, and high-importance requests get conservative methods such as NPO+KL.

Step 3 — Update only the most relevant layers. For a model with L layers, each layer is temporarily masked and the resulting loss deviation is measured, approximating its Shapley value. Layers are ranked by that score and only the top K% — that is, the ⌈KL/100⌉ highest-scoring layers — have their multi-layer perceptron and multi-head attention modules updated; everything else is frozen. Because K is a ratio rather than a fixed count, the update budget scales automatically with model depth.

Evaluation design. Since the original pretraining corpus is unavailable, the authors follow a synthetic proxy approach: they fine-tune a model on the union of the synthesized forget and retain sets to embed the knowledge, and fine-tune a separate copy on the retain set only to serve as the "retain model" gold standard. All experiments use the PCH benchmark across Yi-6B, Llama-2-7b-chat-hf, Llama-3-8B, Llama-3-8B-Instruct, and Qwen3-14B, compared against GA, GA+GD, GA+KL, NPO, NPO+KL, RLabel, PISCES, O³, and ALKN.

Why This Matters

Impact on research. The paper reframes unlearning from a one-off operation into an ongoing process, and it argues that the metrics used in the field (MUSE's four disparate criteria; TOFU's paraphrase-based answers and mismatched forget/retain protocols) make comparisons unreliable. By releasing a category-unified benchmark and two symmetric aggregated metrics computed as geometric means, it gives the community a common yardstick for a problem that previously had none. The finding that aggressive methods collapse entirely after a few hundred requests is a direct challenge to how unlearning methods are currently validated.

Real-world applications.

• Regulatory compliance with the GDPR's "Right to be Forgotten" and the CCPA for deployed LLM services that receive deletion requests continuously.
• Removing copyrighted text from models trained on large web corpora without retraining from scratch.
• Removing harmful or defamatory content, including rumors, from a deployed model's behavior.
• Protecting personal identifiers (names, personal details) in models offered as a service, where a single leaked detail is a privacy failure.

Industry relevance. The paper is co-authored with Ant Group, indicating direct commercial interest in operating LLMs that must honor deletion requests at scale without retraining. The framework's practical selling points are that it reuses off-the-shelf unlearning primitives, bounds computational overhead through ratio-based layer selection, and reports resistance to two realistic attacks — relearning via fine-tuning and int4 quantization, both of which can resurrect "forgotten" knowledge in a model a provider has already shipped.

Future Directions

• Extend the evaluation to the full benchmark suite. The paper's cross-benchmark comparison on MUSE and TOFU is announced but its results are not present in the supplied content, leaving open how FIT's behavior generalizes beyond PCH.
• Close the remaining trade-off gap. On Qwen3-14B, O³ reports higher F.D. at 300 requests (0.86 versus FIT's 0.89 — the two are close) while FIT leads on R.U.; understanding which methods win under which request volumes and content categories remains unresolved.
• Widen the scope of the framework. FIT is presented as orthogonal to the underlying unlearning primitive, so testing whether the three modules transfer to other objectives, other model families, and multimodal models is a natural next step.
• Strengthen guarantees beyond empirical robustness. The threat model includes malicious high-volume unlearning requests intended to cause denial of service, relearning attacks, and quantization attacks, but the framework offers empirical resilience rather than formal unlearning guarantees; connecting the ratio-based layer selection to principled privacy bounds is an open question.

Target Audience

Researchers and engineers working on LLM safety, privacy, and compliance — particularly those building or auditing deployed models that must handle deletion requests in production. It is also relevant to practitioners choosing between unlearning methods, since the benchmark and the F.D./R.U. metrics can be used independently of FIT itself, and to reviewers and method developers who need to understand why single-shot unlearning results do not predict behavior under sequential requests. Readers without prior exposure to gradient-ascent unlearning, NPO, or Shapley attribution will find the paper demanding.

Authors’ abstract

While large language models (LLMs) exhibit remarkable capabilities, they increasingly face demands to unlearn memorized privacy-sensitive, copyrighted, or harmful content. Existing unlearning methods primarily focus on \emph{single-shot} scenarios, whereas real-world deletion requests arrive \emph{continually}. Naïvely applying these methods to sequential requests leads to severe utility degradation and catastrophic forgetting. To address this, we propose \fit, a robust continual unlearning framework to process high-volume sequential deletion streams while resisting both catastrophic forgetting and post-unlearning recovery. \fit stabilizes sequential updates through three synergistic mechanisms: redundancy \underline{F}iltering, \underline{I}mportance-aware adaptive algorithm selection, and \underline{T}argeted layer attribution. Furthermore, to facilitate rigorous evaluation, we introduce \textbf{PCH}, a unified benchmark encompassing \textbf{P}ersonal, \textbf{C}opyrighted, and \textbf{H}armful content, alongside two symmetric metrics, Forget Degree (F.D.) and Retain Utility (R.U.), to systematically quantify forgetting-utility trade-offs. Extensive experiments across five LLMs (up to 14B parameters) demonstrate that \fit consistently achieves state-of-the-art unlearning efficacy and utility preservation. Notably, even after hundreds of sequential requests, \fit preserves strong downstream (\eg, GSM8K, MMLU) performance and exhibits superior resilience against relearning and quantization recovery attacks.