FG-OrIU: Towards Better Forgetting via Feature-Gradient Orthogonality for Incremental Unlearning

AI summary

Overview

Research area: Machine learning — machine unlearning and incremental/continual learning, specifically class-level unlearning in pre-trained vision models.

Technical level: Advanced. The paper assumes familiarity with singular value decomposition, subspace projections, LoRA adapters, and gradient-based optimization.

Scope: The paper proposes FG-OrIU, a framework that enforces orthogonal constraints at both the feature and gradient level to make class forgetting in pre-trained vision models irreversible across sequences of deletion requests.

What This Paper Is About

Pre-trained models sometimes must delete specific classes of data on request (for example, a face identity or a medical category), and in real deployments these deletion requests arrive one after another rather than all at once. The authors show that existing unlearning methods only produce "superficial forgetting": the erased class information is still partially recoverable, because those methods degrade a model's features instead of removing them. FG-OrIU is designed to make forgetting deep and irreversible while keeping the model's performance on the classes that must be retained.

Key Contributions

1. The paper characterizes existing incremental unlearning methods as achieving "superficial forgetting" rather than "deep forgetting," showing that partial feature suppression leaves class-specific residual information recoverable.
2. It introduces FG-OrIU, described as the first framework that unifies forgetting constraints at both the feature level and the gradient level via dual orthogonal projection, with the goal of irreversible forgetting alongside preserved remaining-class performance.
3. It adds a dynamic subspace adaptation procedure in which the forgetting subspace expands and the remaining subspace contracts as new unlearning tasks arrive, without retraining the backbone.
4. It reports experiments across machine unlearning and incremental unlearning scenarios on multiple benchmarks, including an ablation study isolating the four constraint components and a semantic residual analysis using image reconstruction.

Main Findings

• Tunable parameter budget: FG-OrIU reports a tunable-parameter ratio of 1.28%, the same as GS-LoRA and GS-LoRA++, and far below LIRF at 50.66%. The backbone and classification head are frozen; only LoRA modules are trained.

• Static unlearning on CASIA-Face100 (Table 2): In the 100-50 setting FG-OrIU reaches H of 74.18, Acc_r of 73.51, and Acc_f of 0.00; in the 100-90 setting it reaches H of 75.49, Acc_r of 77.15, and Acc_f of 0.00. The pre-trained model before forgetting scored 72.45 Acc_r / 74.88 Acc_f on 100-50 and 72.32 Acc_r / 73.86 Acc_f on 100-90. The paper states that in the settings removing 50 and 90 classes, FG-OrIU improves H by 2.62% and 2.64% and ACC_r by 4.99% and 5.29% relative to GS-LoRA.

• Incremental unlearning on CASIA-Face100 (Table 3): Across four tasks that each forget 20 classes, FG-OrIU records H of 74.45, 74.64, 75.49, and 77.71 for tasks 1 through 4, with Acc_f of 0.00 in every task and Acc_o of 0.00 in tasks 2 through 4.

• Incremental unlearning on ImageNet-100 (Table 4): FG-OrIU records H of 87.01, 88.45, 88.90, and 89.15, with Acc_r rising to 92.10 by task 4. Acc_f is 0.10, 0.10, 0.30, and 0.10, and Acc_o is 0.10, 0.00, and 0.00 in tasks 2 through 4.

• Long-sequence gains: The paper reports improvements in ACC_r of 5.15% and 4.6% in the fourth unlearning task on the two datasets, relative to the compared incremental method.

• Baseline failure modes: Continual learning baselines struggle to forget on ImageNet-100 — DER retains Acc_f of 10.40%, 10.90%, 8.80%, and 3.40% across the four tasks. Machine unlearning baselines fail to sustain forgetting over the sequence; LIRF and SCRUB-S show Acc_o increasing on CASIA-Face100 as the task sequence grows.

• Ablation (Table 5): Removing any of the four components — fop-f, fop-r, gop-f, gop-r — degrades the forgetting/remaining trade-off. Removing the remaining-related regularizer or gradient modification (fop-r, gop-r) markedly lowers remaining-class accuracy (fop-r: Acc_r 73.92 then 74.98; gop-r: 71.45 then 73.96). Removing the forgetting-related terms (fop-f, gop-f) slightly raises forgetting-class accuracy (fop-f: Acc_f 0.20 then 0.42 and Acc_o 1.28; gop-f: Acc_f 0.10 then 4.37 and Acc_o 0.82).

• Recoverability evidence (Table 6): Reconstructed images produced with Deep Image Prior from the unlearned model's last-block features give FG-OrIU SSIM of 0.01, 0.02, and 0.01 and PSNR of 8.37, 8.72, and 8.55 across three experiments, versus pre-trained scores of 0.94, 0.95, 0.93 SSIM and 31.32, 31.15, 31.10 PSNR. Comparisons include DER, FDR, SCRUB, and GS-LoRA.

• Qualitative evidence: The paper reports that reconstructing last-layer features of models unlearned by prior methods produces blurred but recognizable images, and that fixing the backbone and retraining only the classification head recovers part of the original accuracy. A t-SNE visualization compares 5 forgetting classes and 10 remaining classes across the pre-trained model, FG-OrIU, and GS-LoRA.

• Not reported: Results for CUB-200 and Omnibenchmark, MS-Celeb-100, and the full semantic residual analysis in Section 5.4.1 are not included in the available content, so no figures from those experiments are given here.

Methodology in Plain English

The method rests on three stages.

Stage 1 — Split the feature space. For each layer of the pre-trained model, the authors collect feature vectors for the classes to be forgotten and for the classes to be kept. They run singular value decomposition on each set and keep the top directions (the leading singular vectors) that capture most of the matrix's energy, above a threshold. This produces two subspaces per layer: a forgetting subspace S_f that holds the discriminative patterns of the target classes, and a remaining subspace S_r that holds everything else.

Stage 2 — Constrain both forward features and backward gradients. The backbone and the classification head are frozen, and small LoRA modules are inserted for training. On the forward pass, features of forgetting-class samples are pushed to be orthogonal to S_f, while features of remaining-class samples are aligned with S_r by minimizing their projection residual. On the backward pass, the raw gradient is modified twice: the component orthogonal to the forgetting subspace is subtracted, so the update concentrates on the directions that encode the forgetting classes, and the component along the remaining subspace is also removed, so that updates cannot disturb the retained features.

Stage 3 — Adapt the subspaces as tasks accumulate. When a new unlearning task arrives, its forgetting classes were previously in the remaining set. The authors project the new representation matrix against the existing forgetting bases, discard the overlapping part, and run SVD on the residual to get new, orthogonal bases that expand S_f. S_r contracts accordingly. Because the backbone stays frozen and only subspaces and LoRA parameters change, processing a new deletion request is cheap.

Evaluation covers a static machine unlearning setting with varying numbers of removed classes and an incremental setting where 20 classes are removed per task across four tasks, using both classification benchmarks and face recognition benchmarks, with accuracy on remaining classes, accuracy on forgetting classes, and an H-Mean harmonic trade-off metric.

Why This Matters

Impact on research. The paper reframes the evaluation of unlearning: rather than checking only whether forgetting-class accuracy drops, it argues for testing whether erased information can be reconstructed or recovered by re-fitting a classifier head. That gives the field a stricter standard for what counts as a successful deletion and connects unlearning to subspace geometry and gradient projection ideas borrowed from multi-task and continual learning.

Real-world applications:

• Compliance with deletion mandates such as GDPR's "right to be forgotten" for models trained on personal data.
• Face recognition systems that must remove a specific enrolled identity on request while continuing to recognize all other identities.
• Medical imaging models where a patient or a diagnostic category must be withdrawn from a deployed model.
• Content or safety policy enforcement, where restricted classes or concepts are removed from a model as policies evolve.

Industry relevance. Deployed systems face sequential deletion requests and evolving regulation (the paper cites OpenAI's safety policies and the EU AI Act). A method that modifies only 1.28% of parameters and keeps the backbone frozen keeps the cost of each deletion request low, which matters for providers serving many requests over time. The requirement that forgetting be irreversible also matters because a recoverable deletion is not a real deletion from a legal or security standpoint.

Future Directions

1. Extending the framework beyond class-level unlearning in vision classifiers to other modalities and to concept erasure in generative models, which the paper explicitly lists as a separate scenario it does not target.
2. Reporting the remaining experiments — the semantic residual analysis described in Section 5.4.1 and the additional classification benchmarks — to fully quantify how much private information survives.
3. Investigating how the subspace expansion behaves over very long task sequences, since the paper's incremental experiments cover four tasks and the authors note that baseline forgetting degrades as the sequence grows.
4. Determining how subspace decomposition and the SVD rank threshold should be chosen for much larger pre-trained models and much larger numbers of classes than those tested.

Target Audience

Researchers and graduate students working on machine unlearning, continual learning, or parameter-efficient fine-tuning of pre-trained vision models. It is also relevant to practitioners who must implement deletion requests in deployed recognition or classification systems, and to readers interested in geometric approaches to controlling what a network learns and forgets. Readers without background in linear algebra subspaces, LoRA, or unlearning benchmarks will find the method sections dense.

Authors’ abstract

Incremental unlearning (IU) is critical for pre-trained models to comply with sequential data deletion requests, yet existing methods primarily suppress parameters or confuse knowledge without explicit constraints on both feature and gradient level, resulting in \textit{superficial forgetting} where residual information remains recoverable. This incomplete forgetting risks security breaches and disrupts retention balance, especially in IU scenarios. We propose FG-OrIU (\textbf{F}eature-\textbf{G}radient \textbf{Or}thogonality for \textbf{I}ncremental \textbf{U}nlearning), the first framework unifying orthogonal constraints on both features and gradients level to achieve deep forgetting, where the forgetting effect is irreversible. FG-OrIU decomposes feature spaces via Singular Value Decomposition (SVD), separating forgetting and remaining class features into distinct subspaces. It then enforces dual constraints: feature orthogonal projection on both forgetting and remaining classes, while gradient orthogonal projection prevents the reintroduction of forgotten knowledge and disruption to remaining classes during updates. Additionally, dynamic subspace adaptation merges newly forgetting subspaces and contracts remaining subspaces, ensuring a stable balance between removal and retention across sequential unlearning tasks. Extensive experiments demonstrate the effectiveness of our method.