Compensating Distribution Drifts in Class-incremental Learning of Pre-trained Vision Transformers

AI summary

Overview

Research area: Class-incremental learning (CIL) with pre-trained vision transformers (ViTs), specifically the problem of feature-distribution drift that occurs when a shared backbone is sequentially fine-tuned.

Technical level: Advanced. The paper combines regularized least-squares estimation, propagation of Gaussian moments through learned operators, Monte Carlo sampling, knowledge distillation, and references to neural tangent kernel (NTK) theory.

Scope: The paper introduces a learned "latent space transition operator" that maps feature distributions from an older backbone to a newer one so that classifier refinement stays accurate as tasks accumulate.

What This Paper Is About

Sequentially fine-tuning (SeqFT) a pre-trained ViT backbone and then refining the classifier with approximate Gaussian distributions of class features is an effective CIL strategy, but the shared parameters keep changing, so the stored feature distributions for old classes no longer match the updated model. The authors propose Sequential Learning with Drift Compensation (SLDC), which learns the transformation of the feature space between consecutive tasks and uses it to push old class distributions into the new feature space. The stated goal is to compensate for drift after it occurs, rather than only trying to prevent it through distillation, model ensembles, or gradient projection.

Key Contributions

1. A CIL methodology built on a transition operator. The authors formalize a "latent space transition operator" mapping the feature extractor at task t−1 to the feature extractor at task t, and use its approximation to compensate old-class distributions.

2. Two learnable operators plus distillation-enhanced variants. α1-SLDC learns a linear operator by solving a regularized least-squares problem; α2-SLDC learns a "weak-nonlinear" transformation assumed to lie between purely linear and fully nonlinear mappings. Their distillation-enhanced counterparts are β1-SLDC and β2-SLDC.

3. Auxiliary Data Enrichment (ADE). An unlabeled auxiliary data scheme that improves operator estimation when task data is scarce, without storing task-relevant data from previous tasks, so the exemplar-free CIL constraint is maintained.

4. Extensive benchmarking with released code. Experiments on four benchmarks with two pre-trained ViT-B/16 backbones show that combining distillation with SLDC brings SeqFT close to joint training (the empirical upper bound), with code at the listed GitHub repository.

Main Findings

• Vanilla SeqFT forgets severely. With the MoCo-V3 backbone, SeqFT reaches Last-Acc of 64.40% on CUB-200 and 61.37% on ImageNet-R (Table 1).

• Weak-nonlinear compensation gives the largest single-method gain. α2-SLDC lifts CUB-200 to 78.98%, described as a +14.58% absolute gain over SeqFT with MoCo-V3.

• The weak-nonlinear operator beats both linear and fully nonlinear alternatives. Without ADE, α2-SLDC outperforms linear α1-SLDC and the nonlinear MLP-based baseline (MLPDC) on fine-grained datasets: 77.53% vs. 61.01% on Cars-196 and 78.98% vs. 70.42% on CUB-200 with MoCo-V3.

• Distillation pairs well with SLDC. SeqKD improves SeqFT by +12.57% Last-Acc on CUB-200 with the supervised ImageNet-21K (Sup-21K) backbone. β1-SLDC (distillation-enhanced α1-SLDC) nearly matches α2-SLDC, at 80.55% vs. 78.98% on CUB-200 with MoCo-V3.

• Robustness across datasets and backbones. α2-SLDC and β2-SLDC outperform MLPDC by +6.52% on Cars-196 and +2.17% on CIFAR-100 with Sup-21K pre-training, which the authors present as support for their hypothesis that the right operator lies between the linear and nonlinear extremes.

• ADE helps most when task data is limited. α1-SLDC is unstable with Sup-21K pre-training, with Last-Acc dropping on CUB-200 (71.49% vs. SeqFT's 76.57%) and Cars-196 (46.78% vs. 54.53%); adding ADE yields a +26.23% improvement on Cars-196 relative to the non-ADE counterpart.

• Near-parity with joint training. On 10-task CIL scenarios across two pre-trained models and four datasets, the reported accuracy discrepancies relative to joint training fall within +0.50% to −3.29%. An example with Sup-21K: β1-SLDC + ADE reaches 87.18% versus SeqKD's 86.75% on CUB-200 (reported as ↑0.43), against joint training's 88.43%.

• Long-sequence behavior. On a 20-task sequence, α2-SLDC consistently outperforms α1-SLDC when neither distillation nor ADE is used, and adding both techniques improves all SLDC variants; MLPDC is reported as performing particularly poorly on Cars-196 and CUB-200.

• Hybrid CIL setting. When CIFAR-100, Cars-196, CUB-200 and ImageNet-R are each treated as a distinct incremental task, SLDC methods beat SeqFT and MLPDC in all settings, and the gap between α1-SLDC and α2-SLDC narrows substantially because of the larger per-task datasets.

• Not all of the paper's technical material is in the provided content. The main text references Statements 1, 3 and 4 in the appendix (the closed-form Gaussian transformation claim and NTK-based claims about the operator), and Figures 7 to 9 in the appendix; these appendix items are not included in the supplied text.

Methodology in Plain English

The method has three phases.

1. Sequential fine-tuning. A pre-trained ViT backbone is adapted task by task using low-rank adaptation (LoRA), following the SLCA++ configuration, with an optional feature-based distillation loss that keeps the backbone's representations close to the previous task's backbone.

2. Distribution compensation. For the current task's data, the authors collect feature matrices from the old backbone (before fine-tuning) and the new backbone (after fine-tuning), normalize them column-wise, and learn the mapping between them.

   • Linear version (α1-SLDC): solve a regularized least-squares problem for a matrix A, with a re-weighting term that mixes A with the identity matrix when the task has too few samples (w = exp(−n_t / (α_temp · d))).
   • Weak-nonlinear version (α2-SLDC): combine a learnable matrix and an MLP with ReLU activation through learnable non-negative coefficients c1 + c2 = 1, with a regularizer pulling c1 toward 1 so the nonlinear part cannot dominate. The plain-English method section describes a two-layer MLP while the implementation details specify a three-layer MLP with hidden dimension matching the [cls] token.
   • Old-class Gaussians are then moved into the new space: the linear version transforms the mean as A·μ and the covariance as A·Σ·Aᵀ; the weak-nonlinear version draws N = 10d synthetic samples per class from each stored Gaussian, pushes them through the learned transformation, and re-estimates the mean and covariance (using N ≫ d Monte Carlo samples).

3. Classifier refinement. After each task, the classifier is retrained on synthetic features sampled from the compensated Gaussian distributions, improving cross-task decision boundaries.

Distillation-enhanced variants (β1-SLDC, β2-SLDC) add a feature distillation loss plus a feature-norm regularization loss to the cross-entropy objective, with balance coefficients γ_KD = 1.0 and γ_Norm = 0.1. ADE further improves operator estimation by adding unlabeled auxiliary data from arbitrary sources.

Experimental setup: two ViT-B/16 backbones (MoCo-V3 self-supervised pre-training on ImageNet-1K, and supervised pre-training on ImageNet-21K); LoRA rank 4, Adam optimizer with learning rate 10⁻⁴ and weight decay 3 × 10⁻⁵; λ_α1 = 10⁻⁴; default λ_α2 = 0.5 with (c1, c2) = (0.9, 0.1). Four benchmarks, each split into 10 disjoint tasks: CIFAR-100 (100 classes, 500 training samples per class), ImageNet-R (200 classes; 24,000 training and 6,000 test samples), CUB-200 (200 bird species, roughly 60 images per class, train and test split evenly) and Cars-196 (196 car types; 8,144 training and 8,040 testing images). Reported metrics are Inc-Acc (%) and Last-Acc (%), and all experiments use the PILOT framework with consistent random seeds.

Why This Matters

Impact on research. The paper reframes distribution drift as a transformation to be estimated and inverted rather than a problem to be suppressed only through regularization. It also empirically positions the best operator between linear and fully nonlinear mappings, and it argues that combining drift prevention (distillation) with drift compensation (SLDC) can approach the joint-training upper bound — an important target for exemplar-free CIL research with pre-trained models.

Real-world applications.

• Continual personalization of on-device or assistant models as new user categories, intents or content types appear over time.
• Medical imaging systems that must add new disease or lesion categories without storing prior patient data (the exemplar-free constraint matters for privacy).
• Industrial visual inspection, where new defect classes are introduced after deployment while older defect detectors must keep working.
• Robotics and autonomous systems that encounter new object categories in sequence and cannot retain large archives of past raw data.

Industry relevance. SeqFT-style approaches avoid storing exemplars and avoid task-identity prediction, which reduces privacy exposure and inference overhead compared with adapter-selection or prompt-pool methods. Parameter-efficient LoRA tuning plus an added post-hoc compensation step is a relatively low-cost addition to existing pipelines, and the reported near-joint-training accuracy suggests deployment quality close to retraining on all data — a costly option in practice.

Future Directions

• Better operator estimation under data scarcity. ADE improves results, but the paper leaves open how to choose auxiliary data sources and how much auxiliary data to use; its sweeps vary ADE data (CIFAR-10, SVHN, ImageNet) and sample sizes from 512 to 2048. What happens when auxiliary data is unavailable or mismatched is not reported.

• Stabilizing the linear variant. α1-SLDC shows instability with the Sup-21K backbone (for example, CUB-200 at 71.49% versus SeqFT's 76.57%, and Cars-196 at 46.78% versus 54.53%), and the paper notes this can be mitigated by distillation or ADE. A principled fix rather than mitigation is an open question.

• Theory for the transition operator. The main text points to NTK-based statements in the appendix about the characteristics of the transition operator; these are not included in the provided content, so their scope and generality remain to be examined.

• Broader integration and settings. The authors state that SLDC can be combined with other techniques such as CoMA and CoFiMA (which use exponential moving averages on model parameters), and that it applies wherever the base CIL method is SeqFT or SeqKD. Extending evaluation beyond the four benchmarks, the 10- and 20-task sequences, and the hybrid setting, and to other architectures or modalities, is a natural next step.

Target Audience

Researchers and graduate students working on continual or class-incremental learning with pre-trained vision transformers; practitioners who want a drop-in post-hoc compensation step to keep an incrementally fine-tuned model close to joint-training accuracy without storing past data; and readers interested in Gaussian feature modeling, parameter-efficient adaptation, or the interplay between knowledge distillation and distribution compensation. A background in CIL, ViT architectures and basic linear algebra and statistics is helpful, since the method involves regularized least-squares solutions and Gaussian moment propagation.

Authors’ abstract

Recent advances have shown that sequential fine-tuning (SeqFT) of pre-trained vision transformers (ViTs), followed by classifier refinement using approximate distributions of class features, can be an effective strategy for class-incremental learning (CIL). However, this approach is susceptible to distribution drift, caused by the sequential optimization of shared backbone parameters. This results in a mismatch between the distributions of the previously learned classes and that of the updater model, ultimately degrading the effectiveness of classifier performance over time. To address this issue, we introduce a latent space transition operator and propose Sequential Learning with Drift Compensation (SLDC). SLDC aims to align feature distributions across tasks to mitigate the impact of drift. First, we present a linear variant of SLDC, which learns a linear operator by solving a regularized least-squares problem that maps features before and after fine-tuning. Next, we extend this with a weakly nonlinear SLDC variant, which assumes that the ideal transition operator lies between purely linear and fully nonlinear transformations. This is implemented using learnable, weakly nonlinear mappings that balance flexibility and generalization. To further reduce representation drift, we apply knowledge distillation (KD) in both algorithmic variants. Extensive experiments on standard CIL benchmarks demonstrate that SLDC significantly improves the performance of SeqFT. Notably, by combining KD to address representation drift with SLDC to compensate distribution drift, SeqFT achieves performance comparable to joint training across all evaluated datasets. Code: https://github.com/raoxuan98-hash/sldc.git.