Learning with Preserving for Continual Multitask Learning

AI summary

Overview

Research area: Continual learning and multitask learning for computer vision and sensor time-series models (task-incremental learning on a shared input stream).

Technical level: Advanced. The paper builds on continual learning baselines and includes a Reproducing Kernel Hilbert Space (RKHS) argument for why its preservation loss works, though the experimental narrative is accessible.

Scope: The paper defines a setting called Continual Multitask Learning (CMTL), where a model learns a sequence of tasks over the same input domain without access to past task labels, and proposes a replay-free framework, Learning with Preserving (LwP), whose Dynamically Weighted Distance Preservation (DWDP) loss keeps the geometry of the shared latent space stable across tasks.

Note on provenance: this is arXiv:2511.11676v1 [cs.LG], dated 11 Nov 2025, by Hanchen David Wang, Siwoo Bae, Zirong Chen, and Meiyi Ma (Vanderbilt University). The paper states it was accepted at AAAI-2026, and code is available at https://github.com/AICPS-Lab/lwp.

What This Paper Is About

In real systems such as autonomous driving or medical imaging, a model often needs to learn a new label type from the same sensor stream it already uses — for example, after learning to detect traffic signs, it may later be asked to classify traffic lights or vehicle types from the same camera feed. This is hard because the model is only given labels for the new task and never sees the old tasks' labels again, so it tends to overwrite previously learned abilities. The authors call this setting Continual Multitask Learning (CMTL) and aim to build one shared representation that supports all tasks sequentially, without a stored replay buffer of past data.

Key Contributions

1. Formalizing CMTL. The paper defines CMTL as a sequential learning problem over tasks $\mathcal{T}_1, \dots, \mathcal{T}_T$ sharing an input domain $\mathcal{X}$ but with distinct label spaces $\mathcal{Y}_t$, including both stationary and non-stationary input distributions. It argues that conventional continual learning methods are designed to isolate task-specific knowledge and therefore underperform in this setting.

2. Learning with Preserving (LwP). A replay-free framework with a shared feature extractor and one task-specific head per task. At each step, the previous model is frozen as a teacher; the new model is trained with a weighted sum of a supervised current-task loss ($\mathcal{L}{\text{cur}}$), a distillation loss on pseudo-labels from the frozen teacher ($\mathcal{L}{\text{old}}$), and the preservation loss ($\mathcal{L}_{\text{DWDP}}$).

3. The Dynamically Weighted Distance Preservation (DWDP) loss. Instead of preserving task outputs, DWDP regularizes pairwise distances between latent representations so the new latent space $Z'$ keeps the geometry of the frozen latent space $Z$. A per-batch dynamic mask $m_{ij}$ is set to 1 only when two samples share the same label in the current task (0 otherwise), so preservation applies to the complete intra-class pairwise structure and does not fight the class-separation objective. The paper contrasts this with PODNet (uniform spatial feature preservation), RKD (all pairwise distances regardless of class), and Asadi et al. (prototype–sample distances only).

4. A theoretical justification via kernels. Minimizing the preservation loss is equivalent to matching Gram matrices $K(Z) \approx K(Z')$. The paper argues that preserving the kernel matrix implies an isometry $T$ in the RKHS such that $\phi(\mathbf{z}'_i) = T(\phi(\mathbf{z}_i))$, so any learning problem defined on $Z$ has an equivalent solution on $Z'$ with identical risk.

Main Findings

• LwP is the only method to beat the single-task learning (STL) baseline. In Table 2, LwP reaches 78.299 ± 3.828 on BDD100k (3 tasks), 73.484 ± 8.019 on CelebA (10 tasks), 88.242 ± 12.010 on PhysiQ (3 tasks), and 66.482 ± 3.138 on FairFace (3 tasks), compared with STL at 75.123 ± 6.543, 72.230 ± 7.297, 87.167 ± 10.102, and 64.435 ± 3.660 respectively. The paper states this is the only approach to surpass the STL baseline.

• Robustness to non-stationary distributions on BDD100k. Under Weather Shift, Scene Shift, Time-of-Day Shift, and Combined Shift, LwP scores 77.937 ± 4.041, 78.198 ± 3.842, 76.820 ± 5.331, and 74.004 ± 11.268, versus STL at 76.760 ± 5.210, 76.787 ± 5.183, 76.418 ± 5.567, and 76.751 ± 5.180. The paper notes most baselines drop under compounding shifts, while LwP retains an advantage; it acknowledges LwP's Combined Shift standard deviation is large (11.268).

• Gains grow with model and image scale. On CelebA (10 tasks, Table 3), LwP scores 67.388 ± 11.125 with ResNet50 at 32×32, 69.432 ± 10.416 with ResNet101 at 32×32, and 85.064 ± 5.388 with ResNet50 at 224×224. The paper reports that at 224×224 this is about 15 percentage points above the runner-up (the highest non-LwP value in that column is DVC at 70.921 ± 13.823). The authors also observe that larger models at the same input size underperform relative to the ResNet18 configuration, attributing this to insufficient input information leading to overfitting.

• Backward Transfer improves across all benchmarks. Using BWT = $\frac{1}{T-1}\sum_{i=1}^{T-1} R_{T,i} - R_{i,i}$, the paper reports that LwP outperforms all baselines on BWT for every benchmark, consistent with the visualization showing task accuracies maintained from their initial training.

• Ablations validate the design choices (PhysiQ, Table 4). The full model scores 88.2 ± 12.0 with dynamic weighting versus 86.0 ± 12.3 without it; removing $\mathcal{L}_{\text{old}}$ gives 87.1 ± 9.44 (with dynamic weighting) and 85.4 ± 12.1 (without). Alternatives underperform: cosine similarity 85.4 ± 13.1 / 84.1 ± 14.4, RBF kernel 84.5 ± 13.7 / 84.8 ± 14.5, IRD (Co2L) 86.4 ± 11.5 / 79.9 ± 17.1, and RKD 85.1 ± 13.3 / 85.9 ± 11.9.

• Hyperparameter robustness. The paper reports that LwP's performance stays high and stable across a range of hyperparameter values, and that its entire performance range surpasses the best-performing baseline on the BDD100k weather shift scenario.

• Preservation is demonstrated on a constructed toy problem. In a concentric-circle task followed by an XOR task, the representation space degrades without $\mathcal{L}_{\text{DWDP}}$; with it, the structure needed for the later XOR task is retained.

Methodology in Plain English

The setup is a sequence of tasks over the same input space. For each new task, the researchers copy the previous model, freeze it, and attach a new randomly initialized linear head for the new label set. Three losses then drive training: (1) ordinary supervised training on the new task's labels; (2) distillation, where the frozen teacher produces pseudo-labels for old tasks and the student is trained to match them; and (3) the DWDP loss.

The DWDP loss works on latent representations. For batches of data, it computes pairwise distances between points in the current model's representation and compares them to the corresponding pairwise distances from the frozen model's representation, penalizing squared differences. A mask disables this penalty whenever two samples have different labels in the current task, so the model is still free to push classes apart. The authors' claim is that locking down those intra-class distance relationships preserves "implicit knowledge" — the geometry of the feature space — so old tasks remain solvable and future tasks can reuse the structure.

Experiments compare LwP against ten continual learning baselines (LwF, oEWC, ER, SI, GSS, FDR, DER, DERPP, DVC, OBC), plus STL and naive fine-tuning, on four datasets spanning two modalities: BDD100K, CelebA, PhysiQ, and FairFace. Image datasets use a ResNet feature extractor with a linear head per task; PhysiQ, which is IMU sensor time-series, uses a 3-layer 1D-CNN. For CelebA and FairFace the baselines get a replay buffer of 512 samples, and for PhysiQ a buffer of 46, corresponding to roughly 2–3% of the training set; LwF and LwP use no buffer. Each model is trained five times with different random seeds, for 20 epochs with early stopping, at batch size 256 for image datasets and 32 for PhysiQ. For PhysiQ only the average accuracy at the final task iteration is compared, due to training instability from the smaller dataset. Additional MTL comparisons (basic simultaneous MTL, PCGrad, IMTL, NashMTL) and other results are reported in the appendix.

Why This Matters

Impact on research. The paper argues that standard continual learning benchmarks and Task-Incremental Learning protocols encourage methods that isolate task-specific knowledge, which is the wrong objective when tasks share an input domain and are supposed to share a representation. By reframing the target as preserving the geometry of latent space rather than preserving outputs, and by showing that a replay-free method can beat independently trained single-task models, it pushes back on the assumption that replay buffers are necessary for strong continual learning. It also connects representation preservation to kernel/RKHS arguments, giving a theoretical handle on why distance preservation should transfer to unseen tasks.

Real-world applications (as described in the paper):

• Autonomous driving: learning to detect traffic signs, then classify traffic lights, scene types, or vehicle types from the same camera feed, including under weather, scene, and time-of-day shifts.
• Medical imaging: after tumor classification, updating the model to recognize secondary characteristics such as tissue density or shape from the same patient scans.
• Wearable and sensor health monitoring: PhysiQ, IMU-based exercise quality human activity recognition, demonstrating the method is not image-specific.
• Privacy-constrained deployments: because LwP requires no replay buffer, it avoids storing and reusing raw past data, which the authors highlight for sensitive domains like healthcare.

Industry relevance. Avoiding the need to retrospectively annotate an entire existing dataset with new labels is framed as a practical cost argument. No replay buffer also means lower memory and storage overhead and fewer privacy compliance issues for deployed systems that must keep adapting to new labels from a live sensor stream.

Future Directions

• Scaling and architecture dependence. The paper finds that larger models at the same low input resolution do not outperform the ResNet18 configuration, attributing this to insufficient input information and overfitting; working out when scale helps versus hurts in CMTL remains open. The reported gap in the Combined Shift scenario (with a standard deviation of 11.268) also suggests room to improve under compounding distribution shifts.

• Beyond the intra-class restriction. The dynamic mask deliberately limits preservation to intra-class pairs of the current task, which the authors describe as a trade-off that "reduces the scope for preservation." Whether inter-class structure can be preserved without conflicting with separation objectives is not resolved.

• Broader and harder task sequences. The evaluation covers 3-task (BDD100k, PhysiQ, FairFace) and 10-task (CelebA) sequences; behavior on much longer task streams, and on task boundaries that are not clearly defined (as in task-free continual learning, discussed in related work), is not reported.

• Theory-to-practice gap. The RKHS isometry argument is developed for Gaussian kernels and pairs with the empirical observation that squared Euclidean distance performs better than the RBF kernel; the paper does not report a bound linking the preservation loss value to actual accuracy retention on real benchmarks.

Target Audience

Researchers and practitioners in continual learning, multitask learning, and representation learning, particularly those working on deployment scenarios where past data cannot be stored or re-annotated. Engineers building perception or sensor-processing systems that must acquire new label types over time will find the CMTL framing and the buffer-free loss practical, while theoretically inclined readers will be most interested in the kernel-based justification for distance preservation.

Authors’ abstract

Artificial intelligence systems in critical fields like autonomous driving and medical imaging analysis often continually learn new tasks using a shared stream of input data. For instance, after learning to detect traffic signs, a model may later need to learn to classify traffic lights or different types of vehicles using the same camera feed. This scenario introduces a challenging setting we term Continual Multitask Learning (CMTL), where a model sequentially learns new tasks on an underlying data distribution without forgetting previously learned abilities. Existing continual learning methods often fail in this setting because they learn fragmented, task-specific features that interfere with one another. To address this, we introduce Learning with Preserving (LwP), a novel framework that shifts the focus from preserving task outputs to maintaining the geometric structure of the shared representation space. The core of LwP is a Dynamically Weighted Distance Preservation (DWDP) loss that prevents representation drift by regularizing the pairwise distances between latent data representations. This mechanism of preserving the underlying geometric structure allows the model to retain implicit knowledge and support diverse tasks without requiring a replay buffer, making it suitable for privacy-conscious applications. Extensive evaluations on time-series and image benchmarks show that LwP not only mitigates catastrophic forgetting but also consistently outperforms state-of-the-art baselines in CMTL tasks. Notably, our method shows superior robustness to distribution shifts and is the only approach to surpass the strong single-task learning baseline, underscoring its effectiveness for real-world dynamic environments.