CD^2: Constrained Dataset Distillation for Few-Shot Class-Incremental Learning

AI summary

Overview

• Research area: Computer Vision — specifically Continual Learning, with focus on Few-Shot Class-Incremental Learning (FSCIL).
• Technical level: Advanced (requires familiarity with incremental learning, knowledge distillation, and metric learning).
• Scope: A framework that applies dataset distillation to build a compact, knowledge-rich memory and uses a dual-constraint distillation loss to preserve old-class knowledge during incremental sessions.

What This Paper Is About

Few-Shot Class-Incremental Learning (FSCIL) requires a model to keep learning new classes over time from only a handful of examples per class, without forgetting what it previously learned. Existing approaches store an external memory of past data (real samples, generated samples, or prototypes), but they compress knowledge crudely — diluting the most important class-discriminative details and treating the memory as if it were equivalent to real new data, which causes a distribution shift. This paper proposes CD², which uses dataset distillation to synthesize information-dense memory samples and a dual-constraint distillation loss to keep old-class distributions stable across sessions.

Key Contributions

1. Introducing dataset distillation to FSCIL. CD² is the first framework (to the authors' knowledge) to bring dataset distillation into the FSCIL pipeline, enabling a memory that captures critical class-related knowledge rather than diluted or redundant information.
2. Dataset Distillation Module (DDM). A module that synthesizes a small set of condensed samples per class by minimizing the Maximum Mean Discrepancy (MMD) between the feature distribution of synthetic and real data, producing memory items that carry the essential class clues.
3. Distillation Constraint Module (DCM). A loss combining a Feature Retention (FR) term that aligns old-class output logits between the previous and current session, and a Structure Retention (SR) term inspired by relational knowledge distillation that preserves inter-sample structural relationships — jointly reducing covariate shift.
4. Strong empirical results. CD² achieves state-of-the-art average accuracy on three benchmark datasets (CIFAR100, mini-ImageNet, CUB200), with particularly slow accuracy decay across incremental sessions.

Main Findings

• State-of-the-art average accuracy. On CIFAR100, CD² reaches 68.67% average accuracy, outperforming NC-FSCIL (67.50%), Revisiting-FSCIL (67.02%), and CABD (66.14%). It leads in every single session, not just the average.
• Slower accuracy decay. CD² improves on NC-FSCIL by 0.80% in the base session and by 2.60% to 0.39% across incremental sessions, indicating more stable retention of old knowledge as sessions progress.
• DDM outperforms prototype-based memory. Compared to prototype computing with the same DCM, DDM improves accuracy by more than 0.44% in the first incremental session and more than 1.33% in the final session.
• DCM components are complementary. Using SR loss alone risks displacement of feature representations; FR loss alone is too rigid and limits adaptation to new classes. Using both together yields the best results (see Table 2: 79.42% first / 57.36% final vs. lower values with single losses).
• Diminishing returns from K. Increasing the number of synthetic samples per class K improves accuracy, but effects plateau at K ≥ 2, so the authors set K = 2 to balance performance and resource cost.
• t-SNE visualization confirms knowledge transfer. Synthetic samples cluster tightly within their own classes and novel classes have closer inter-class distances than base classes, showing that the model successfully encodes class-discriminative features into the distilled memory.

Methodology in Plain English

The framework operates in two phases.

Base session: The model is trained normally on a large, well-labeled dataset (60 classes for CIFAR100 and mini-ImageNet). After training, instead of storing raw or random samples, the authors synthesize a small distilled set for each class by optimizing synthetic images so that their feature distribution matches the real data's feature distribution (measured by MMD). These distilled samples go into a memory buffer.

Incremental sessions: The backbone is frozen, and only the classifier is fine-tuned on the new session's few examples plus the memory. Two extra losses are added alongside the standard cross-entropy:

• The Feature Retention (FR) loss forces the classifier's output vectors for memory samples to stay close to what the previous session's model produced, effectively anchoring old-class decision boundaries.
• The Structure Retention (SR) loss, adapted from relational knowledge distillation, preserves the geometric relationships (angles between feature triplets) among memory samples, letting the model adjust more flexibly while still retaining old knowledge.

The weighting α on the SR loss is set adaptively (growing with the number of accumulated classes), while β on FR is fixed at 0.1. After each session, the DDM runs again to produce a distilled set for the just-learned classes, which is appended to the memory.

Why This Matters

Impact on research: This work shows that dataset distillation — mainly studied in data-efficient training — can be repurposed for the memory construction problem in continual learning. It reframes memory not as "a few random old samples" or "a mean prototype" but as a learned, optimized summary of class knowledge, opening a new axis for FSCIL method design.

Real-world applications:

• Robotics: robots that must recognize new objects or terrain with only a few demonstrations while still remembering previously learned environments.
• Healthcare: medical imaging systems that expand to new disease categories as they are discovered, using only a few annotated scans per new condition.
• Remote sensing: satellite or drone systems that continuously incorporate new land-cover or object classes without retraining from scratch on all historical imagery.
• Consumer devices / edge AI: on-device personalization where privacy and storage constraints prohibit keeping large memories of raw user data — distilled synthetic samples avoid direct data retention.

Industry relevance: The approach targets a practical pain point — deploying models that must evolve over time under tight data and storage budgets. Because the backbone is frozen after the base session, the method is computationally light and privacy-friendlier than replaying raw user data, which makes it attractive for edge deployment and regulated industries.

Future Directions

• Scaling to larger backbones and higher-resolution data. The experiments use ResNet12/ResNet18; whether distillation-based memory still helps with transformer-scale backbones is untested.
• Extending beyond frozen-backbone protocols. The method freezes the backbone after the base session; relaxing this while avoiding catastrophic forgetting remains an open question.
• Adaptive memory size and distillation frequency. K is fixed at 2 and distillation runs every session — smarter policies for when and how much to distill could improve efficiency.
• Handling harder real-world conditions. Domain shift, class imbalance, noisy labels, or streaming data with non-stationary distributions are not addressed and would stress-test the covariate-shift mitigation claims.

Target Audience

Researchers and graduate students working on continual learning, incremental learning, or few-shot learning, as well as practitioners building systems that must adapt to new classes over time with limited data — particularly those in computer vision, edge AI, and privacy-sensitive domains like healthcare. A solid grounding in deep learning and familiarity with knowledge distillation and prototype-based methods is assumed.

Authors’ abstract

Few-shot class-incremental learning (FSCIL) receives significant attention from the public to perform classification continuously with a few training samples, which suffers from the key catastrophic forgetting problem. Existing methods usually employ an external memory to store previous knowledge and treat it with incremental classes equally, which cannot properly preserve previous essential knowledge. To solve this problem and inspired by recent distillation works on knowledge transfer, we propose a framework termed \textbf{C}onstrained \textbf{D}ataset \textbf{D}istillation (\textbf{CD$^2$}) to facilitate FSCIL, which includes a dataset distillation module (\textbf{DDM}) and a distillation constraint module~(\textbf{DCM}). Specifically, the DDM synthesizes highly condensed samples guided by the classifier, forcing the model to learn compacted essential class-related clues from a few incremental samples. The DCM introduces a designed loss to constrain the previously learned class distribution, which can preserve distilled knowledge more sufficiently. Extensive experiments on three public datasets show the superiority of our method against other state-of-the-art competitors.