Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling

AI summary

Overview

Research area: Computer Vision — dataset distillation (DD) under long-tailed, class-imbalanced data distributions.

Technical level: Advanced. The paper builds on trajectory-matching and distribution-matching distillation literature and formulates its method through Batch Normalization (BN) statistics alignment, class-reweighted losses, and a uni-level (single-level) optimization objective.

One-sentence scope: The paper replaces bi-level trajectory matching with a uni-level statistical alignment framework that debiases the expert models used for image recovery and relabeling, and demonstrates accuracy gains over state-of-the-art long-tailed dataset distillation baselines on four benchmarks.

What This Paper Is About

Dataset distillation compresses a large training set into a small synthetic set, but nearly all existing methods assume balanced class frequencies. Under long-tailed distributions, where a few head classes dominate and tail classes are sparse, these methods produce synthetic sets that over-represent head classes and inherit biased feature statistics. The authors target the one prior method that explicitly addresses this setting, DAMED, which they argue still relies on a biased expert, mishandles mid-frequency classes, and carries heavy computation and GPU memory costs. Their goal is a cheaper, explicitly debiased framework that recovers synthetic images through fair statistics alignment and assigns them unbiased soft labels.

Key Contributions

1. A reformulation of long-tailed dataset distillation as a uni-level statistical alignment framework rather than bi-level trajectory matching, on the argument that debiasing and trajectory matching are fundamentally in tension.
2. Expert model debiasing for both an observer model (used for statistics-based recovery) and a teacher model (used for soft relabeling), via a mixture consistency loss and a dynamically weighted class-wise debias loss.
3. Fair BN statistics recalibration using a full forward pass with dynamically adjusted momentum, combining per-class accumulation (to remove intra-class bias) with uniform averaging over classes (to remove inter-class bias).
4. Confidence-guided multi-round initialization of the synthetic set, selecting high-confidence, diverse augmentations scored by the teacher model, with zero-filled placeholders to equalize batch structure across classes.

Main Findings

• Headline accuracy gains: Under IPC=10 and IF=10, the method improves top-1 accuracy by 15.6% on CIFAR-100-LT (47.1 ± 0.1 versus DAMED's 31.5 ± 0.2) and 11.8% on Tiny-ImageNet-LT (37.8 ± 0.4 versus DAMED's 26.0 ± 0.3).
• Consistent gains across imbalance levels: On CIFAR-10-LT the method reaches 63.6 ± 0.5 (IF=10, IPC=10), 62.7 ± 0.1 (IF=100, IPC=10) and 68.8 ± 0.4 (IF=100, IPC=50), against DAMED's 58.1 ± 0.3, 53.4 ± 0.1 and 64.0 ± 0.9.
• Large-scale benchmarks: On ImageNet-LT, the method reports 24.7 ± 0.4 (IF=5, IPC=10) and 23.5 ± 0.2 (IF=10, IPC=10) versus DAMED's 20.8 ± 0.2 and 20.3 ± 0.1.
• Severe imbalance: With IF=256 on Tiny-ImageNet-LT the method reports 29.8 ± 0.1 at IPC=20 versus DAMED's 18.0 ± 0.5. With ResNet-50 evaluation on ImageNet-LT under IF=10 it reports 48.2 ± 0.7 (IPC=10) and 48.9 ± 0.2 (IPC=20), versus DAMED's 17.2 ± 0.2 and 17.9 ± 0.2.
• Extreme compression (IPC=1): The method reports 44.8 ± 0.3 on CIFAR-10-LT (IF=100), 31.8 ± 0.1 on CIFAR-100-LT (IF=50) and 20.1 ± 0.2 on Tiny-ImageNet-LT (IF=100), versus DAMED's 24.1 ± 0.5, 7.8 ± 0.3 and 6.0 ± 0.3. The paper describes this as over 2× accuracy improvement over all baselines on most datasets.
• Cross-architecture transfer: The same distilled CIFAR-10-LT set (IF=100, IPC=10) yields 62.7 (ConvNet-3), 64.6 (VGG-11), 58.7 (ResNet-18) and 56.5 (AlexNet), compared with DAMED's 53.4, 29.7, 43.6 and 37.9.
• Ablation (CIFAR-100-LT, IF=50): Removing model debiasing drops accuracy to 31.7/32.3/32.8 across IPC=10/20/50; removing statistics recalibration gives 40.9/41.8/42.1; removing adapted initialization gives 40.8 at IPC=10; the full method gives 42.1/43.4/44.2.
• Efficiency: At IPC=1, expert model training takes 2395s (CIFAR-10-LT, IF=100) versus DAMED's 31388s, and distilled data synthesis takes 118s versus DAMED's 30141s. On CIFAR-100-LT (IF=50) the figures are 2183s versus 26269s and 273s versus 29328s.
• Memory: Peak GPU memory at IPC=10 is 3.1 GB on CIFAR-100-LT versus DAMED's 10.2 GB, and 6.1 GB on Tiny-ImageNet-LT at IPC=10 versus more than 24.0 GB.
• Class-wise behavior: Per-class accuracy plots on CIFAR-10-LT (IF=100, IPC=10 and IPC=20) show DAMED underperforming on tail classes and suppressed on mid-frequency classes, which the authors attribute to biased expert training and frequency-adjusted trajectory matching.

Methodology in Plain English

The authors keep a small, class-balanced synthetic set (IPC images for every class) but change how it is produced. Instead of simulating a long training trajectory and matching it, they train two models on the real long-tailed data: an observer that supplies feature statistics and a teacher that supplies soft labels.

Both models are debiased during training with two losses. A mixture consistency loss compares two augmented views of the same image and pushes their representations toward each other, which stabilizes feature learning. A debias loss blends a class-frequency-reweighted cross-entropy term with a plain cross-entropy term, using a weight that grows over training so that minority classes matter more later while early training stays stable.

For the observer, the authors then recompute BN statistics. Standard BN uses a fixed momentum, so recent batches dominate and tail-class samples get washed out. They instead run a full forward pass over the real data and update each class's running mean with a momentum proportional to how many of that class's samples have been seen so far, so every sample contributes equally regardless of order. They then average the per-class means uniformly across classes, which removes class-frequency effects from the alignment target.

Synthetic images start from real images. For each real image, several augmentations (for example crops) are generated and scored by the teacher using negative cross-entropy. Selection proceeds in rounds: in each round every image offers its most confident unused augmentation, and top-scoring candidates fill the class quota, which spreads selection across many source images instead of concentrating on a few. Classes with fewer real images than the largest class get zero-initialized placeholders so batch shapes stay consistent.

Finally, images are optimized so their BN statistics in the observer model match the recalibrated real-data statistics (both globally and per class), and each image is relabeled by the teacher to obtain a soft label. Evaluation trains a student model from scratch on the distilled set with a combined loss of cross-entropy against the hard one-hot label plus squared error against the soft teacher label. Students are trained for 1000 epochs; experiments use a depth-3 ConvNet for CIFAR-10-LT and CIFAR-100-LT, a depth-4 ConvNet for Tiny-ImageNet-LT and ImageNet-LT, ResNet-50 for ImageNet-LT under severe imbalance, and are repeated five times, primarily on a single NVIDIA RTX 3090 GPU. The long-tailed versions of the benchmarks are built with an exponential decay schedule |D_c| = |D_0| φ^c where φ^c = β^{-(c/(C-1))} and β is the imbalance factor.

Why This Matters

Impact on research. The paper argues that debiasing and trajectory matching are structurally incompatible, because reweighting or logit correction changes the expert's optimization path and breaks the premise of trajectory matching, while post-hoc correction cannot control per-class representation quality. It proposes statistics alignment as an alternative objective that admits explicit debiasing, and it shows that the standard fixed-momentum BN update is itself a source of bias under long-tailed data — a point that applies beyond distillation.

Real-world applications.

• Training on rare medical conditions or adverse events, where head categories dominate and rare cases carry most of the diagnostic value.
• Industrial defect detection and rare-failure monitoring, where defective samples are scarce relative to normal production output.
• Autonomous driving and robotics datasets, where safety-critical scenarios (pedestrians at night, unusual road geometry) are the rarest classes.
• Edge and embedded deployment, where the distilled set and the reported 3.1–6.1 GB peak memory footprint make on-device or single-GPU retraining practical.

Industry relevance. The runtime and memory tables show reductions of roughly an order of magnitude in both expert training and synthesis time relative to DAMED, which matters when distillation is a repeated preprocessing step rather than a one-off experiment. Cross-architecture results also matter operationally, since a single distilled set can be reused across different deployment backbones without re-running distillation.

Future Directions

• Whether the uni-level statistical alignment objective can be extended to other compression regimes, such as dataset condensation combined with pruning or quantization, is not explored.
• The paper reports ImageNet-LT results with ConvNet variants and ResNet-50; behavior with other large-scale backbones (for example vision transformers) is not reported.
• The recalibrated BN statistics require a full forward pass over the real dataset with frozen parameters, and how the approach scales to datasets far larger than ImageNet-1K is not reported.
• The soft relabeling step depends on a teacher trained on the same long-tailed data; whether stronger or externally pretrained teachers change the observed gains is left open.
• The paper does not report results on non-CIFAR, non-ImageNet domains such as medical or remote-sensing long-tailed benchmarks, leaving domain transfer unverified.

Target Audience

Researchers and graduate students working on dataset distillation, dataset condensation, or coreset selection, particularly those interested in imbalanced and long-tailed regimes. It also suits practitioners who need to compress large imbalanced training corpora under GPU memory and time constraints, and readers of the broader long-tailed recognition literature interested in how BN statistics and reweighted losses interact with data synthesis rather than with standard training. Familiarity with Batch Normalization, soft labels, and bi-level optimization is assumed; the paper reads as advanced material.

Authors’ abstract

Dataset distillation creates a small distilled set that enables efficient training by capturing key information from the full dataset. While existing dataset distillation methods perform well on balanced datasets, they struggle under long-tailed distributions, where imbalanced class frequencies induce biased model representations and corrupt statistical estimates such as Batch Normalization (BN) statistics. In this paper, we rethink long-tailed dataset distillation by revisiting the limitations of trajectory-based methods, and instead adopt the statistical alignment perspective to jointly mitigate model bias and restore fair supervision. To this end, we introduce three dedicated components that enable unbiased recovery of distilled images and soft relabeling: (1) enhancing expert models (an observer model for recovery and a teacher model for relabeling) to enable reliable statistics estimation and soft-label generation; (2) recalibrating BN statistics via a full forward pass with dynamically adjusted momentum to reduce representation skew; (3) initializing synthetic images by incrementally selecting high-confidence and diverse augmentations via a multi-round mechanism that promotes coverage and diversity. Extensive experiments on four long-tailed benchmarks show consistent improvements over state-of-the-art methods across varying degrees of class imbalance. Notably, our approach improves top-1 accuracy by 15.6% on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT under IPC=10 and IF=10. Codes are available at https://github.com/2018cx/RLDD.