Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation

AI summary

Overview

Research area: Dataset distillation (a data-centric computer-vision technique for compressing datasets) combined with long-tailed recognition, with a focus on soft labels and their calibration.

Technical level: Advanced. The paper builds on a generalization bound, KL-divergence decompositions, and logit-adjustment theory, though the core idea is simple enough to follow at a conceptual level.

Scope in one sentence: This paper diagnoses why soft labels become biased when dataset distillation is applied to long-tailed data, traces that bias to two distinct sources, and proposes a lightweight post-hoc calibration module (ADSA) that corrects it across several distillation baselines and datasets.

What This Paper Is About

Dataset distillation compresses a large dataset into a small synthetic one that trains models nearly as well, but almost all existing work assumes balanced class distributions. On real-world long-tailed data, where a few head classes dominate and many tail classes are sparse, distillation produces biased synthetic images and biased soft labels, and tail-class accuracy collapses. This paper's goal is to explain the origin of that soft-label bias and remove it without redesigning the distillation pipeline.

Key Contributions

1. An imbalance-aware generalization bound. The authors derive a long-tailed version of the D3S generalization bound (Theorem 3.1), showing that under shared class-conditional distributions but differing class priors, the discrepancy term splits into two equivalent forms (Eq. 5 and Eq. 6) with additional terms that standard bounds omit.

2. Identification of two entangled bias sources. Through a controlled perturbation study that separates image synthesis from label generation, they show that soft-label bias comes from (a) the imbalanced distillation/labeling model and (b) the imbalanced distilled images themselves, formally written as an additive decomposition in Eq. 8.

3. ADSA, a post-hoc calibration module. The Adaptive Soft-label Alignment module reuses the distilled images as a hold-out set, measures class-wise average soft-label confidence, and optimizes a single logit-adjustment strength so that confidence is uniform across classes.

4. Broad empirical validation. ADSA is plug-and-play and improves SRe2L, GVBSM, EDC, MTT, and DREAM on CIFAR-10-LT, CIFAR-100-LT, and ImageNet-1k-LT, including at very small soft-label budgets.

Main Findings

• Soft-label bias is measurably harmful and has two separable causes. In the perturbation study on CIFAR-100 (20 tail classes, 80 head classes), config (1) with imbalanced images only, config (2) with an imbalanced labeling model only, and config (3) with both imbalanced all underperformed config (4), where both were balanced.

• Imbalanced labeling models hurt more than imbalanced images. Config (2), which used an imbalanced model for labeling, showed a greater performance decrease than config (1), which used only imbalanced distilled images.

• More imbalance produces less informative soft labels. Entropy of the soft labels increases as the dataset becomes more imbalanced, indicating a lack of class-discriminative information.

• Both mechanisms push confidence in the same biased direction. Imbalanced synthetic images and imbalanced labeling models both yield soft labels that are overconfident for head classes and underconfident for tail classes, a bias the evaluation model then inherits.

• The additive bias decomposition does not perfectly hold. The authors note that config (3), with both biases present, does not produce the worst performance, which is why they calibrate the entangled bias adaptively rather than subtracting each source separately.

• Large gains on CIFAR long-tailed benchmarks. With SRe2L, integration improves overall accuracy by up to ~10% on both CIFAR-10-LT and CIFAR-100-LT. On CIFAR-10-LT with EDC at IPC=50, IF=100, accuracy rises from 56.0 to 74.8.

• Large-scale gains on ImageNet-1k-LT. With EDC and IPC=50, ADSA improves tail-class accuracy by up to 11.8% (12.4 to 24.2) and raises overall accuracy to 41.4% (from 38.6). With SRe2L at IPC=10, tail accuracy rises from 7.6% to 14.2% and overall from 25.6% to 27.2%.

• Reduced head-class over-prediction. Class-wise accuracy plots show the uncalibrated model over-predicts head classes, causing tail-class misclassification; calibration improves both tail and overall accuracy.

• Works across matching paradigms. On CIFAR-10-LT, ADSA improves MTT from 33.4 to 40.4 at IPC=10 and from 53.0 to 56.6 at IPC=50, and improves DREAM from 56.0 to 59.9 at IPC=10 and 58.6 to 65.7 at IPC=50, outperforming the soft-label-enhanced variants of both.

• Effective under small soft-label budgets. Labels reused after only 1 epoch give EDC 28.0 → 31.0 (+3.0) and SRe2L 17.3 → 17.7 (+0.4); at 10, 100, and 300 epochs, gains for EDC are +2.7, +2.8, and +3.2 respectively, and for SRe2L +2.3, +2.0, and +2.1.

• Both bias types are individually mitigated. With balanced labels but biased images (IF=100), SRe2L at IPC=50 goes from 54.8 to 58.4; with biased labels but balanced images (IF=100), it goes from 31.5 to 57.6.

Methodology in Plain English

The authors start from an existing theoretical result that bounds the test loss of a model trained on distilled data by its training loss plus a distribution-discrepancy term. They re-derive that bound for the long-tailed case, where the training and test sets share per-class feature distributions but differ in how many examples each class has. The rewritten bound highlights a term that depends on how well the distilled data's label distribution matches the balanced test label distribution — exactly what soft labels control.

To find out where that term goes wrong, they split the normal distillation pipeline into two independent halves: one model creates synthetic images, a separate model assigns soft labels. By mixing balanced and imbalanced versions of each half, they isolate how much damage comes from biased images versus a biased labeler. They then measure confidence, entropy, and tail accuracy across increasing levels of imbalance.

The fix, ADSA, is a post-hoc step. The distilled images are treated as a small hold-out set relative to the original training distribution. For each of them, the pretrained model's logits are adjusted using a standard logit-adjustment formula that subtracts a scaled log of class frequency, parameterized by a single strength value. The authors sweep that strength and pick the one that makes the average predicted confidence as even as possible across all classes. The resulting calibrated posteriors replace the original soft labels, and no change is needed to image distillation or model training.

Why This Matters

Impact on research. Most dataset distillation research assumes balanced classes, and the one prior long-tailed work the paper discusses (LTDD) addresses parameter-level bias rather than soft-label bias. This paper reframes the problem as a label-distribution mismatch, supplies a bound that motivates it, and shows that a single post-hoc hyperparameter can recover much of the lost tail-class performance — suggesting that other soft-label-based distillation methods may have the same latent, easily fixed defect.

Real-world applications:

• On-device and edge model training, where storage and compute budgets are tight and class frequencies are naturally skewed.
• Medical or rare-event datasets, where the classes of greatest interest are typically the rarest and hardest to collect.
• Continual learning, which the paper explicitly cites as a real-world application of dataset distillation.
• Large-scale model development pipelines seeking to cut data storage, transmission, and computational costs, and to reduce associated carbon emissions.

Industry relevance. Distillation promises cheaper data storage, faster experimentation, and more accessible training of large models. Demonstrating that this promise holds under realistic imbalanced distributions — and that the fix costs almost nothing computationally — makes the technique more credible for production data-curation workflows.

Future Directions

• Extending the calibration module to other sources of distribution shift beyond class imbalance, such as domain or covariate shift.
• Applying the perturbation framework (separating image synthesis from label generation) to study other forms of distillation bias.
• Combining ADSA with resampling-based distillation model training, which the paper explores in an appendix, to see whether the gains are complementary.
• Determining whether the calibration strength can be predicted rather than searched, since it is currently selected by minimizing class-wise confidence variance on the distilled images.

Target Audience

Researchers and graduate students working on dataset distillation, long-tailed recognition, or data-centric AI, as well as practitioners who need compact training sets from imbalanced real-world data. Readers without a machine-learning background will find the theoretical sections demanding, but the diagnostic framing and the plug-and-play module are accessible and directly reusable.

Authors’ abstract

Dataset distillation compresses large-scale datasets into compact, highly informative synthetic data, significantly reducing storage and training costs. However, existing research primarily focuses on balanced datasets and struggles to perform under real-world long-tailed distributions. In this work, we emphasize the critical role of soft labels in long-tailed dataset distillation and uncover the underlying mechanisms contributing to performance degradation. Specifically, we derive an imbalance-aware generalization bound for model trained on distilled dataset. We then identify two primary sources of soft-label bias, which originate from the distillation model and the distilled images, through systematic perturbation of the data imbalance levels. To address this, we propose ADSA, an Adaptive Soft-label Alignment module that calibrates the entangled biases. This lightweight module integrates seamlessly into existing distillation pipelines and consistently improves performance. On ImageNet-1k-LT with EDC and IPC=50, ADSA improves tail-class accuracy by up to 11.8% and raises overall accuracy to 41.4%. Extensive experiments demonstrate that ADSA provides a robust and generalizable solution under limited label budgets and across a range of distillation techniques. Code is available at: https://github.com/j-cyoung/ADSA_DD.git.