UNSEEN: Enhancing Dataset Pruning from a Generalization Perspective

AI summary

Overview

Research area: Computer Vision / efficient deep learning — specifically dataset pruning (coreset selection) for image classification.

Technical level: Intermediate. The core idea is intuitive, but the paper relies on cross-validation, scoring metrics (Entropy, EL2N, AUM, Forgetting, Margin), and pruning-rate benchmarking conventions that assume familiarity with model training.

Scope: The paper introduces UNSEEN, a plug-and-play scoring framework that selects training samples using models that never saw those samples during training, plus a multi-stage "incremental selection" extension, and evaluates it on CIFAR-10, CIFAR-100, ImageNet-1K, and three fine-grained visual categorization (FGVC) datasets.

What This Paper Is About

Training on huge datasets is expensive, and much of the data is redundant. Dataset pruning tries to keep only a small, informative subset (a "coreset") that trains a model nearly as well as the full dataset. The problem this paper identifies is that almost all existing pruning methods score samples using a model that was trained on those same samples — and because such models fit the training data nearly perfectly, the scores collapse into a narrow range, making samples hard to distinguish and selection unstable. UNSEEN instead scores each sample using a model that never encountered it during training, producing spread-out, more discriminative and more reproducible scores.

Key Contributions

1. A generalization-perspective scoring framework (UNSEEN). The authors expose the limitations of fitting-based sample scoring and propose a plug-and-play framework that scores samples with models trained on folds that exclude those samples, using cross-validation across K mutually exclusive subsets.

2. Incremental Selection (IS). They scale UNSEEN from a single-step to a multi-step procedure: an initial coreset is built, a scoring model is trained on it, and the highest-scoring remaining samples are incrementally refilled into the coreset over J stages, dynamically refining coreset quality.

3. State-of-the-art results across benchmarks. UNSEEN outperforms existing SOTA methods on CIFAR-10, CIFAR-100, and ImageNet-1K, and on ImageNet-1K it reduces training data by 30% with the abstract describing this as lossless performance (the table reports a 0.06% accuracy drop). The authors also report cross-architecture transfer and fine-grained dataset results.

4. Class-level difficulty analysis. They extend the notion of sample difficulty to the class level, showing that UNSEEN implicitly prioritizes hard classes and that minimizing inter-class accuracy disparity — rather than treating all classes uniformly — underlies its gains.

Main Findings

• Fitting-based scores are densely clustered. Under the conventional fitting framework, more than 90% of CIFAR-10 samples and 78% of CIFAR-100 samples have Entropy values close to zero, reducing sample discrimination.
• Fitting-based selection is unstable. Across two identical CIFAR-100 trials with different random seeds, the Pearson correlation coefficient between sample rankings was 0.43 under fitting versus 0.92 under UNSEEN.
• ImageNet-1K 30% pruning. UNSEEN reached 73.55% accuracy at a 30% pruning rate against a full-dataset accuracy of 73.61%, a 0.06% gap; the abstract characterizes this as lossless performance while reducing training data by 30%.
• CIFAR-10 gains over full data. Full-dataset accuracy on CIFAR-10 is 95.50%; UNSEEN reached 95.59% at a 30% pruning rate, i.e., it outperformed full-data training. At 70% pruning it still reached 94.16%.
• CIFAR-100 at 30% pruning. Full-dataset accuracy is 79.24%; UNSEEN reached 78.61%, a 0.63% drop. At 70% pruning it reached 68.49%, versus 62.47% for plain Entropy and 48.48% for Forgetting.
• Plug-and-play gains. Adding UNSEEN to classical methods raised them near or above the previous SOTA: on CIFAR-100 at 30% pruning, Margin with UNSEEN reached 78.19% and Least Confidence with UNSEEN reached 78.17%.
• Both components help, together they are best. Ablation on CIFAR-10 (30%/50%/70%) gives Entropy 94.77/93.87/90.83, UNSEEN only 95.39/95.33/94.05, IS only 95.27/94.82/92.96, and UNSEEN+IS 95.59/95.35/94.16. On CIFAR-100 the same four settings give 76.73/72.21/62.47, 78.24/74.56/62.23, 76.92/74.31/65.73, and 78.61/75.15/68.49.
• Moderate scoring models are best. On CIFAR-100 at 70% pruning, UNSEEN with K=2 scored 64.53, K=4 scored 67.90, K=10 scored 67.52, and K=20 scored 66.41 — overly strong (small K) or overly weak (large K) scoring models underperform.
• It is generalization, not just smaller proxies, that matters. Entropy-proxy (K=4, scoring within each subset) reached 76.83/73.22/65.08 on CIFAR-100 at 30%/50%/70%, well below UNSEEN's 78.61/75.15/67.90.
• Incremental selection is cheap. UNSEEN incurs the same total iteration count as previous methods, N(2−p)/B. Incremental selection at a 70% pruning rate incurs only 0.15× the total cost of previous methods.
• Hard classes get prioritized. Rank-difference analysis showed most samples from the hardest class dropped in rank (gaining selection priority) under UNSEEN, while the easiest class showed the opposite. Inter-class variance of accuracy grew rapidly under the fitting framework as pruning increased but stayed lower and more stable under UNSEEN.

Methodology in Plain English

The insight is simple: if you judge whether a sample matters using a model that has already memorized that sample, the model will call almost everything "easy" and the scores will bunch together. So instead, the authors split the full training set into K mutually exclusive, equal-sized folds. For each fold, a model is trained on just that fold, and that model is then used to assign scores (the basic Entropy loss is used as the scoring function) to the samples in the other folds — samples it never trained on. Every sample is therefore scored only by models that are "unseen" to it. Scores are normalized and the top M₁ highest-scoring samples form an initial coreset.

They then scale this to multiple steps. Given a target pruning rate p, the target coreset size is M = ⌊N(1−p)⌋, split into per-stage sizes Mⱼ summing to M. At each subsequent stage j, a model is trained on the current coreset Sⱼ₋₁ and used to score the remaining unselected samples; the top Mⱼ are added, and the process repeats until the coreset reaches size M. Defaults are K=4 folds and J=2 stages, with M₂ = ⌈N·10%⌉ selected in the second stage, consistent across all pruning rates, and all results averaged over five runs.

The rescoring models are the key lever: because they never see the samples they score, they spread the scores over a wider range, so the resulting ranking is both more discriminative and far more reproducible across random seeds. Because a weak or overly strong scoring model is less useful, K controls how "moderate" the scoring models are.

Why This Matters

Impact on research. The paper reframes dataset pruning as a generalization problem rather than a fitting problem, and shows the framing itself — not just a new score function — drives large gains. Because UNSEEN is plug-and-play, it can be layered onto existing criteria (the authors demonstrate this with Margin and Least Confidence), meaning prior work in the area can be re-examined through this lens. The class-level argument also connects pruning to class imbalance, a largely separate literature.

Real-world applications (as motivated by the paper's setting):

• Large-scale image classification pipelines, where the reported 30% reduction on ImageNet-1K with a 0.06% accuracy drop translates directly into lower training cost.
• Fine-grained visual categorization, such as the CUB-2011, Stanford Dogs, and Stanford Cars datasets the authors test on, where the differences between classes are subtle.
• Model pre-training and fine-tuning workflows, where a reusable compact coreset can be shared across downstream models rather than repartitioned per experiment.
• Cross-architecture deployment, since a coreset pruned with ResNet-18 is reported to transfer well to ResNet-34 and ResNet-50, letting one pruned dataset serve multiple model sizes.

Industry relevance. Compute and energy budgets for training are a first-order cost for any organization training on web-scale data, so a method that reports no additional computational overhead over existing pruning pipelines and only minimal overhead for the incremental variant is practically attractive. The involvement of EPIC Lab, SJTU and Alibaba Group points toward large-scale production image pipelines.

Future Directions

• Beyond classification. All reported experiments are image classification on CIFAR-10/100, ImageNet-1K, CUB-2011, Stanford Dogs, and Stanford Cars; whether generalization-based scoring helps detection, segmentation, or vision-language pretraining is not reported.
• Principled choice of K and J. The authors show performance is sensitive to K (K=4 best among 2, 4, 10, 20) and that increasing J costs more while leaving performance nearly unchanged; a principled way to set these without a search remains open.
• Beyond the Entropy score. The paper adopts basic Entropy as the scoring function and only demonstrates plug-and-play enhancement with Margin and Least Confidence; combining unseen-based scoring with other families such as geometry-based or decision-boundary-based criteria is untested.
• Theoretical grounding. The paper offers an empirical and class-level explanation (reduced inter-class variance) for why generalization-based scoring works, but no formal characterization of the relationship between unseen-model scores and coreset generalization error.

Target Audience

Researchers and engineers working on dataset efficiency, coreset selection, and data-centric machine learning; practitioners who train vision models at scale and want training-cost reductions without retraining pipelines from scratch; and graduate students looking for an accessible example of how a change in how scores are produced — rather than a new score itself — can yield substantial benchmark gains.

Authors’ abstract

The growing scale of datasets in deep learning has introduced significant computational challenges. Dataset pruning addresses this challenge by constructing a compact but informative coreset from the full dataset with comparable performance. Previous approaches typically establish scoring metrics based on specific criteria to identify representative samples. However, these methods predominantly rely on sample scores obtained from the model's performance during the training (i.e., fitting) phase. As scoring models achieve near-optimal performance on training data, such fitting-centric approaches induce a dense distribution of sample scores within a narrow numerical range. This concentration reduces the distinction between samples and hinders effective selection. To address this challenge, we conduct dataset pruning from the perspective of generalization, i.e., scoring samples based on models not exposed to them during training. We propose a plug-and-play framework, UNSEEN, which can be integrated into existing dataset pruning methods. Additionally, conventional score-based methods are single-step and rely on models trained solely on the complete dataset, providing limited perspective on the importance of samples. To address this limitation, we scale UNSEEN to multi-step scenarios and propose an incremental selection technique through scoring models trained on varying coresets, and optimize the quality of the coreset dynamically. Extensive experiments demonstrate that our method significantly outperforms existing state-of-the-art (SOTA) methods on CIFAR-10, CIFAR-100, and ImageNet-1K. Notably, on ImageNet-1K, UNSEEN achieves lossless performance while reducing training data by 30\%.