Grounding and Enhancing Informativeness and Utility in Dataset Distillation

AI summary

Overview

Research area: Dataset distillation (DD) — compressing large training datasets into tiny synthetic replacements — with a focus on knowledge-distillation-based methods for image classification.

Technical level: Intermediate. The paper is dense with formal definitions (Shapley Value, gradient flow, utility bounds), but its central idea — pick the most informative patches and the most influential samples — is conceptually approachable.

Scope: A theoretically grounded framework and method (InfoUtil) that defines what an optimal distilled dataset looks like and delivers state-of-the-art accuracy on ImageNet-scale benchmarks at a fraction of the compute cost of prior work.

What This Paper Is About

Dataset distillation tries to shrink a huge dataset into a handful of synthetic images that train a model almost as well as the original. Existing knowledge-distillation-based methods work surprisingly well but rely on heuristics — random cropping and loss-based scoring — with no principled reason to expect the resulting images to be good. This paper asks what an optimal distilled dataset should contain, defines that mathematically in terms of "informativeness" and "utility," and builds a practical method that optimizes both.

Key Contributions

1. A formal definition of Optimal Dataset Distillation (Definition 4). The authors introduce two measurable properties — Informativeness (how much of a sample's key content survives compression) and Utility (how much a sample matters for training) — and frame distillation as jointly maximizing them. This gives the field a theory-grounded target instead of a purely empirical one.

2. InfoUtil, a two-stage method matching that definition. Stage one uses the game-theoretic Shapley Value to identify and keep the most informative image patches; stage two uses Gradient Norm scoring to retain the samples with the highest training utility.

3. A utility upper bound via gradient norm (Theorem 1). Directly measuring a sample's utility would require retraining the model with and without it, which is computationally infeasible. The paper proves utility is bounded by the sample's gradient norm, making utility selection cheap and principled.

4. Strong empirical gains plus large efficiency wins. InfoUtil beats the prior state of the art (RDED) by roughly 6.1% on ImageNet-1K and 16% on ImageNet-100, while using ~50× less time and ~100× less memory than trajectory-matching methods like TESLA, and finishing ImageNet-21K distillation in 5.83 hours on a single A100.

Main Findings

• Shapley Value attribution beats heuristic cropping. Replacing random patches with Shapley-selected patches produces images that capture semantically meaningful regions (e.g., the object) rather than background clutter. On ImageNet-1K at IPC=10, Shapley selection reached 43.88% versus 30.39% for Grad-CAM — a 13.49-point gap that the authors attribute to Shapley's axiomatic guarantees (efficiency, symmetry, dummy, linearity) and Grad-CAM's tendency toward gradient saturation.

• Gradient-norm scoring is a strong, cheap utility proxy. Swapping loss-based scoring for gradient-norm scoring alone lifted ImageNette (IPC=50) from 80.4% to 85.0% (+4.6%) and ImageNet-1K (IPC=10) from 42.0% to 43.5% (+1.5%), confirming the theoretical bound translates into practical gains.

• The two components are complementary. Combining attribution-guided cropping with gradient-norm scoring gave further gains — 86.2% on ImageNette and 44.2% on ImageNet-1K — showing informativeness and utility capture distinct, additive value.

• Noise injection is essential, not cosmetic. Without added noise in patch selection, performance collapses at higher IPC (86.2% → 70.6% on ImageNette at IPC=50). Deterministic, greedy patch selection produces redundant samples that hurt generalization.

• Gains hold across architectures and scales. InfoUtil outperforms SRe2L and RDED when teachers and students differ in architecture (e.g., a 10-point jump when a VGG-11 teacher transfers to a Swin-V2-Tiny student), and remains ahead on large-IPC settings (IPC=200 on ImageNet-1K: 63.4% vs. RDED's 62.5% on ResNet-18).

• Distilled data transfers to downstream tasks. In a 5-step continual learning setup on ImageNette, InfoUtil's distilled samples retained far more accuracy at later stages (e.g., 0.4739 vs. 0.2191 at stage 5), indicating the synthetic data preserves knowledge useful beyond a single classification task.

• Soft-label choice interacts with IPC. Analysis shows early, high-entropy teacher checkpoints help at IPC=1, while fully converged (low-entropy) teachers work better at IPC=10 — practical guidance for anyone generating labels.

Methodology in Plain English

The pipeline has two stages, both applied to a teacher model pretrained on the real data.

Step 1 — Keep the informative patches. Take each real image and treat it as a grid of patches. Rather than guessing which patches matter, the authors apply the Shapley Value, a tool from cooperative game theory that fairly divides credit among participants. Here, each patch is a "player" and the model's output is the "payoff." The Shapley Value assigns each patch a score reflecting its marginal contribution to the model's decision. Pooling these scores gives a heatmap; the authors keep the highest-scoring regions, compressing the image (e.g., to one-quarter of its size). A small amount of random noise is added to the heatmap so different images within a class don't all select the identical patch — this diversity turns out to be crucial.

Step 2 — Keep the influential samples. From the pool of compressed samples, select those that matter most for training. The paper defines utility as how much removing a sample would perturb the model's training dynamics (gradient flow), then proves this quantity is bounded by the sample's gradient norm. Since gradient norms are easy to compute, the authors simply rank all compressed samples by gradient norm and keep the top ones.

Reconstruction and labels. For low-resolution data, one compressed image per class is used directly. For high-resolution data, four quarter-sized crops from the same class are tiled into one full-size image. Soft labels come from intermediate checkpoints of the pretrained teacher, chosen based on IPC.

The whole process runs on a single NVIDIA A100 GPU.

Why This Matters

Impact on research. Dataset distillation has been dominated by methods tuned through trial and error. This paper reframes the problem with explicit definitions and a provable relationship (utility ≤ gradient norm), giving future work a vocabulary — informativeness, utility, optimal distillation — and a theoretical baseline to build on or challenge. It also offers a clean argument for why Shapley-style attribution belongs in data curation.

Real-world applications:

• Edge and mobile deployment. Models distilled on small synthetic datasets can be trained or fine-tuned on-device without shipping terabytes of raw data.
• Privacy-preserving ML. Synthetic distilled datasets can stand in for sensitive records (medical images, financial data) where the original data cannot leave a secure enclave.
• Regulated industries needing transparency. Because sample selection is driven by named, principled criteria rather than opaque heuristics, the method offers an auditable rationale for what data a model was trained on — valuable for compliance.
• Rapid prototyping and continual learning. Distilling large corpora in hours on one GPU lets teams iterate on model architectures quickly, and the demonstrated retention across continual-learning stages supports incremental deployment scenarios.

Industry relevance. The efficiency numbers are the headline for practitioners: replacing multi-GPU, multi-day pipelines with a single-GPU, single-digit-hour workflow removes a major barrier to using dataset distillation at production scale, particularly for the ImageNet-class datasets that matter commercially.

Future Directions

• Scaling attribution cost further. Shapley Value computation is approximated via KernelShap, but per-image attribution is still far more expensive than random cropping. Faster estimators or amortized attribution models could close that gap.
• Extending the framework beyond classification. The definitions of informativeness and utility are written for classification with cross-entropy loss. Adapting the utility bound to detection, segmentation, or generative objectives is an open problem.
• Tighter utility approximations. Gradient norm is an upper bound, not the utility itself. The gap between the bound and true utility — and whether a tighter, still-cheap proxy exists — is unexplored.
• Better handling of dataset imbalance. The authors note they had to switch to full images at IPC=200 to mitigate imbalance. Whether the informativeness/utility framework can actively correct class imbalance rather than work around it remains unresolved.

Target Audience

Researchers and graduate students working on dataset distillation, dataset pruning, or coreset selection will find the theoretical framing directly useful. Practitioners who need to train or fine-tune vision models under compute, storage, or privacy constraints — ML engineers on edge platforms, teams in regulated sectors, and anyone managing very large training corpora — will find the efficiency results and the practical recipe (Shapley patches + gradient-norm selection + noise) most actionable. Readers should be comfortable with gradient-based training and basic attribution concepts; the formal proofs in the appendices are aimed at a more theoretically inclined audience.

Authors’ abstract

Dataset Distillation (DD) seeks to create a compact dataset from a large, real-world dataset. While recent methods often rely on heuristic approaches to balance efficiency and quality, the fundamental relationship between original and synthetic data remains underexplored. This paper revisits knowledge distillation-based dataset distillation within a solid theoretical framework. We introduce the concepts of Informativeness and Utility, capturing crucial information within a sample and essential samples in the training set, respectively. Building on these principles, we define optimal dataset distillation mathematically. We then present InfoUtil, a framework that balances informativeness and utility in synthesizing the distilled dataset. InfoUtil incorporates two key components: (1) game-theoretic informativeness maximization using Shapley Value attribution to extract key information from samples, and (2) principled utility maximization by selecting globally influential samples based on Gradient Norm. These components ensure that the distilled dataset is both informative and utility-optimized. Experiments demonstrate that our method achieves a 6.1\% performance improvement over the previous state-of-the-art approach on ImageNet-1K dataset using ResNet-18.