IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves

AI summary

Overview

Research area: Computer vision — knowledge distillation, multi-teacher distillation, and data-free/synthetic training data for Vision Transformers (ViTs).

Technical level: Intermediate. The paper assumes familiarity with knowledge distillation, Vision Transformer architecture, and standard transfer/segmentation/depth evaluation protocols; the proposed loss formulation is mathematically compact but not elementary.

Scope: The paper proposes IDeaL ("Improved Dead Leaves"), a pixel-space optimization procedure that turns procedurally generated Dead Leaves noise into teacher-tailored synthetic samples for fully data-free multi-teacher distillation, and evaluates the resulting students against ImageNet-distilled oracles across classification, transfer, segmentation, and depth tasks.

What This Paper Is About

Multi-teacher distillation trains one student model to mimic several complementary teacher models at once, but existing methods assume the teachers' images (or a large web dataset such as DataComp-1B) are available for training the student. This is often impossible when teachers were trained on proprietary, private, or legally restricted data. The paper asks how far multi-teacher distillation can go with no real images at all, and answers by optimizing procedurally generated structured noise so that it becomes maximally informative for the specific set of teachers being distilled.

Key Contributions

1. A systematic study of data-free multi-teacher distillation. The authors first quantify how student performance degrades as real ImageNet distillation subsets shrink (1M, 100K, 10K, 1K), establishing an oracle, then replace real images with procedurally generated surrogates — Gaussian noise, FractalDB, and Dead Leaves — under a controlled protocol where all four teachers are trained only on ImageNet.

2. A teacher-tailored pixel-optimization method. Starting from Dead Leaves, the pixels of each synthetic image are treated as learnable parameters and updated by backpropagating through frozen teacher ViT encoders, using two decorrelation losses plus a total-variation regularizer. The output is a new dataset called Improved Dead Leaves (IDeaL).

3. Patch-level and image-level decorrelation objectives. The patch decorrelation loss drives the pairwise patch cosine-similarity matrix of each teacher layer toward the identity, and the image decorrelation loss does the same at the batch level on global (CLS) representations, jointly summed over all teachers.

4. Demonstration that the synthesized data transfers. IDeaL samples are optimized using only the teachers, so they are independent of student architecture; the paper shows they also work when distilling the same four ViT-B/16 teachers into a smaller ViT-S/16 student.

Main Findings

• Reducing real ImageNet distillation data is nearly free until a point. Going from 1M to 100K Images gives 83.2 → 81.7 Top-1 on ImageNet and 73.1 → 71.6 transfer Top-1; dropping to 10K (77.0 / 66.5) or 1K (72.6 / 65.3) causes a far more drastic drop. The reproduced UNIC model on the full 1.28M ImageNet set reaches 83.3 / 73.3 / 39.5 mIoU / 0.523 RMSE. The paper states that nearly 90% of ImageNet can be discarded with negligible impact.

• Gaussian noise is essentially useless; FractalDB is weak; Dead Leaves is the best procedural baseline. Gaussian noise students reach 21.1–22.7 ImageNet Top-1 across all sizes; FractalDB reaches 27.1–32.7; Dead Leaves reaches 64.7–67.1 ImageNet Top-1 and 63.9–65.6 transfer Top-1. The paper notes 100K Dead Leaves achieves 65.6 transfer, on par with the 65.3 obtained with a 1K ImageNet subset.

• IDeaL substantially closes the gap to real-data distillation. At 1M samples, IDeaL students reach 78.2 ImageNet Top-1, 70.6 transfer Top-1, 34.5 mIoU on ADE20K, and 0.573 RMSE on NYUd — relative gains of 17%, 8%, 18%, and 9% over Dead Leaves. At 1K samples: 74.1 / 68.6 / 33.9 / 0.594, with relative gains of 14%, 7%, 17%, and 7%. The largest relative gains are on semantic segmentation (18% at 1M, 19% at 100K, 18% at 10K, 17% at 1K).

• With 1K images, IDeaL matches or beats ImageNet subsets in classification. The 1K IDeaL student beats the 1K ImageNet student by +1.5 ImageNet Top-1 (74.1 vs. 72.6); for transfer learning, IDeaL beats ImageNet subsets by +3.3 and +3.4 Top-1 at 1K and 10K respectively, and the 10K IDeaL student matches the 10K ImageNet student on ImageNet classification. This advantage does not extend to 100K or 1M, where procedural data — including IDeaL — scales worse than ImageNet, particularly on dense tasks.

• IDeaL students exceed the minimum teacher on three of four task families. With as few as 1K samples, students surpass the minimum teacher (DINO at 78.4 ImageNet / 72.4 transfer / 30.4 mIoU / 0.570 RMSE) on transfer learning, segmentation, and depth. At 1M, transfer Top-1 is +2.3 over the minimum teacher, and segmentation is +4.1 mIoU, outperforming all teachers except the top performer (iBOT at 36.6 mIoU). Only on ImageNet classification do IDeaL students stay slightly below the minimum teacher (78.2 vs. 78.4 at 1M), which the authors attribute to all teachers having been trained on ImageNet.

• Transfer gains vary by sub-task. Breaking the 15 transfer benchmarks into concept generalization, fine-grained, and long-tail classification, IDeaL improves over Dead Leaves by an overall 7%; for long-tail tasks at 100K and 1M sizes the gain is 12%.

• Joint optimization over all teachers matters. Samples optimized with one teacher yield 74.3–74.9 ImageNet Top-1; with pairs of teachers, 75.5–76.0; with all four teachers jointly, 77.0 (transfer 69.9, 34.5 mIoU, 0.592 RMSE). Combining four independently optimized sets of 2.5K samples each gives only 75.1 / 69.4 / 33.9 / 0.583, showing that joint optimization is what projects complementary teacher information into the data. Samples generated with iBOT, the strongest teacher on dense tasks, also produce students strong on dense tasks.

• Both decorrelation losses contribute. In the 10K ablation, unoptimized Dead Leaves gives 66.6 / 65.1 / 29.1 / 0.655. Adding only the image decorrelation loss gives 73.4 / 68.2 / 32.4 / 0.663; adding only the patch decorrelation loss gives 75.4 / 69.1 / 33.4 / 0.595; combining both gives 77.0 / 69.9 / 34.5 / 0.592.

• Results hold for a different student architecture. Distilling into ViT-S/16 at 1M samples: IDeaL reaches 72.6 / 66.3 / 30.6 / 0.634 versus Dead Leaves at 59.2 / 60.1 / 24.2 / 0.673 (relative gains of 23%, 10%, 26%, 6%) and ImageNet oracle at 80.8 / 69.6 / 36.6 / 0.572. At 1K samples, IDeaL reaches 66.8 / 63.4 / 29.6 / 0.632 versus Dead Leaves 55.8 / 57.9 / 23.3 / 0.690 (gains of 20%, 9%, 27%, 8%), surpassing the 1K ImageNet oracle (53.7 / 49.6 / 27.6 / 0.640) on all tasks.

Methodology in Plain English

The framework is UNIC's multi-teacher distillation setup, unmodified except for the distillation data. Four ViT-B/16 teachers — two self-supervised (DINO, iBOT) and two supervised (DeiT-3, dBOT-ft) — are each trained only on ImageNet, so any difference observed comes from the data pipeline rather than from teacher training sets. All teachers use 768-dimensional features, 12 blocks, 224×224 inputs, and patch size 16. The student is a ViT-B/16 distilled for 100 epochs; a ViT-S/16 student is used only in the architecture-transfer experiment.

Data generation is a two-stage, decoupled process:

1. Generate the synthetic dataset. Each image starts as a Dead Leaves sample resized to 224×224, and its pixel values are treated as learnable parameters. Gradients flow through the frozen teacher encoders, so the image is optimized to be informative for all teachers at once. The generation loss combines three terms: a patch decorrelation loss pushing the per-layer, head-averaged pairwise patch cosine-similarity matrix toward the identity (so patches inside one image differ from each other); an image decorrelation loss doing the same on global CLS (or average-pooled patch) embeddings within a batch (so different images differ from each other); and a total-variation regularizer for pixel smoothness. Weights are 1, 1, and 0.05. Optimization uses pools of 250 samples, random subsets of 40 samples, 10 iterations per subset before resampling, 4000 iterations per pool, random horizontal flips, and Adam with learning rate 0.1. Feature normalization is applied at the end of the forward pass.

2. Distill the student. The optimized images are stored and then used exactly as if they were real data: the student minimizes the multi-teacher distillation loss, which sums, over teachers, similarity terms between the student's CLS and patch features and each teacher's CLS and patch features.

The advantages claimed for this design are that no labels, classification heads, or teacher training statistics are needed (unlike BatchNorm-statistics methods, and unlike generator-based approaches that co-train an adversarial generator with the student), and that generation cost is paid once because the dataset can be reused across student architectures.

Evaluation follows UNIC: ImageNet validation Top-1; transfer Top-1 averaged over 15 classification benchmarks (the 5 ImageNet-CoG concept-generalization levels, 8 fine-grained datasets — Aircraft, Cars196, DTD, EuroSAT, Flowers, Pets, Food101, SUN397 — and 2 long-tailed datasets, iNaturalist 2018 and 2019); mIoU on ADE20K; RMSE on NYUd. All use linear probing on frozen encoder outputs. Each subset size is reported as the mean over three subsets from different seeds, with ImageNet subsets enforcing at least one sample per class. Ablations use 10K samples per configuration.

Why This Matters

Research impact. The paper removes the assumption that distillation data must be real, and it is, to the authors' knowledge, the first work to fully replace real images with synthetic surrogates in multi-teacher distillation while staying competitive across a large number of downstream tasks. It also reframes what synthetic distillation data is for: the images are not meant to look realistic or to recover the teachers' training set, only to capture how the teachers process data. This makes it applicable to self-supervised teachers and to any teacher combination, including ones without classification heads.

Real-world applications (implied by the paper's motivation).

• Distilling foundation models whose pretraining corpora are proprietary, licensed, or otherwise non-redistributable.
• Building compact student encoders in settings where privacy rules prevent using or sharing the original training images.
• Producing reusable synthetic distillation datasets, generated once and then applied to multiple student architectures or distillation configurations.
• Low-budget distillation workflows, where only 1K synthetic samples are needed to match or surpass a 1K real-image subset on classification tasks.

Industry relevance. The method directly addresses a practical bottleneck for companies deploying distilled backbones: teacher data is often unavailable for legal or commercial reasons, yet practitioners still want a single versatile encoder that combines the strengths of several models. The paper notes that performance does not scale as well as ImageNet at 100K and 1M samples, particularly for dense prediction, so the current sweet spot is data-scarce regimes rather than large-budget ones.

Future Directions

• Fixing the scaling behavior. IDeaL and procedural data generally do not scale as well as ImageNet at 100K and 1M samples, especially for segmentation and depth; the paper reports a scaling analysis only in supplementary material, and closing this gap is the clearest open problem.
• Closing the ImageNet classification gap. IDeaL students remain slightly below even the minimum teacher on ImageNet (78.2 vs. 78.4 at 1M), which the authors attribute to all teachers being trained on that dataset; whether a different initialization or objective can address this is unresolved.
• Improving transfer gains on fine-grained and concept-generalization tasks. Transfer gains are consistently smaller than segmentation gains, and long-tail tasks only gain 12% at 100K and 1M; the paper defers per-sub-task results to supplementary material.
• Extending beyond Dead Leaves initialization and beyond ViT-B/16 teachers. The method is initialized from Dead Leaves and demonstrated with four ImageNet-trained ViT-B/16 teachers distilled into ViT-B/16 and ViT-S/16 students; behavior with other initializations, teacher families, teacher counts, or non-ViT architectures is not reported.

Target Audience

Researchers and engineers working on knowledge distillation, model compression, and vision foundation models, particularly those who need to transfer knowledge from teachers whose training data cannot be accessed or redistributed. It is also relevant to practitioners interested in data-free or synthetic-data training, procedural image generation for pretraining, and the evaluation of distilled encoders across classification, transfer, segmentation, and depth. Readers should already be comfortable with Vision Transformer internals and standard distillation terminology.

Authors’ abstract

Multi-teacher distillation has emerged as a way to combine complementary teacher models into a single student model that exhibits the strengths of all its teachers. The student is trained to mimic the output of the teachers on a set of images, typically the union of the individual teacher's training sets, assuming this data is available. In this paper, we question that assumption and explore alternative options. We first study how far one can go when distilling from teachers fed with different types of noise. Then, we show that information contained in the teachers can be leveraged to tailor the noise for multi-teacher distillation: we propose a method that, thanks to decorrelation losses at both patch and image levels, generates teacher-specific, improved samples optimized for data-free distillation. Experiments show that our most effective samples, IDeaL, lead to strong students that successfully capture complementary information from the teachers, yielding surprisingly competitive results that substantially narrow the gap with students distilled from real images. Moreover, given a limited budget of 1K images for distillation, students distilled using our IDeaL samples match or surpass the performance of those distilled using a 1K-image subset of ImageNet.