PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors

AI summary

Overview

• Research area: Dataset distillation (DD) — generating compact synthetic training sets — with a focus on the diversity and generalization limits of large-scale distillation.
• Technical level: Advanced. The paper assumes familiarity with bi-level distillation objectives, batch-normalization statistic matching, and ImageNet-scale training and evaluation protocols.
• Scope: The paper introduces PRISM, a framework that decouples the logit-matching objective from the batch-normalization (BN) alignment regularizer so that different teacher architectures supervise each term, and evaluates it on ImageNet-1K at 10, 50, and 100 images-per-class (IPC).

Authors are affiliated with the German Research Center for Artificial Intelligence (DFKI) and RPTU Kaiserslautern-Landau. The paper is licensed CC BY 4.0, arXiv:2511.09905v2 [cs.LG], 30 Nov 2025.

What This Paper Is About

Dataset distillation aims to produce a small synthetic dataset that trains models nearly as well as the full real dataset. Existing methods typically rely on a single pre-trained teacher model for all supervision, so the synthetic images inherit that one model's inductive bias and drift toward overly smooth, homogeneous samples with little intra-class variety, especially as dataset size grows. PRISM attacks this by splitting the supervision signal across different teacher architectures instead of a single one.

Key Contributions

1. A decoupled-prior framework. PRISM separates the logit-matching term from the BN-alignment regularization term and supervises them with different teacher models, so multiple architectural "views" contribute to different parts of the synthesis rather than one model filling both roles.
2. A systematic study of teacher-selection strategies. The paper compares pre-distillation selection (teachers fixed per synthetic image before optimization) with intra-distillation selection (teachers re-sampled during optimization), and reports that pre-distillation selection of diverse teachers is highly effective.
3. New state-of-the-art results plus a diversity measurement. The paper reports up to 70.4% top-1 accuracy with ResNet-101 at IPC=100 and quantifiably greater intra-class diversity, measured as lower cosine similarity between features.
4. A scalable synthesis scheme. PRISM uses cross-class batch formation, processing each IPC index independently, which the authors describe as simple and massively parallelizable across GPUs, in contrast to the intra-class batching of G-VBSM, EDC, and DELT.

Main Findings

• Gains concentrate at mid and high IPC. On ImageNet-1K, PRISM reaches 49.4% (ResNet-18, IPC=10), 59.0% (ResNet-18, IPC=50), and 60.9% (ResNet-18, IPC=100) under the authors' evaluation protocol, versus EDC at 48.6%, 58.0%, and not reported at IPC=100. On ResNet-50 PRISM reaches 65.1% at IPC=50 and 67.5% at IPC=100; on ResNet-101 it reaches 65.9% at IPC=50 and 68.6% at IPC=100. All Table 1 numbers are mean ± std over three seeds.
• It also wins under a competing evaluation protocol. Using DELT's evaluation procedure, PRISM reaches 59.6% and 62.7% on ResNet-18 at IPC=50 and IPC=100, 69.4% on ResNet-50 at IPC=100, and 70.4% on ResNet-101 at IPC=100.
• Single-teacher methods are the weakest baseline. SRe2L scores 21.3%, 46.8%, and 52.8% on ResNet-18 at IPC=10, 50, and 100; G-VBSM scores 31.4%, 51.8%, and 55.7% on the same backbone.
• Diversity is measurably higher. Intra-class semantic cosine similarity, computed with a pretrained ResNet-18 as feature extractor, is between 0.86 and 0.92 for SRe2L, G-VBSM, and DELT, while PRISM has mean values of 0.83 and below.
• Pre-distillation teacher selection beats intra-distillation selection. In the recovery-only study at IPC ≈ 1200 with ResNet-18 (no knowledge distillation), single-teacher alignment rises from 18.3% (intra-distillation) to 32.4% (pre-distillation), and dual-teacher decoupling rises from 19.0% to 36.2%. Adding more BN teachers under pre-distillation selection gives 37.4% (2 teachers), 38.7% (3), and 39.1% (4), against a real-data baseline of 70.0%.
• More BN teachers costs VRAM. Maximum VRAM during distillation at a batch size of 100 rises from 6.5 GB (single teacher) to 13.0 GB (dual-teacher decoupling) and further to 18.5 GB, 26.0 GB, and 32.5 GB for 2, 3, and 4 BN teachers.
• Pipeline choices matter in a specific order. Ablations on ImageNet-1K, ResNet-18, IPC=10 show: replacing a single relabeler with an ensemble average improves 21.22% to 23.75%, and adding MSE with a 0.1 ground-truth term gives 29.53%; varying distillation iterations per image gives 31.30%; a relabeling batch size of 50 is best at 44.04% (16 gives 40.41%, 32 gives 43.40%, 64 gives 43.79%, 128 gives 42.59%); aligning the primary logit teacher with the primary BN teacher to the relabeling model gives 45.73%; a recovery learning rate of 0.05 gives 47.35% (0.10 gives 46.63%, 0.25 gives 45.73%); and swapping EfficientNet for AlexNet in both teacher pools gives 47.77%.
• The SSRS learning-rate schedule closes the gap. Adopting EDC's SSRS decayed cosine schedule with a slowdown coefficient of ζ = 2.5 is described as providing the final significant leap leading to the state-of-the-art results; the resulting accuracy is not reported in the visible ablation tables.

Methodology in Plain English

The starting point is SRe2L, a two-part recipe. A pre-trained teacher model produces soft labels, and the synthetic images are optimized to match that teacher's outputs (the logit-matching term). A second term keeps the synthetic images from collapsing into noisy, adversarial-like patterns by matching the mean and variance recorded in the teacher's batch-normalization layers (the BN-alignment regularizer). Both terms ordinarily come from the same model.

PRISM's move is to let those two terms come from different models. The logit term is supervised by one teacher, and the BN-alignment term by a different architecture — for example, an EfficientNet for logits and a ResNet for BN statistics. This produces a gradient that is the sum of two architectural perspectives rather than one.

The generalized version scales this up: instead of one BN teacher, the method keeps a pool of source models, samples a random subset of them for each synthetic image (uniformly, up to a VRAM-constrained maximum), and minimizes the expected loss over that random selection. The authors call this "PRIors from diverse Source Models."

The paper then separates two ways of choosing teachers. In pre-distillation selection, a synthetic image is assigned its logit teacher and BN teacher subset once, before optimization, and keeps them. In intra-distillation selection, teachers are re-sampled at each optimization step. Batch formation is the other design axis: PRISM follows SRe2L and processes each IPC index independently, forming cross-class batches containing the i-th image from every class, which makes each IPC slice separately parallelizable. Other methods form intra-class batches to enable regularizers such as data densification or diversity-driven pushing, which the authors trade away for simplicity and parallelism.

Experiments use ImageNet-1K only — the authors state they deliberately skip CIFAR-10/100 because PRISM targets architectural bias at large, high-class-count scale. The synthetic dataset is initialized by selecting each real ImageNet training image exactly once, with no coreset selection. Distillation runs for a maximum of 4000 iterations, and results are reported under both the authors' protocol and DELT's protocol. Teacher pools for the multi-teacher study include ResNet18, ResNet34, ShuffleNetV2-0.5, MobileNetV2, and EfficientNet-B0, with ResNet18 used for logit maximization to stay comparable with G-VBSM. Validation uses AdamW, a base learning rate of 0.001, weight decay 0.01, 300 epochs, and batch sizes of 50 (IPC 10) and 100 (IPC 50, 100); recovery uses Adam, a base learning rate of 0.05, momentum β₁, β₂ = 0.5, 0.9, a BN weight α_BN of 0.01, and batch size 100.

Why This Matters

For research, PRISM frames architectural decoupling as an axis orthogonal to existing scaling attempts — training schedules (DELT), data initialization (RDED), and post-evaluation pipelines (EDC) — rather than a replacement for them. It also supplies a quantitative diversity metric (intra-class cosine similarity) alongside accuracy, which is the axis the paper argues has been underexamined in SRe2L-derived work.

Real-world applications cited or implied by the paper's motivation for dataset distillation:

• Privacy-preserving data sharing. Distillation is motivated by safeguards against membership inference and model inversion, relevant where raw data cannot be distributed. The appendix sketches an analogous "architectural noise" argument inspired by Private Aggregation of Teacher Ensembles, though the paper states PRISM adds no calibrated noise or clipping and therefore carries no formal differential privacy guarantee.
• Robustness against adversarial attacks. Adversarial robustness is one of the three stated motivations for DD, and the homogeneity the paper targets is linked in prior work to reduced robustness.
• Efficient data compression and storage. Smaller training sets reduce storage and transfer costs for data-heavy pipelines.
• High-stakes domains with bias risk. The broader impact statement names healthcare, hiring, and surveillance, warning that biased teachers can transmit and even amplify their biases through more diverse synthetic samples.

For industry, the practical argument is a synthesis pipeline that scales: cross-class batches make individual IPC slices independently distributable across GPUs, avoiding the intra-batch dependencies that other methods introduce. The trade-off is VRAM, which grows from 6.5 GB to 32.5 GB as BN teachers increase from one to four at a distillation batch size of 100.

Future Directions

• Memory-efficient teacher ensembles. VRAM capacity limits how many BN teachers can be used simultaneously per image; the authors propose model offloading and parameter-efficient fine-tuning as avenues.
• Beyond batch normalization. The framework currently depends on BN statistics, so extending the decoupling to LayerNorm or GroupNorm priors is identified as a natural next step, which would open the method to architectures that do not use BN.
• Cross-architecture evaluation. The paper declines to evaluate with separate downstream architectures such as Vision Transformers or ConvNeXt, arguing that PRISM already embeds heterogeneous CNN priors and that adding more backbones would conflate sources of generalization rather than isolate one. Whether that argument holds empirically is left open.
• Privacy and bias guarantees. The appendix frames architectural noise as an implicit, PATE-like regularizer but without formal guarantees; formalizing that effect and characterizing how diverse teachers transmit or amplify societal bias are both unaddressed.

Target Audience

Researchers working on dataset distillation, dataset compression, and synthetic data generation will find the core contribution directly relevant, as will practitioners of large-scale ImageNet training who need reproducible baselines. The paper is also useful to readers interested in ensemble and multi-teacher supervision, in diversity metrics for synthetic data, and in the privacy and bias considerations of distilled datasets. It is less suited to beginners: the method is described relative to SRe2L's loss formulation, and the evaluation section presupposes familiarity with IPC benchmarking, recovery stages, and knowledge distillation in post-recovery pipelines.

Authors’ abstract

Dataset distillation (DD) promises compact yet faithful synthetic data, but existing approaches often inherit the inductive bias of a single teacher model. As dataset size increases, this bias drives generation toward overly smooth, homogeneous samples, reducing intra-class diversity and limiting generalization. We present PRISM (PRIors from diverse Source Models), a framework that disentangles architectural priors during synthesis. PRISM decouples the logit-matching and regularization objectives, supervising them with different teacher architectures: a primary model for logits and a stochastic subset for batch-normalization (BN) alignment. On ImageNet-1K, PRISM consistently and reproducibly outperforms single-teacher methods (e.g., SRe2L) and recent multi-teacher variants (e.g., G-VBSM) at low- and mid-IPC regimes. The generated data also show significantly richer intra-class diversity, as reflected by a notable drop in cosine similarity between features. We further analyze teacher selection strategies (pre- vs. intra-distillation) and introduce a scalable cross-class batch formation scheme for fast parallel synthesis. Code will be released after the review period.