Fixed Anchors Are Not Enough: Dynamic Retrieval and Persistent Homology for Dataset Distillation

AI summary

Overview

Research area: Computer vision, specifically dataset distillation (DD), with tools borrowed from topological data analysis and generalization theory.

Technical level: Advanced. The paper assumes familiarity with decoupled dataset distillation, Rademacher complexity bounds, and persistent homology (filtration, Betti curves, persistence images).

One-sentence scope: The paper introduces RETA, a decoupled dataset distillation framework that replaces the statically preselected real patches used in residual-matching pipelines with a per-stage, per-class retrieval step plus a differentiable persistent-homology regularizer, and evaluates it on CIFAR-100, Tiny-ImageNet, ImageNette, ImageWoof, and ImageNet-1K.

What This Paper Is About

Decoupled dataset distillation compresses a large real corpus into a small set of synthetic images by matching the statistics of a frozen teacher network, avoiding expensive bilevel optimization. The paper argues that the current residual-matching pipeline (FADRM) is limited because it always injects the same statically chosen real patch into the synthetic image at every residual step, which creates a fit–complexity gap and a pull-to-anchor effect that shrink intra-class diversity and weaken generalization. RETA's goal is to make that anchor selection adaptive — choosing which real patch to connect at each stage — while adding a global regularizer that keeps the class geometry of the synthetic set faithful to the real data.

Key Contributions

1. Analysis of two failure modes. The paper theoretically characterizes a fit–complexity gap (fixed anchors either misalign with current synthetic features or inflate hypothesis complexity) and a pull-to-anchor effect (repeated anchoring contracts intra-class synthetic features, causing premature cluster merging), both of which erode intra-class diversity and generalization.
2. Dynamic Retrieval Connection (DRC). A module that builds per-class pools of real patches, embeds the current synthetic image and all candidates with a frozen teacher, and selects the anchor that minimizes a fit–complexity score trading off feature fit against an injected-complexity penalty.
3. Persistent Topology Alignment (PTA). A differentiable persistent-homology regularizer that builds a class-balanced mutual k-NN graph on teacher features, computes persistence images of connected components and loops, and penalizes discrepancies between real and synthetic topological signatures.
4. State-of-the-art results. RETA reports gains over the strongest decoupled baseline (FADRM+) across CIFAR-100, Tiny-ImageNet, ImageNette, ImageWoof, and ImageNet-1K on ResNet-18/50/101, including 64.3% top-1 on ImageNet-1K with ResNet-18 at 50 images per class, which the paper states is +3.1% over the best prior.

Main Findings

• Main accuracy comparison. On Table 1, RETA reports the best result in every setting and every backbone tested. With ResNet-18, RETA reaches 42.4 / 70.3 / 73.6 on CIFAR-100, 43.7 / 56.2 / 61.3 on Tiny-ImageNet, 42.7 / 72.5 / 87.2 on ImageNette, 24.7 / 60.3 / 74.4 on ImageWoof, and 16.8 / 53.2 / 64.3 on ImageNet-1K for IPC = 1 / 10 / 50. Gains over FADRM+ on ResNet-18 are +1.8, +2.4, +2.3 (CIFAR-100), +3.3, +3.4, +0.1 (Tiny-ImageNet), +2.1, +3.5, +1.6 (ImageNette), +1.9, +3.0, +1.8 (ImageWoof), and +2.1, +2.3, +3.1 (ImageNet-1K). Gains are described as most pronounced on smaller backbones and higher-resolution datasets, and persist with ResNet-101 (typically +0.9 to +3.2 points).

• Cross-architecture generalization. On ImageNet-1K with IPC = 10 (Table 2), RETA is best on every architecture tested: EfficientNet-B0 (39.6M parameters) 53.6, MobileNetV2 (3.4M) 47.4, ShuffleNetV2-0.5x (1.4M) 33.6, Swin-Tiny (28.0M) 41.5, Wide ResNet50-2 (68.9M) 60.7, DenseNet121 (8.0M) 56.8, DenseNet169 (14.2M) 59.4, DenseNet201 (20.0M) 60.8. Gains over FADRM+ range from +0.9 to +3.4 points, largest on ShuffleNetV2, and hold for both convolutional and transformer backbones.

• Component ablation and synergy. Under IPC = 10 with ResNet-18 using FADRM as baseline (Table 3), DRC alone gives 69.0 (+1.1) / 54.5 (+1.7) / 70.9 (+1.9) / 51.8 (+0.9) and PTA alone gives 68.5 (+0.6) / 53.7 (+0.9) / 69.8 (+0.8) / 51.6 (+0.7) on CIFAR-100 / Tiny-ImageNet / ImageNette / ImageNet-1K. Enabling both gives 70.3 (+2.4) / 56.2 (+3.4) / 72.5 (+3.5) / 53.2 (+2.3). The paper notes the combined improvement exceeds the sum of the individual gains, with the largest boost on Tiny-ImageNet.

• Hyperparameter optima are interior. For the DRC trade-off coefficient λ on ImageNet-1K at IPC = 10, accuracy rises from 51.7% at λ = 0 to a peak of 52.9% at λ = 0.1, then falls to 50.8% at λ = 1.0; λ = 0.1 is adopted as default. For the PTA weight λ_topo, accuracy rises from 51.7% at λ_topo = 0.1 to a peak of 52.5% at λ_topo = 0.5, then declines to 51.1% at λ_topo = 3.0; λ_topo = 0.5 is adopted as default.

• Computational cost. On ImageNet-1K, RETA runs at 1.31 s per image with 13.4 GB peak GPU memory, versus 1.09 s and 11.0 GB for FADRM+. For comparison, EDC is reported at 4.99 s and 17.9 GB and CV-DD at 8.20 s and 23.4 GB. The paper describes the overhead over FADRM+ as modest given the accuracy gains (e.g., +2.3 points on ImageNet-1K at IPC = 10).

• Robustness to corruptions. Training a ResNet-18 on distilled sets with IPC = 1 and evaluating on ImageNet-Subset-C averaged over 15 corruption types and 5 severities, RETA is best on all corrupted subsets listed (ImageNette-C, ImageWoof-C, ImageFruit-C, ImageYellow-C, ImageMeow-C, ImageSquawk-C), outperforming MTT, IDC, and FADRM+. Mean improvement over FADRM+ is about +3.2 points, with the largest margins on ImageFruit-C and ImageYellow-C. (The per-dataset numbers in Table 4 are not reproduced in the provided content.)

• Continual learning utility. In a class-incremental continual learning setting on ImageNet-1K where distilled images serve as the replay buffer, RETA achieves higher test accuracy than an EDC-based buffer and a random real-image buffer across buffers of 200 to 1000 stored samples. The advantage is already visible at 200–400 samples, and RETA keeps improving as the buffer grows while EDC and random selection saturate or slightly degrade.

• Qualitative difference in synthetic images. Visualizations contrast FADRM, which produces visually tidy but uniform prototypes with reduced intra-class diversity, against RETA, which produces richer structural variation in viewpoints, textures, and layouts without losing class semantics.

Methodology in Plain English

The starting point is a decoupled distillation pipeline where synthetic images are optimized by pixel updates against a frozen teacher's statistics, but every so often a real image patch is blended back into the synthetic image through a residual connection (a convex combination with weight α). FADRM uses a fixed, preselected patch for this; RETA makes two changes.

First, Dynamic Retrieval Connection prebuilds a pool of real patches for each class — one 1×1-grid patch per real image, i.e., the whole image. At each residual step, a frozen encoder embeds the current synthetic image and every candidate patch in that class's pool, and a score decides which candidate to use. The score combines two terms: how close the candidate's normalized embedding is to the synthetic image's embedding (the fit term), and a complexity term measuring how much the candidate's gradient magnitude fluctuates spatially, even after Gaussian smoothing. A scalar λ balances the two. The winning patch is resampled to the current resolution and blended in. Intuitively, the fit term pulls down the bias of the connection, and the complexity term stops the connection from injecting noisy, sharp-edged content that would inflate the hypothesis class.

Second, Persistent Topology Alignment addresses a global problem: because retrieval favors anchors already close to the queries in teacher space, repeatedly blending them in contracts intra-class features, and distinct clusters merge too early. The authors describe this as connected components merging at smaller radius than in real data (a left shift of the component Betti curve) and loops disappearing earlier (suppressed loop Betti curve). To counteract it, they subsample equal numbers of real and synthetic teacher features per class, build a class-balanced mutual k-NN graph, induce a Vietoris-Rips-like filtration by thresholding edges and completing cliques, and compute persistence diagrams for degree 0 (components) and degree 1 (loops). Each diagram is converted into a persistence image on the birth-persistence plane using Gaussian kernels over grid centers with bandwidth σ and persistence weights. The loss is the squared difference between the real and synthetic persistence images, summed over classes, with a loop weight γ, and is added to the distillation objective with weight λ_topo.

The teacher encoder stays frozen but remains in the computation graph, so gradients from the topology loss flow back to the synthetic inputs. To keep cost manageable, the authors cache features and refresh them every T steps, using k ≈ 10–20 and a 32×32 persistence-image grid, applying the topology loss at stage ends or every T steps. The authors also present Theorem 4.1, a Rademacher-complexity bound showing the post-connection risk decomposes into a term scaled by (1−α) times the real-versus-synthetic complexity difference plus L·α(1−α)·Δ, where Δ is the average fit gap; the proof is referred to FADRM rather than rederived.

Why This Matters

Impact on research. The paper reframes residual matching in decoupled dataset distillation as an anchor-selection problem rather than a fixed design choice, and it introduces persistent homology as a differentiable regularizer for feature geometry in distillation. This connects dataset distillation to topological data analysis in a way that other distillation methods could reuse, and it supplies an explicit complexity-based argument for why fixed anchors are suboptimal.

Real-world applications (bullets):

• Compressing large image corpora into small synthetic surrogates for cheaper model training and repeated experimentation on limited hardware.
• Continual learning systems that need a small, high-value replay buffer — the paper's ImageNet-1K class-incremental experiment stores only 200 to 1000 samples.
• Datasets for robustness testing, since distilled sets with better topology preservation transferred better under 15 corruption types and 5 severities on ImageNet-Sub

Authors’ abstract

Decoupled dataset distillation (DD) compresses large corpora into a few synthetic images by matching a frozen teacher's statistics. However, current residual-matching pipelines rely on static real patches, creating a fit-complexity gap and a pull-to-anchor effect that reduce intra-class diversity and hurt generalization. To address these issues, we introduce RETA -- a Retrieval and Topology Alignment framework for decoupled DD. First, Dynamic Retrieval Connection (DRC) selects a real patch from a prebuilt pool by minimizing a fit-complexity score in teacher feature space; the chosen patch is injected via a residual connection to tighten feature fit while controlling injected complexity. Second, Persistent Topology Alignment (PTA) regularizes synthesis with persistent homology: we build a mutual k-NN feature graph, compute persistence images of components and loops, and penalize topology discrepancies between real and synthetic sets, mitigating pull-to-anchor effect. Across CIFAR-100, Tiny-ImageNet, ImageNet-1K, and multiple ImageNet subsets, RETA consistently outperforms various baselines under comparable time and memory, especially reaching 64.3% top-1 accuracy on ImageNet-1K with ResNet-18 at 50 images per class, +3.1% over the best prior.