Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift

AI summary

Overview

Research area: Computer vision / machine learning — specifically dataset distillation, knowledge distillation, and efficient supervision-signal design (arXiv:2512.15647v3 [cs.CV], listed under keywords "Machine Learning, ICML"). Authors are affiliated with the Department of Machine Learning, MBZUAI, Abu Dhabi, United Arab Emirates.

Technical level: Advanced. The paper combines a new training paradigm with formal definitions, bounds, and theorems (Definitions 3.1, 3.3, 3.4; Lemmas 3.2; Theorems 3.5, 3.6, 3.7; Corollary 3.8).

Scope in one sentence: The paper analyzes why using few pre-stored teacher soft labels per image causes "local semantic drift," then proposes the HALD soft→hard→soft training schedule that uses hard labels as a corrective signal, reporting gains on Tiny-ImageNet and ImageNet-1K distillation benchmarks.

What This Paper Is About

In dataset distillation and knowledge transfer, soft labels produced by a teacher model are the standard supervision signal because they carry richer inter-class similarity information than plain class labels. But storing a separate soft label for every image crop is expensive — the paper reports that on ImageNet-1K the distilled data occupies 750 MB while the soft labels occupy 28.33 GB. Reducing the number of crops (and therefore soft labels) per image cuts storage, but the authors show it introduces a subtler problem: crops that show only part of an object can receive soft supervision that drifts toward a different category than the image's true class. The goal of the paper is to characterize this drift theoretically and to correct it by reintroducing hard labels into the training pipeline.

Key Contributions

1. A hybrid training paradigm, HALD (Hard Label for Alleviating Local Semantic Drift). HALD follows a Soft→Hard→Soft schedule: Stage A (soft pretraining) learns from the finite soft-label pool, Stage B (hard-label calibration) uses ground-truth-anchored targets to suppress crop-induced variance, and Stage C (soft refinement) returns to the teacher's soft labels.
2. A formal characterization of Local-View Semantic Drift (LVSD). Definition 3.1 decomposes the finite-crop supervision error into an irreducible oracle gap and a drift term that scales as Tr(Σ)/s, and gives a distribution-free Cantelli bound showing the class-inversion probability decreases monotonically in the number of crops s and vanishes as s→∞.
3. Theoretical guarantees that hard labels help. Theorem 3.5 gives a strictly positive lower bound on the expected mismatch between the finite-crop loss and the ideal full-coverage loss of order Θ(s^(−1/2)); Theorem 3.6 gives an Ω(1/s) lower bound on the excess ideal loss of the finite-crop ERM; Theorem 3.7 and Corollary 3.8 show soft and hard gradients are aligned and that calibration raises the effective sample size to at least s/(1−ρ*²).
4. Extensive empirical validation under aggressive soft-label compression. Experiments across four generation methods (SRe²L, LPLD, RDED, FADRM) and two datasets show consistent improvement over soft-only and other baselines.

Main Findings

• Drift is a real and quantifiable phenomenon. The paper provides an exact error decomposition for the s-crop aggregate p̂_s: E‖p̂_s − e_y‖² = ‖p̄ − e_y‖² (oracle gap, irreducible) + Tr(Σ)/s (LVSD-induced, vanishes as s→∞). Lemma 3.2 shows Cov(p̂_s) = Σ/s and E[‖p̂_s − p̄‖²] = Tr(Σ)/s, so the deviation is strictly positive for any finite s and decays as O(1/s).
• Fewer crops means a larger mismatch with the ideal objective. Theorem 3.5 lower-bounds the expected deviation E[|L_s − L_ideal|] by (σ/√s)·(16/(25√5))·min{1/κ, 1/3}, and Theorem 3.6 lower-bounds the excess ideal loss E[L_ideal(θ̂_s) − L_ideal(θ̂*)] by (1/(2s))·tr(H*⁻¹Σ*) minus C₁/s^(3/2), C₂/s², and C_b·δ_s terms.
• Soft and hard gradients become increasingly aligned during training. Theorem 3.7 bounds the expected cosine similarity between the soft-label and hard-label gradients from below by 1 − (D/m₀)·C_align(x̃, α); Figure 3 shows the measured cosine similarity increasing over training on both real-image crops and optimization-based distilled data.
• Hard-label calibration enlarges the effective sample size. Corollary 3.8 gives s_eff ≥ s/(1 − ρ*²), which the authors say explains the variance reduction and the reduction of overfitting from finite-s soft-label supervision.
• Improved generalization on Tiny-ImageNet (64×64, C=200). In Table 1, at IPC=50 with SLI=2 (SLC=100, 7.60 MB storage) HALD reaches 38.2% versus LPLD's 34.3%, FADRM's 36.0%, RDED's 27.9%, and SRe²L's 31.9%. At SLI=1 (SLC=50, 3.80 MB) HALD reaches 30.7% versus LPLD 24.1%, FADRM 27.8%, RDED 20.5%, and SRe²L 24.0%.
• Improved generalization on ImageNet-1K (224×224, C=1000). In Table 2, at IPC=50 with SLI=5 (SLC=250, 475 MB) HALD reaches 49.5% versus LPLD 39.4%, FADRM 45.5%, RDED 34.9%, and SRe²L 39.5%. At IPC=50 with SLI=10 (SLC=500, 950 MB) HALD reaches 53.7% versus LPLD 48.6% (reported), FADRM 52.7%, RDED 43.5%, and SRe²L 46.8%.
• Headline storage-efficiency claim. The abstract and introduction state that on ImageNet-1K the method achieves 42.7% accuracy with only 285M soft-label storage — a 100× reduction — outperforming prior state-of-the-art LPLD by 9.0%. The truncated Table 2 content provided does not display the row corresponding to 42.7%, so that figure is reported here only as stated in the abstract and introduction text.
• A separate reported comparison at 50-IPC. The paper text states that with SLI=5 and a 50-IPC distilled dataset (SLC=250), HALD reaches 49.5% Top-1 on ImageNet-1K, surpassing the previous SOTA LPLD by +10.1%. Table 2 lists 49.5% for HALD and 39.4% for LPLD in that cell.
• Stage-wise integration beats simple loss mixing. Table 3 compares methods under identical storage and training budgets. At SLC=100: GIFT 27.0, Joint Objective λ=1 5.9, λ=0.1 7.0, λ=0.01 13.1, Soft Only 26.9, Ours 43.5. At SLC=200: 39.1 / 8.1 / 9.4 / 17.5 / 39.2 / 47.3. At SLC=300: 46.7 / 9.5 / 10.3 / 20 (the Table 3 content is truncated at this point).
• Storage cost scales with SLC, not IPC alone. The paper defines SLC = ipc × SLI and Storage(SLC) = SLC·(Cb), arguing that SLC is the proper control for label-side storage because equal SLC yields the same number of stored soft labels per class regardless of IPC.
• Train–test mismatch is visible in loss landscapes. Figure 2 shows train and test loss landscapes on an IPC=10 distilled dataset with SLC=50, comparing finite soft-label coverage with the proposed method.

Methodology in Plain English

The authors start from a simple observation: teacher soft labels are attached to individual image crops, and a crop that shows only part of an object can get soft supervision that points at the wrong class. They formalize this as Local-View Semantic Drift and show mathematically that the damage shrinks as you average over more crops, so when you deliberately use only a handful of crops per image to save storage, the damage becomes systematic. Hard labels, by contrast, are tied to the image's true class no matter how the image is cropped.

Their fix is a three-stage training schedule. In Stage A the student trains normally on the small pool of pre-generated soft labels, which aligns the model so that its gradients from soft supervision start pointing in the same direction as gradients from hard supervision. In Stage B the training temporarily switches to ground-truth-anchored targets — described as "heavily-flattened hard labels," meaning label-smoothed targets rather than strict one-hot vectors, because the synthetic distilled images have less clear semantics and need stronger flattening for stable calibration. Stage B also uses CutMix, so crops and mixing geometry are resampled at every step, giving diverse ground-truth-anchored local views. In Stage C training returns to the soft-label pool to restore fine-grained teacher consistency on the now variance-reduced representation. The epoch split is set by first estimating n_soft, the budget needed to converge on the soft pool, then setting n_hard = n_total − n_soft; Stage A takes floor(n_soft/2), Stage B takes n_hard, and Stage C takes n_soft − floor(n_soft/2). If n_total ≤ n_soft, no hard-label phase is used.

Evaluation covers Tiny-ImageNet and ImageNet-1K, using four distilled-data generation methods (SRe²L, LPLD, RDED, FADRM) spanning synthetic versus real-selected data. Baselines are Soft-Only (soft supervision alone), GIFT (which fuses hard information into soft targets), and a Joint Objective L = L_soft + λ·L_hard with several λ values. Unless stated otherwise, generation methods use the strongest baseline (Soft-Only) for training, while HALD uses FADRM for generation and HALD for training.

Why This Matters

Impact on research. The paper reframes hard labels — often treated as too coarse and discarded in favor of soft labels — as a content-invariant semantic anchor. If the theoretical account holds, it changes how the field should think about label-storage budgets: the interesting quantity is not just how few soft labels you store, but how you compensate for the drift those omissions cause. The result also reframes the storage problem itself, since the paper notes soft labels can exceed the size of the distilled images that motivated distillation in the first place.

Real-world applications (bullets):

• On-device and edge deployment, where a model must be trained or adapted from a very small stored dataset and label-side storage competes directly with image storage.
• Teacher-free post-training pipelines, where the teacher must be isolated from the training stage to avoid information leakage and any direct contact with raw full data, as required by the dataset distillation setting.
• Large-class-scale classification, since ImageNet-1K is used with C=1000 classes, where per-crop logit storage is most punishing.
• Low-resolution or low-data regimes, represented by the Tiny-ImageNet experiments (64×64, C=200), where distillation is often the practical way to obtain a usable training set.

Industry relevance. The core benefit is a cost trade-off: less label storage for better accuracy under identical training budgets, as shown in Table 3. Anywhere dataset distillation is used to shrink training data for cheaper retraining or deployment, the ability to compress the label budget by a reported 100× while improving accuracy is directly relevant to storage and serving costs. Code is released at https://github.com/Jiacheng8/HALD.

Future Directions

• Adaptive stage scheduling. The current schedule derives n_soft, n_hard, T_A, T_B, and T_C from a convergence estimate on the soft pool, with n_hard = 0 when n_total ≤ n_soft. Whether the split can be chosen automatically per dataset, IPC, or SLI is not resolved in the content provided.
• Extension beyond the tested generation families and datasets. Results are reported for SRe²L, LPLD, RDED, and FADRM on Tiny-ImageNet and ImageNet-1K; behavior on other distillation families listed in the related work (gradient matching, meta-model matching, trajectory matching, distribution matching, diffusion-based) is not reported here.
• Relationship to label-fusion alternatives. The paper positions HALD against GIFT (which fuses hard information into soft targets) and against a Joint Objective with λ weights, and Table 3 favors the stage-wise design; the underlying reason for why staged integration beats simultaneous mixing, beyond the gradient-alignment argument, is an open question.
• Tighter characterization of the slack terms. Theorem 3.6's bound includes −C₁/s^(3/2), −C₂/s², and −C_b·δ_s terms whose constants depend on (μ, L_H) and on the local-uniform-concentration assumptions (A1)–(A5); the practical tightness of these bounds under real training is not reported.

Target Audience

Researchers and practitioners working on dataset distillation, knowledge distillation, and efficient supervision-signal design, particularly those concerned with the storage cost of pre-computed soft labels at large class scales. The paper is also relevant to readers interested in the theory of finite-sample supervision — the LVSD definition, the Cantelli-based class-inversion bound, the Ω(1/s) excess-loss result, and the effective-sample-size corollary are self-contained arguments that do not depend on the specific distillation method. Because the paper assumes familiarity with soft labels, ERM, Hessian-based local analysis, and standard distillation benchmarks, it is most accessible to readers with graduate-level machine learning background.

Authors’ abstract

Soft labels from teacher models are a de facto practice for knowledge transfer and large-scale dataset distillation (e.g., SRe2L, LPLD). However, when we limit the number of crops per image to reduce the substantial cost of storing precomputed soft labels, these methods suffer severely from local semantic drift: visually ambiguous crops can cause soft supervision to deviate from the image-level ground-truth semantics, leading to persistent errors and a train-test distribution mismatch. We revisit the overlooked role of hard labels and show that, when properly integrated, they can act as a content-invariant semantic anchor that calibrates such drift. We theoretically analyze the emergence of drift under sparse soft-label supervision and demonstrate that hybridizing hard and soft labels restores alignment between visual content and semantic supervision. Building on this insight, we propose a new training paradigm, Hard Label for Alleviating Local Semantic Drift (HALD), which uses hard labels as intermediate corrective signals while preserving the fine-grained benefits of soft labels. Extensive experiments on dataset distillation and large-scale classification benchmarks show consistent generalization improvements. On ImageNet-1K, our method achieves 42.7% accuracy with only 285M soft-label storage (reduces by 100X), outperforming prior state-of-the-art LPLD 9.0%.