[Re] Pure Noise to the Rescue of Insufficient Data

AI summary

Overview

Research area: Computer Vision — long-tailed (class-imbalanced) image classification, specifically a reproducibility study of the OPeN method ("Oversampling with Pure Noise Images") submitted to the ML Reproducibility Challenge format.

Technical level: Intermediate. Readers should be comfortable with convolutional networks, batch normalization, resampling/re-weighting strategies for imbalanced data, and standard CIFAR-style benchmark protocols.

Scope (one sentence): The paper re-implements and empirically tests the four main claims of the original OPeN paper across CIFAR-10-LT and CIFAR-100-LT at imbalance ratios 50 and 100, then extends the analysis with a ResNet-32 architecture, an investigation of dataset-generation variance, and a study of how OPeN shifts model priors.

What This Paper Is About

Real-world image datasets often have long-tailed label distributions, and standard deep networks become biased toward majority classes, performing poorly on rare ones. The original OPeN paper proposed replacing some oversampled minority-class images with pure noise during training, paired with a special batch normalization layer (DAR-BN) that normalizes noise and natural images separately, claiming this improves minority-class accuracy. This paper independently rebuilds that pipeline from scratch (the original code repository was not populated) and tests whether the claims hold, where they break down, and why the method might work.

Key Contributions

1. Independent re-implementation and verification of four claims. The authors rebuilt the full training pipeline in PyTorch — including the CIFAR-10/100-LT dataset construction, WideResNet-28-10, the pure-noise sampling snippet, and DAR-BN — and reproduced the original accuracy figures to within 0.6% on the primary CIFAR-10-LT (IR=100) comparison of Deferred Resampling (DRS) and OPeN.

2. Architecture transfer to ResNet-32. Because prior long-tailed work used ResNet-32 rather than WideResNet-28-10, the authors re-ran OPeN on ResNet-32 to place it on the same footing as LDAM-DRW and M2m, finding that OPeN's advantage shrinks under the smaller network and simpler augmentation.

3. Discovery of a reproducibility hazard in CIFAR-10-LT construction. The authors show that different random seeds used to subsample the balanced CIFAR-10 dataset produce long-tail variants sharing on average only around 25% unique images, and that this alone shifts accuracy noticeably.

4. A prior-shift analysis of why OPeN helps. Through t-SNE embeddings, prediction histograms on noise and out-of-distribution inputs, and confusion matrices, the authors test three hypotheses about how OPeN changes what the model predicts.

Main Findings

• OPeN beats resampling baselines on CIFAR-10-LT (IR=100). Reported versus reproduced accuracy: ERM 79.6 vs 81.18, RS 75.1 vs 74.82, DRS 83.0 vs 83.22, OPeN 84.6 vs 85.04. The DRS and OPeN figures land within 0.6% of the reported values.

• The gain comes from minority classes. Compared with DRS, OPeN improves accuracy on the two least frequent classes by 8.2% while sacrificing only 0.9% on the two most frequent classes.

• The gain holds across imbalance ratios and datasets. On CIFAR-10-LT (IR=50): OPeN 87.8 versus DRS 86.9, ERM 84.9, RS 80.9. On CIFAR-100-LT (IR=100): OPeN 52.1 versus DRS 50.8, ERM 47.1, RS 41.6. On CIFAR-100-LT (IR=50): OPeN 56.5 versus DRS 55.8, ERM 52.7, RS 46.5.

• DAR-BN is essential, not incidental. On CIFAR-10-LT, Standard BN reaches 81.81, Auxiliary BN 82.23, and DAR-BN 85.04 — without DAR-BN, OPeN falls below the DRS baseline of 83.22. The same ordering holds on CIFAR-100-LT (49.26 / 51.27 / 52.12) and on ResNet-32 (74.37 / 75.08 / 77.52).

• Robustness to augmentation strength is confirmed. Flip-and-Crop: ERM 74.6, DRS 75.4, OPeN 79.9. Add Cutout: 78.7 / 79.5 / 83.9. Add SimCLR: 80.7 / 83.2 / 84.3. The authors excluded AutoAugment because it was optimized on the full balanced dataset and is therefore unfair for an imbalanced subset.

• Pure noise helps on balanced data — with WideResNet. On full balanced CIFAR-10 with WideResNet, adding noise raised accuracy from 87.16 to 88.57 (+1.6%), versus the +0.9% reported by the original authors (who did not report absolute accuracies).

• The balanced-data benefit does not transfer to ResNet-32. With ResNet-32, accuracy went from 86.51 without pure noise to 86.19 with it, a change of −0.37%.

• The advantage narrows on ResNet-32 against prior art. OPeN reaches 77.52 on CIFAR-10-LT (IR=100), above its own ERM (71.70), RS (70.09), and DRS (75.78), but below the previously reported LDAM-DRW (77.03) and M2m (78.3 ± 0.16) when those are considered alongside M2m's own ERM (68.7 ± 1.43) and DRS (75.2 ± 0.26) numbers. The authors caution that M2m used a different dataset variant.

• Dataset seed matters. Training on the CIFAR-10-LT variant from Cui et al. gives ERM/DRS/OPeN of 79.26 / 80.87 / 84.19; from M2m's variant 78.37 / 81.64 / 87.11; from LDAM's variant 81.18 / 83.22 / 85.04. The authors used the LDAM variant because it was closest to the original numbers.

• Normalization statistics matter too. Using the full CIFAR-10 statistics instead of the standard values drops ERM/DRS/OPeN to 80.50 / 82.39 / 84.72; using statistics computed from CIFAR-10-LT (IR=100) itself drops them to 79.28 / 81.03 / 84.12, though OPeN's ranking is preserved.

• OPeN shifts priors, but not by embedding noise near minority classes. t-SNE shows noise images cluster together, separable from all class clusters, and out-of-distribution images remain dispersed — the same pattern appears under DRS, so it is not unique to OPeN. However, models trained with OPeN are less likely to classify OOD images as majority classes, all 1000 noise images are predicted as classes 2, 4, or 6, and the confusion matrix shows fewer minority-class images misclassified into majority classes.

• More noise is not monotonically better. Accuracy by noise ratio: 1/6 → 83.23, 2/6 → 85.04, 3/6 → 84.34, 4/6 → 84.23, 5/6 → 83.31, 6/6 → 80.01. The default of 1/3 (i.e. 2/6) is optimal, and train accuracy falls as the noise ratio rises.

• Batch size 128 is best among those tested. 32 → 74.38/80.15/80.50, 64 → 78.69/82.89/83.89, 128 → 81.18/83.22/85.04, 256 → 79.11/75.28/82.36, 512 → 75.19/71.39/79.22 (ERM/DRS/OPeN).

Methodology in Plain English

The team started from the original paper's description rather than its code, since the linked repository was empty. They took long-tailed dataset generation from prior work (LDAM), took a WideResNet-28-10 implementation from prior work, and swapped its batch normalization layers for DAR-BN. They reused the original authors' two published helper functions: one that, given a batch, picks indices and replaces natural images with pure noise, and one that applies DAR-BN given a batch, noise indices, and a BN layer.

The core training recipe: for the first 160 epochs the model trains normally with SGD (momentum 0.9, weight decay 2e-4, initial learning rate 0.1, linear warmup for 5 epochs, batch size 128, dropout 0.3). At epoch 160 the learning rate decays, and from that point the model switches to oversampling — either resampling natural minority images (DRS) or replacing a fraction δ = 1/3 of oversampled images with fresh pure noise (OPeN). The replacement probability for class i is scaled by (1 − n_i / max_j n_j) · δ, so rarer classes get more noise substitution. Noise images are regenerated at the start of each epoch.

For evaluation they used the original balanced CIFAR-10/100 validation sets of 10000 images, which penalizes models that overfit to majority classes. Beyond reproducing the four claims, they ran ablations on batch normalization type, augmentation strength, batch size, noise ratio, input normalization statistics, and dataset seed, and they swapped WideResNet for ResNet-32 to compare against published baselines. They contacted the original authors by email to confirm undocumented settings: batch size 128, 5-epoch warmup, no effective-number-of-samples weighting, oversampling that does not increase examples seen per epoch, learning rate 0.001 for the balanced-data experiments, and dropout 0.3. Compute totaled roughly 432 GPU hours on a single NVIDIA RTX A5000 (24 GB VRAM, 12 vCPU, 62 GB RAM); a bare ERM run took about 1 hour 50 minutes, deferred oversampling 22 minutes, and OPeN 45 minutes from the epoch-160 checkpoint.

Why This Matters

This is a replication of a method that claims a counterintuitive result — that training on pure random noise can improve accuracy on rare classes — and replication is what separates a durable technique from a lucky run. The paper confirms the claims while adding two corrections that affect how future results in long-tailed classification should be read: reported gains depend on which CIFAR-10-LT subset was used, and on which normalization statistics were applied. It also shows OPeN's benefit overlaps substantially with the benefit of a bigger network or stronger augmentation, which changes how the method should be positioned.

Real-world applications of long-tailed image classification, the setting this work addresses:

• Rare species or rare-condition recognition, where some categories have far fewer labeled examples than others.
• Medical imaging, where uncommon findings are underrepresented relative to common ones in training archives.
• Industrial defect detection, where defect classes are intrinsically rare compared with normal samples.
• Biased-source visual datasets, where the acquisition process favors a few labels rather than reflecting true class frequency.

Industry relevance: the finding that noise augmentation can supplement — but partially substitutes for — gains from larger models and heavier augmentation means teams with constrained compute or data budgets should not assume OPeN is a free add-on. The reproducibility findings on dataset seeds and normalization statistics directly affect benchmark comparability for anyone publishing or evaluating long-tailed methods.

Future Directions

• Disentangle OPeN from model capacity and augmentation strength. The authors' own ResNet-32 results suggest the improvement is partly orthogonal to the gains from bigger networks or richer augmentation; a systematic sweep over both would quantify the overlap.
• Explain the prior shift mechanistically. OPeN makes the model predict minority classes more often for noise, OOD, and misclassified inputs, but not by embedding noise near minority classes. What representation change causes this remains open.
• Standardize long-tailed dataset construction. The authors explicitly call for future work to report which CIFAR-10-LT subset and which normalization statistics were used, and they publish their image indices to make this possible.
• Extend beyond CIFAR. The original paper also used CelebA-5, ImageNet-LT, and Places-LT; this reproduction covered only CIFAR-10-LT and CIFAR-100-LT, since the original's ablations were confined to those two.

Target Audience

Most useful to researchers and engineers working on class-imbalanced or long-tailed recognition who are considering OPeN or DAR-BN and need to know what gains to expect and under what conditions. It is also valuable to reproducibility practitioners and benchmark maintainers, because of its concrete documentation of how dataset sampling seeds and normalization statistics silently shift reported accuracies. Readers wanting an example of a well-executed replication study — including what to do when the original code is unavailable — will find the methodology and the "what was easy / what was difficult" sections instructive.

Authors’ abstract

We examine the main claims of Zada, Benou, and Irani (2021), which states that in an image classification task with imbalanced training data, (i) using pure noise to augment minority-class images encourages generalization by improving minority-class accuracy. This method is paired with (ii) a new batch normalization layer that normalizes noise images using affine parameters learned from natural images, which improves the model's performance. Moreover, (iii) this improvement is robust to varying levels of data augmentation. Finally, the authors propose that (iv) adding pure noise images can improve classification even on balanced training data. We implemented the training pipeline from the description of the paper using PyTorch and integrated authors' code snippets for sampling pure noise images and batch normalizing noise and natural images separately. We reproduced the main claims that (i) oversampling with pure noise improves generalization by improving the minority-class accuracy, (ii) the proposed batch normalization (BN) method outperforms baselines, (iii) and this improvement is robust across data augmentations. Our results also support that (iv) adding pure noise images can improve classification on balanced training data. However, additional experiments suggest that the performance improvement from OPeN may be more orthogonal to the improvement caused by a bigger network or more complex data augmentation.