Data Augmentation and Learned Invariances

Augmentation is safe only relative to a target and context. The transformations below are not equally risky, and the ones with the strongest published record are not the ones that look most dramatic on a sample image.

The model does not know which examples are “original.” It learns from the mixture produced by the augmentation pipeline and sampling probabilities. A transformation therefore encodes a claim about which changes should leave the target unchanged, transform it predictably, or represent plausible future conditions.

The first augmentation of the modern era was a crop. In 2012 the AlexNet authors cut random 224×224 patches from 256×256 images, and flipped them. That step “increases the size of our training set by a factor of 2048”. Without it, they wrote, the network “suffers from substantial overfitting”. Mixup went further in 2018. It “trains a neural network on convex combinations of pairs of examples and their labels”, which “regularizes the neural network to favor simple linear behavior in-between training examples”. The training distribution then holds examples no camera ever recorded.

The distribution also moves in ways nobody wrote down. In 2019 four researchers took the most ordinary augmentation in vision — the random crop — and measured what it does to the training distribution rather than to any single sample. Their abstract states the finding: “This paper first shows that existing augmentations induce a significant discrepancy between the size of the objects seen by the classifier at train and test time: in fact, a lower train resolution improves the classification at test time!” Nobody wrote a crop policy in order to change apparent object scale. The pipeline changed it anyway. Repairing that single side-effect by fine-tuning at test resolution gave 77.1% top-1 on ImageNet for a ResNet-50 trained on 128×128 images and 79.8% for one trained at 224×224. It took a ResNeXt-101 32x48d to 86.4% top-1 / 98.0% top-5.

Using the right word clarifies the expected model behavior, and each word carries a measured price.

Symmetry can be built into the architecture or learned from augmented data. In 2016 Cohen and Welling measured both. On rotated MNIST their P4CNN reached 2.28% test error against 5.03% for the standard Z2CNN. On CIFAR-10 a p4m ResNet44 reached 6.46% error with no augmentation at all and 4.94% with it, against 9.45% and 5.61% for the planar Z2 ResNet44. The built-in symmetry without augmentation beat the plain network with it. The two mechanisms are not rivals: “G-CNNs benefit from data augmentation in the same way as convolutional networks, as long as the augmentation comes from a group larger than G.”

The same knife has another edge. An invariance the task does not actually have costs accuracy instead of buying it. Weiler and Cesa built models globally invariant to reflections in 2019, and found they could not distinguish reflected digits from merely rotated ones. In their words, “For N = 1 this leads to a worse performance than that of the baseline” — worse than the non-equivariant network. That is the Invariance column's stated risk, “erase meaningful cues”, expressed as a number rather than a caution. Equivariance is the answer when the target must move with the input. Robustness is a claim about nuisance severity. Synthetic diversity is a claim about conditions that were never recorded. The tests differ because the claims differ.

Each stage deserves its own tests. The transformation is chosen and its parameters are sampled. The input is transformed, and the target is transformed with it where the task requires. Only then is the resulting pair checked for whether it still represents the intended task. A pipeline that skips the fourth stage silently mislabels; one that skips the fifth ships the mislabelling into training.

A play rehearsed under different lighting, stage sizes, and background noise teaches its performers what must remain stable. Some changes, however, alter the story itself.

A cast cannot cheat the exercise. A model can, learning the fingerprint the augmentation code leaves behind instead of the invariance that code was meant to teach. The distinction worth holding onto is between nuisance variation and edits that change the task.

Use matched training budgets and isolate one family at a time. State the intended invariance, build label-preservation tests on paired examples, and log the transformation type, probability, severity and target composition actually seen. Run controlled ablations against no augmentation and against each family alone. Only then inspect real slices.

The last step is where policies are most often oversold, and it needs a benchmark rather than an intention. ImageNet-C is that benchmark. Hendrycks and Dietterich published it in 2019: 15 corruption types drawn from noise, blur, weather and digital categories, applied to ImageNet validation images. Their Figure 1 caption sets the scale: “Each type of corruption has five levels of severity, resulting in 75 distinct corruptions.” The dataset ships as five tar archives running from 7.1 GB to 22.6 GB. The authors forbid training on it — “Networks should be trained on datasets such as ImageNet and not be trained on ImageNet-C corruptions” — for exactly the reason this step exists. A model trained on the corruptions it is scored against measures its own augmentation policy, not natural variation. Their reading of the benchmark is worth keeping next to any claim of progress: “from AlexNet to ResNet, corruption robustness in itself has barely changed”.