Augmentation, Synthetic Data, and Simulation

Flipping a cat image horizontally usually preserves the label “cat.” Flipping an X-ray or a road sign may change anatomical laterality or symbol meaning.

Every transformation asserts an invariance. The assertion must be justified by the task, sensor, and decision rather than borrowed from a generic recipe.

The assertion is also testable. When it is wrong, the cost shows up in the metric. Later in this lesson, four published geometric recipes measured side by side on chest X-rays each score below doing no augmentation at all. A copy-paste pipeline loses eight points of detection accuracy to nothing but the seams around the pasted objects.

These four methods differ in control, realism, annotation cost, and failure mode.

Geometric transforms alter position, scale, crop, orientation, or perspective. They are useful for viewpoint variation, they can cut away the evidence the label depends on, they oblige you to transform coordinates and masks with the pixels, and they risk producing geometry the camera could never have produced.

Photometric transforms change brightness, contrast, color, blur, or noise. They are useful for capture variation, they can violate physical sensor behavior, they may destroy subtle diagnostic cues, and they risk unrealistic appearance.

Compositing and copy-paste insert foreground objects or regions into new contexts. They offer label control and raise rare-object counts, they require plausible boundaries and occlusion, and they risk paste artifacts becoming shortcuts.

Simulation or generative synthesis renders scenes or generates images from a model. It controls rare scenarios and can vary geometry and metadata, it inherits simulator or generator bias, and it risks the synthetic-to-real gap.

The compositing row has measurements attached to it. In 2021 Google Research pasted objects at random, with no context modelling of any kind, on top of an already strong baseline. The abstract of Simple Copy-Paste reports: “On COCO instance segmentation, we achieve 49.1 mask AP and 57.3 box AP, an improvement of +0.6 mask AP and +1.5 box AP over the previous state-of-the-art.” On rare categories the margin is larger. They beat the LVIS 2020 Challenge winning entry by +3.6 mask AP.

A second organisation reproduced the direction of the effect on its own runs. OpenMMLab's MMDetection reimplementation reports 43.5 box / 39.1 mask AP with standard scale jittering, against 45.1 box / 40.3 mask AP with scale jittering plus Simple Copy-Paste, at a 270k schedule. Rare-object counts are not the abstract virtue of compositing. They are the +3.6 mask AP.

A useful augmentation must survive semantic, physical, and operational scrutiny. Each of these tests has a failure with a number behind it.

Label validity: the transformed example should retain the intended target or receive an updated target. A crop that removes the evidence is not a harder example, it is a wrong one.

Plausibility: the result should belong to conditions that could arise in the deployment process. This is the test the chest X-ray recipes below fail. Rotated and translated radiographs are not images a scanner produces, and the accuracy gap between recipe and no recipe was 93.42% against 97.95%.

Coverage value: the transformation should add variation missing from the observed training set, which is what the +3.6 mask AP on rare LVIS categories buys.

Artifact control: the operation should not introduce a cue that reveals how the sample was produced. On the GMU Kitchens benchmark that cue was worth the difference between 65.9 and 73.7 mAP, with the objects in the identical places both times.

These examples show why augmentation policies must be task-specific. The medical case is not a caution. It is a measurement.

Chest radiography has the sharpest number. In 2021 Elgendi and 13 co-authors compared 17 deep-learning models for COVID-19 detection on chest X-rays, with and without four published geometric augmentation recipes taken from other papers. Their abstract states the result: “The MCC without augmentation (MCC = 0.51) outperformed four recent geometrical augmentations (MCC = 0.47 for Data Augmentation 1, MCC = 0.44 for Data Augmentation 2, MCC = 0.48 for Data Augmentation 3, and MCC = 0.49 for Data Augmentation 4).”

All four recipes lost to the empty policy. Retraining a published model without augmentation raised its accuracy from 93.42% to 97.95%. The McNemar chi-square was 163.2, with p = 2.23 x 10^-37, so the gap is not sampling noise. The authors attribute it to training on rotated and translated X-rays that do not occur in real acquisition. The network spent capacity on a world the scanner never produces.

Treat augmentation changes as experiments with explicit hypotheses.

1. Name the missing variation. Identify the deployment condition the policy is intended to cover. If the condition cannot be named, the recipe is inherited, not chosen. That is how four published geometric recipes ended up on chest X-rays that no scanner produces.

2. Define label transformations. Specify how boxes, masks, keypoints, depth, and metadata change under the operation.

3. Create matched training runs. Hold data split, model, and evaluation protocol constant, so that the only difference is the policy. The 65.9 against 73.7 mAP above is informative precisely because the object placement was identical in both arms.

4. Measure targeted slices. Test the condition the augmentation should improve and the slices it might harm; report both directions.

5. Validate on real evidence. Require gains on independent real samples before accepting the policy. This step now has regulatory weight behind it as well as methodological weight. The US Food and Drug Administration's draft guidance of 7 January 2025 asks sponsors to describe “Data augmentation or synthesis” among the methods applied to inputs and outputs. It also asks them to show that test data are independent of the training data and sequestered from the developers, with the testing data processed only in a way representative of the real-world data the model will meet in use.

If synthetic samples dominate, the model may optimize for their textures, object frequencies, or camera assumptions. Simple ratios such as 50:50 have no universal meaning.

The mechanism has now been measured, in the extreme case where the synthetic data comes from the model's own predecessors. Shumailov and five co-authors reported in Nature in 2024 that “indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear”. They found it across large language models, variational autoencoders and Gaussian mixture models. What vanishes first is the rare end of the distribution, which is exactly the part augmentation was supposed to reinforce.

A separate group reached the same conclusion on images. Alemohammad and seven co-authors at Rice University named it Model Autophagy Disorder, and report that quality or diversity decays within a few generations unless enough fresh real data enters each generation. Two independent teams, two model families, the same failure and the same condition on avoiding it.

So tune the mixture using real validation slices and keep the final real test set untouched. Consider curriculum schedules, synthetic pretraining, or targeted oversampling rather than swapping real data out wholesale.

An audit should make the policy reproducible and expose what it was designed to change. In one regulated domain the contents of that audit are no longer a matter of house style.

The US Food and Drug Administration issued draft guidance on AI-enabled device software functions on 7 January 2025. Its section on data collection states: “Synthetic data used in support of a regulatory submission should be accompanied by a comprehensive explanation of how the data were generated and why they are fit-for-purpose.” The same guidance asks sponsors to describe data augmentation or synthesis among the methods applied to inputs and outputs, and to show that test data are independent of training data and sequestered from the developers.

Fit-for-purpose is the operative phrase. It is what the studies in this lesson were each answering with a number.