Self-Supervised Learning: Creating Training Signals From the Data Itself

Self-supervised learning builds its prediction target out of the observation itself, or out of related observations. Predicting masked tokens, future frames, missing image regions, or the representation of another view are all the same construction.

No manual task label is required for the pretraining objective. Human choices still define masking, augmentations, positive pairs, architecture, data collection, and the downstream evaluation.

How far a target the data supplies itself can carry was measured in 2020. Geoffrey Hinton and three co-authors trained SimCLR on ImageNet with no labels at all. The only signal was pairs of differently augmented crops of the same image. A linear classifier fitted on the frozen representation reached 76.5% top-1 accuracy. That was a 7% relative improvement on the previous state of the art, and equal to a supervised ResNet-50. Fine-tuned on 1% of the labels, the same representation reached 85.8% top-5. Labels were not needed to learn the representation. They were still needed to find out whether it was any good.

Pretext tasks are not interchangeable. Different ones push a representation to preserve different information — and none of them can be graded on its own.

A large-scale study established that in 2019, by crossing self-supervised methods against backbone architectures instead of testing either alone. Its finding: “Importantly, neither is the ranking of architectures consistent across different methods, nor is the ranking of methods consistent across architectures.” The best objective changed when the network changed. The best network changed when the objective changed. VGG19-BN was consistently the worst backbone for self-supervision, despite performing comparably to ResNet50 on standard supervised vision benchmarks. Its three authors concluded that pretext tasks “should not be considered in isolation, but in conjunction with underlying architectures”.

So read the four families below as a choice made together with the architecture, not before it. How much a single rule inside one family can matter is visible in masked prediction. Kaiming He and five co-authors reported in 2022: “Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task.” Hiding 75% of an image's patches and reconstructing the missing pixels accelerated training by 3x or more. A vanilla ViT-Huge reached 87.8% ImageNet-1K accuracy, the best among methods using only ImageNet-1K data. Naming the family is not the design decision. The masking ratio is.

The generated target depends on the structure available in the modality. In three of these the exchange has been measured, and the measurement is the point: unlabeled data buys a representation, labels are still spent to find out what it is worth.

The term should describe where the training signal comes from, not merely whether a large dataset is used. Self-supervised targets are generated from the input, or from relationships among inputs. Supervised pretraining takes human or operational labels as its source task. A reconstruction objective reproduces input information, and may or may not be self-supervised.

Each buys a different thing, and each carries its own failure. An objective the designer chose and may have chosen badly. A source label whose bias transfers with the features. A representation that preserves low-level detail and nothing you wanted.

Scale is not only an advantage. Large unlabeled corpora can magnify both useful structure and hidden defects. Six steps run from naming the invariance you want, through protecting semantics and searching for shortcuts, to monitoring collapse, evaluating transfer, and auditing the corpus. Two of them have already been measured for you.

Monitoring collapse has an exact demonstration. Xinlei Chen and Kaiming He trained SimSiam in 2021 with and without a single operation. With stop-gradient it scored 67.7% (± 0.1) on ImageNet linear evaluation. “Solely removing stop-gradient, the accuracy becomes 0.1%, which is the chance-level guess in ImageNet.” The failure had announced itself twice before the accuracy did. The training loss fell immediately to its minimum possible value. The per-channel standard deviation of the l2-normalized output went to zero. Keep both instruments on screen — a loss sitting on its floor, an output standard deviation going to nothing. A perfectly satisfied objective and a worthless representation can be the same run.

Auditing the corpus has a dated precedent. LAION-5B, the scraped image-text dataset used to train Stable Diffusion, was examined in December 2023 in a Stanford Internet Observatory report by David Thiel: “Through this process, we identified 3,226 dataset entries of suspected CSAM, much of which was confirmed as CSAM by third parties.” LAION took the dataset down. On 30 August 2024 it republished the set as Re-LAION-5B, with 5,526,641,167 pairs, after removing 2,236 links matched against hash lists supplied by the Internet Watch Foundation and the Canadian Centre for Child Protection. Nobody had labeled those 3,226 entries into the corpus, because nobody had labeled anything. That is what made the corpus cheap. It is also why the audit had to be run separately from the training.

The training targets are generated automatically. The model does not judge whether the learned representation is fair, factual, safe, or useful. Large-scale pretraining can encode both broad regularities and broad social biases.

Downstream adaptation may change behavior. It does not erase the pretraining distribution. Keep corpus provenance and evaluation attached to the model.

A broad scorecard settles a narrow question. Does pretraining help only one benchmark, or does it support reusable structure?

The first two exams below are listed separately because they have been shown to disagree. Alejandro Newell and Jia Deng evaluated self-supervised pretraining in 2020 across synthetic datasets, with full control of dataset difficulty and label budget. They reported: “We also find that linear evaluation does not correlate with finetuning performance.” The measurement the field uses as its standard yardstick does not tell you what happens once the encoder is allowed to move.