Self-Supervised Learning: Designing the Objective

Self-supervised learning creates targets from structure already present in the data. A system might predict a missing token, reconstruct a masked image region, decide which views belong together, or predict a slowly updated target representation. These targets are generated without manual task labels, but they are not neutral. The objective defines what distinctions the model is rewarded for preserving and what variations it is encouraged to ignore.

The failure that follows is specific, and it has been measured. When random cropping was SimCLR's only augmentation, patches taken from a single image shared a colour distribution distinctive enough that a colour histogram was sufficient to tell images apart. The encoder could therefore satisfy the contrastive objective by matching colour rather than content. The loss fell either way. Nothing in the training curve separates the representation you wanted from the one that reads a colour histogram.

Objective design is therefore a representation-design decision. The question is not whether pretraining converges. It is whether the learned features support the downstream tasks and slices that matter.

Different objective families make different assumptions about useful structure. Each has a result you can look up rather than a claim you have to accept.

Reconstruction gets interesting when you hide most of the input. Masked autoencoders, published in 2021, mask “a high proportion of the input image, e.g., 75%”, which yields “a nontrivial and meaningful self-supervisory task”. Training accelerated “by 3x or more”, and a vanilla ViT-Huge reached “the best accuracy (87.8%) among methods that use only ImageNet-1K data”. The saving is architectural as much as statistical. Their encoder “operates only on the visible subset of patches”: three quarters of the image never reaches it.

The bootstrap branch has a named result of the same kind. BYOL trains an online network to predict the representation of a slow-moving target network, and it does so with no negative pairs at all. The abstract states the claim plainly: “While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them.” DeepMind published it in 2020. It reached 74.3% ImageNet top-1 under linear evaluation with a ResNet-50, and 79.6% with a larger ResNet.

Unusually, the number survived an outside check. Table 4 of Chen and He's 2021 paper is their own reproduction of the competitors, and it puts BYOL at 74.3 after 800 epochs — the highest in the table, against SimCLR at 70.4, MoCo v2 at 72.2, SwAV at 71.8 and SimSiam at 71.3, all ResNet-50. The top row of that table is not the authors' own method.

These questions connect the raw-data structure to downstream requirements. Each of them has, at some point, been answered the wrong way in a published system, which is why they are worth asking on paper before a cluster is booked.

A self-supervised objective can be optimized without learning the abstraction you hoped for. The masking family is the clearest case of an idea that travels across modalities, and the honest way to see what it bought is to attach a label budget to it.

wav2vec 2.0 carried masked prediction into audio in 2020. It masks the speech input in latent space and solves a contrastive task over learned quantised units. Meta's own account of the release describes the rate directly: “About half the audio representations are masked before being fed into the transformer”. The result appears in the paper's abstract as a label budget rather than a loss: “Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER.” Those are LibriSpeech clean and other. With the full labelled set the same abstract gives 1.8/3.3 WER.

The pair of numbers is a run, not a constant. Meta's blog announcing the release reported 5.2/8.6 for the same ten-minute setting — 5.2 percent on clean speech, 8.6 percent on noisy. Same system, same setting, two published pairs. Worth remembering when a headline WER is compared across write-ups.

The evaluation plan should be chosen before expensive pretraining begins.

Step 3 has a canonical shape. SimCLR's headline evidence is a linear probe on frozen features, not a pretraining loss: “A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50.” That figure is the widest model, a ResNet-50 (4×); a standard-width ResNet-50 reached 69.3% under the same linear evaluation. Add a label budget and the probe becomes a fine-tuning result. Google's account of the same run reports that “When fine-tuned on only 1% of the labels, SimCLR achieves 63.0% / 85.8% top-1 / top-5 accuracy, compared to 52.7% / 77.9% from previous best (CPC v2)” — CPC v2 there being a ResNet-161. With standard width the 1%-label numbers were 48.3%/75.5%.

Step 6 is not a formality, and two large evaluations are the reason. The same Edinburgh study tested 13 top self-supervised models on 40 downstream tasks: “We find ImageNet Top-1 accuracy to be highly correlated with transfer to many-shot recognition, but increasingly less so for few-shot, object detection and dense prediction. No single self-supervised method dominates overall, suggesting that universal pre-training is still unsolved.” A second evaluation, from AI2 in 2021, came at it from another direction. It ran “over 700 training experiments including 30 encoders, 4 pre-training datasets and 20 diverse downstream tasks” and concluded that “ImageNet classification does not appear to be a strong indicator of self-supervised encoder performance for various computer vision tasks”. Two independent studies of that size, and in both the ranking changes with the task. One benchmark number is not a selection procedure.

A pretraining objective should be challenged with alternative tasks that expose the invariances it rewards. The surface-shortcut layer is not a thought experiment, and the earliest documented instance was a camera artefact rather than anything about the content of the data.

A network trained to predict the relative location of two image patches found another way to do it. Doersch and colleagues reported the discovery in 2015, in section 3.1: “However, even these precautions are not enough: we were surprised to find that, for some images, another trivial solution exists. We traced the problem to an unexpected culprit: chromatic aberration.” They proved it with an ablation rather than an intuition. A regressor trained to predict a patch's absolute (x, y) position reached RMSE .255 on the top 10% of images, against .371 for always predicting the image centre. Their 'projection' pre-processing blunted the cue and pushed that error back up to .321; colour dropping was the other fix. The shortcut was in the camera, not in the content.

The phenomenon reproduces on demand. Minderer and colleagues noted in 2020 that “Doersch et al. (2015) express their surprise at finding that convolutional neural networks learn to exploit chromatic aberrations to solve the task”, then built a synthetic version. On CIFAR-10 with the Rotation pretext task, an added chromatic-aberration shortcut cut downstream accuracy on frozen features from 81.8±0.30% on clean data to 19.3±2.42%. Their adversarial 'lens' recovered it to 73.5±1.23%. A shortcut took frozen-feature accuracy from 81.8% to 19.3%, and it was invisible in the pretext objective, which the network was solving better than ever.