Skip to content
AI.info

Computer vision

Self-Supervised Vision and Foundation Features

Understand contrastive, masked, teacher-student, and image-text pretraining, plus transfer diagnostics for visual foundation features.

By the end you can

Visual

Four ways to manufacture a learning signal

Self-supervision removes manual task labels from pretraining, not assumptions from the system.

Each of the four routes below manufactures its training signal from the data itself. Each one also decides what the representation is allowed to forget. The rest of this lesson is about the published measurements of what those decisions cost.

FigureHierarchy · 4 levels

  • Contrast views

    Bring related augmentations together while separating other examples.

    • Predict missing content

      Reconstruct masked patches, tokens, or latent targets.

      • Match a teacher

        Align a student with a slowly updated or stop-gradient target network.

        • Align modalities

          Pair images with text, audio, or another synchronized source.

Unlabeled does not mean objective-free

The designer still chooses views, masks, positives, negatives, architectures, and data sources. Those choices define what the representation should preserve or ignore.

A model can learn camera identity, border artifacts, or website templates instead of the intended visual structure. The canonical measurement of that is as old as the field's first pretext task. In 2015 Doersch and colleagues trained a network to predict the relative location of two image patches. They had already blocked the obvious cheats. The network found another one:

“However, even these precautions are not enough: we were surprised to find that, for some images, another trivial solution exists. We traced the problem to an unexpected culprit: chromatic aberration.”

The lens, not the object.

That cue is more learnable than it sounds. A regressor trained to predict absolute (x,y) patch coordinates on ImageNet reached a root-mean-square error of .255 on the top 10% of images, against a chance RMSE of .371. The authors' colour-projection preprocessing pushed that error back up to .321. The fix shows up in the pretext metric, because what it removed was real information the network had been using.

Minderer and colleagues at Google Brain restate the finding from outside the original group: “The challenge of avoiding trivial 'shortcut' solutions to pretext tasks was first discussed by Doersch et al. (2015), who described how chromatic aberrations and matching of patterns across boundaries act as shortcuts for predicting the relative location of image patches”. They add the website-template half of the problem. Logos and watermarks are the shortcut for rotation prediction.

Probe the learned invariances rather than assuming they are useful.

Self-supervision moves supervision into the task design.

Comparison

Pretraining families optimize different relationships

No single pretext task dominates every downstream problem. Two separate teams have shown the ranking itself moving under your feet.

The first ranking moved with the backbone. In 2019 Kolesnikov and colleagues at Google Brain ran a large-scale study of self-supervised pretext tasks across CNN architectures. Their conclusion is blunt: “Most importantly, though, we reveal that neither is the ranking of architectures consistent across different methods, nor is the ranking of methods consistent across architectures.” RevNet50 v2 was the best backbone under rotation prediction and nowhere else. Relative patch location was the best method on ResNet50 v1 and not on other backbones.

The second moved with the evaluation protocol instead. He and colleagues report it for ViT-L: MoCo v3 beats masked autoencoding on linear probing, 77.6% against 73.5%. Then it loses at every depth of partial fine-tuning. “MoCo v3 has higher linear probing accuracy; however, all of its partial fine-tuning results are worse than MAE. The gap is 2.6% when tuning 4 blocks.”

So the comparison below is a map of what each family optimizes, not a leaderboard. Fix the backbone and fix the evaluation protocol before you choose. Either one can reverse the answer.

FigureComparison · 4 columns

Contrastive learning

Optimizes relative agreement among views or paired modalities.

  • Strong retrieval geometry
  • Sensitive to view design
  • Batch or queue choices matter
  • Example: image embeddings

Masked prediction

Infers hidden image content from visible context.

  • Uses each image internally
  • Supports encoder pretraining
  • Pixel targets may favor low-level detail
  • Example: masked autoencoder

Teacher-student learning

Matches a target representation without explicit negative pairs.

  • Can avoid large negative sets
  • Collapse prevention is architectural
  • Teacher update matters
  • Example: bootstrap features

Image-text alignment

Learns correspondence from paired language and imagery.

  • Open-vocabulary transfer
  • Inherits caption bias
  • Pair quality matters
  • Example: zero-shot classification

Case

Two pretraining families reached supervised accuracy by different routes

Contrastive learning and masked prediction each have measured results attached to them.

Chen and colleagues published SimCLR in February 2020. The abstract puts two claims in one sentence: “A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.” Parity with supervision under a linear probe. And a hundredfold reduction in labels for a different accuracy target.

Masked prediction gets there another way. He and colleagues, in November 2021, “mask random patches of the input image and reconstruct the missing pixels”. The design choice that makes it work is the severity of the masking: “masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task”. The encoder never sees the masked patches, so the same choice lets them “accelerate training (by 3x or more) and improve accuracy”. At the end of it, “a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data”.

Two different objectives, two different routes, both measured on the same benchmark under stated protocols. Neither result travels to your data by itself.

Key idea

A constant representation can satisfy a badly constrained objective

If every image maps to the same vector, some agreement objectives become trivially small. Successful methods prevent collapse through negatives, stop-gradient paths, variance constraints, predictors, centering, or other design choices.

Xinlei Chen and Kaiming He ran the controlled version of this experiment. SimSiam is a weight-sharing Siamese network with no negative pairs and no momentum encoder. They removed the stop-gradient and changed nothing else. The loss fell to its minimum possible value of −1. The standard deviation of the L2-normalised output went to zero. And the accuracy went with it: “SimSiam achieves a nontrivial accuracy of 67.7%. This result is reasonably stable as shown by the std of 5 trials. Solely removing stop-gradient, the accuracy becomes 0.1%, which is the chance-level guess in ImageNet.” Removing the prediction MLP instead gave the same 0.1%. Their abstract states the conclusion: “Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing”.

A KAIST group re-analysed the same collapse at ICLR 2022, refuting the original explanatory claims and attributing collapse-avoidance to the predictor's de-centering and de-correlation effects. The mechanism is still argued over. The failure is not.

BYOL is the design that removes negative pairs and survives. Grill and colleagues trained an online network “to predict the target network representation of the same image under a different augmented view”, updating the target “with a slow-moving average of the online network”. As the abstract puts it, “While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them”. That was June 2020. It reached 74.3% top-1 on ImageNet under linear evaluation with a ResNet-50, and 79.6% with a larger ResNet. Removing negatives moves the burden onto the rest of the design, and SimSiam shows what it looks like when one piece of that design is taken away.

Preventing collapse is not a footnote for whoever implements it. Monitor feature variance, rank, and neighborhood diversity. The output standard deviation that went to zero in SimSiam is exactly the signal, and it moved before anyone looked at a downstream label.

A low pretraining loss can coexist with useless features.

Analogy

Learning a city from repeated routes

A traveller crosses the same city through rain, night, detours, and different maps. Stable landmarks become useful because they persist across views, while route-specific noise fades.

Rain and night arrive on their own. Augmentations decide which changes count as harmless. A wrong view policy can erase features the downstream task needs. A policy that leaves a nuisance cue intact does the opposite damage: it invites the model to navigate by the cue, the way the patch-location network navigated by chromatic aberration.

The pretraining view policy defines the invariances the model practices.

Example

Tests for representations that learned the wrong thing

Probe features before committing to large downstream experiments.

The site split has a canonical measurement behind it. Zech and colleagues trained pneumonia CNNs on 158,323 chest radiographs from three hospital systems, and published the result in PLOS Medicine in November 2018. Internal AUC beat external AUC in 3 of 5 natural comparisons: the NIH-trained model scored 0.750 at home against 0.695 at Mount Sinai, P < 0.001. Why is not a mystery. “A CNN trained to identify hospital systems accurately identified 22,050 / 22,062 (99.95%, 95% CI 0.9991–0.9997) of NIH, 8,386 / 8,388 (99.98%, 95% CI 0.9991–1.0000) of MSH, and 737 / 771 (95.59%, 95% CI 0.9389–0.9693) of IU test radiographs”. Simply sorting by hospital system scored AUC 0.861 for pneumonia on the joint dataset. That is a classifier that never looks at a lung.

Geirhos and colleagues describe the same events from outside the original team: “The model had unexpectedly learned to identify particular hospital systems with near-perfect accuracy (e.g. by detecting a hospital-specific metal token on the scan...)”.

None of the six probes below would have cost as much as that external validation did, and each answers a different question about what the representation actually carries.

  • Background swap: Replace the scene while retaining the foreground object.
  • Border crop: Remove watermarks, frames, or acquisition overlays — logos and watermarks are the shortcut Minderer and colleagues identified for rotation prediction.
  • Site split: Hold out hospitals, factories, cameras, or websites. This is the test that turned 0.750 into 0.695 for the NIH-trained pneumonia model at Mount Sinai, and the one nobody runs before deployment.
  • Linear probe: Test accessible information under a restricted classifier.
  • Nearest-neighbor review: Inspect whether neighborhoods reflect semantics or data-source artifacts. A neighborhood organised by acquisition site is what an AUC of 0.861 from sorting by hospital looks like from the inside.
  • Small-label curve: Measure transfer as labeled data increases from scarce to abundant.

Steps

Compare transfer strategies without moving the goalposts

Use matched data and metrics for each adaptation regime. And know what each rung of the ladder is worth before you climb it.

He and colleagues measured the whole ladder for ViT-L. Linear probing gives 73.5%. Tuning half a block, the MLP sub-block, gives 79.1%. One full Transformer block gives 81.0%. All 24 blocks give 84.9%. One unfrozen block recovers most of the distance between the frozen probe and full fine-tuning. Their warning about reading one rung off another is flat: “Table 1 shows that linear probing and fine-tuning results are largely uncorrelated.”

That balance is not a constant of nature. It is a property of the encoder you are holding. DINOv2 has the opposite shape: frozen linear evaluation of 86.5% on ImageNet-1k against 88.5% fine-tuned at resolution 224, a gap of 2.0 points. Oquab and colleagues say so directly: “a strong property of our approach is that finetuning is optional”. Meta AI's release post puts the same number in plain words: “DINOv2 allows skipping the model adaptation phase (fine-tuning) — our linear evaluation performance is close to their fine-tuned counterpart (within 2 percent on ImageNet-1k).”

Eleven points for one model, two for another. And the same protocols can rank two methods in opposite orders.

Run all five steps on your own encoder and your own splits. The gap you find is evidence about that encoder in that domain. It is not a general fact about freezing.

FigureProcess · 5 steps

  1. 1. Freeze the encoder

    Train a simple head to measure linearly accessible information.

  2. 2. Tune the head and normalization

    Establish a low-cost adaptation baseline.

  3. 3. Fine-tune selectively

    Unfreeze later blocks or adapters under controlled learning rates.

  4. 4. Fine-tune end to end

    Measure gains, instability, and compute against simpler choices.

  5. 5. Compare domain pretraining

    Test whether in-domain unlabeled data adds value beyond generic features.

Key idea

A celebrated benchmark feature may fail on your acquisition process

Foundation representations are trained on particular images, captions, filters, and legal or social contexts. Transfer depends on domain, granularity, label definition, and nuisance variation.

Those filters are documentable rather than metaphorical. DINOv2's training set, LVD-142M, began as a crawl of web pages for <img> links. What happened to that crawl is on the record: “We discard URLs that are unsafe or restricted by domains, and post-process the downloaded images (PCA hash deduplication, NSFW filtering, and blurring identifiable faces). This results in 1.2B unique images.” A self-supervised retrieval step then selected the training set from that pool, alongside curated sources including ImageNet-22k, the ImageNet-1k train split, Google Landmarks and fine-grained datasets. Meta AI states the funnel plainly: “This approach enabled us to produce a pretraining dataset totaling 142 million images out of the 1.2 billion source images.”

Deduplication, safety screening and face blurring are each a decision about what the frozen features can represent. If your own acquisition process blurs nothing, dedupes nothing, and photographs one factory line under one lamp, the two funnels do not match.

Two teams have measured what that mismatch costs in one target domain. Raghu and colleagues, at NeurIPS 2019, tested transfer on medical images: “A performance evaluation on two large scale medical imaging tasks shows that surprisingly, transfer offers little benefit to performance, and simple, lightweight models can perform comparably to ImageNet architectures.” Ke and colleagues at Stanford then compared 16 convolutional architectures on CheXpert, and found the benchmark ranking carries no information at all for that task: “we find no relationship between ImageNet performance and CheXpert performance for both models without pretraining and models with pretraining”. They also truncated the models to be 3.25x more parameter-efficient on average, with no statistically significant drop.

CLIP is the case usually cited for the other direction, and it is a real one. Radford and colleagues trained it in 2021 on 400 million image-text pairs collected from the internet. Zero-shot, it matched the accuracy of the original ResNet-50 on ImageNet without using any of the 1.28 million training examples that model saw. That is a genuine result about ImageNet. It is not a claim about your acquisition process or your label definitions. Use zero-shot or linear-probe performance as evidence, not a warranty, and maintain domain-specific validation and failure slices.

Pretraining changes the starting point; it does not remove the target-domain test.

Example

Practice: choose pretraining for wildlife cameras

You have ten million unlabeled frames, sparse species labels, day-night variation, and repeated empty backgrounds.

The penalty for getting the camera-location shortcut wrong here has already been measured twice, so size the risk instead of imagining it. Beery, Van Horn and Perona built a wildlife benchmark out of 20 camera-trap locations, split into seen (cis) and unseen (trans) sites: 13,553 training images, 15,827 cis-test images, and 23,275 trans-test images across 9 held-out trans-locations. Same animals, same classifier, new places. “On the cis-location test set we achieve a top-1 error of 20.83%, and a top-1 error of 41.08% on the trans-location test set with a 97% cis-to-trans increase in error.”

Koh and colleagues reproduce the effect independently in the WILDS benchmark, on iWildCam2020-wilds: 182 species across more than 200 training cameras, and macro F1 falling from 47.0 (1.4) in-distribution to 31.0 (1.3) on disjoint camera traps. A different dataset, a different team, the same shortcut.

Design the five items below so that your own numbers are comparable to those. A single reported accuracy that does not say which locations it was measured on tells you nothing about a new camera.

  • Choose a pretext objective and explain what it should preserve.
  • Define augmentations that avoid erasing species-specific markings. The colour-projection fix for chromatic aberration cost measurable pretext accuracy, so state what yours is expected to cost.
  • Add a test for camera-location shortcuts: hold out whole locations the way the cis/trans split does, and report both errors, not their average.
  • Compare frozen, partial, and full fine-tuning on the same splits. For ViT-L one unfrozen block scored 81.0% against 73.5% frozen and 84.9% fully tuned, and frozen and tuned rankings were largely uncorrelated.
  • Specify a representation diagnostic before species classification: feature variance and neighborhood composition, checked against location rather than species.

Key takeaways