Contrastive and Bootstrap Representation Learning

A representation objective is easier to understand as a set of geometric pressures. Two views of the same underlying item are created. Both are encoded into a shared space. Similarity between them is increased. Then something has to stop the encoder from answering with a single point for every input.

That last pressure is where the families diverge: explicit negatives, a variance or covariance constraint, or the asymmetry of a slow-moving target network and a stop-gradient. It is also the part of the design that has been measured rather than assumed. The sections that follow replace an intuition about each pressure with a published number — what a shortcut cue is worth in accuracy, what a positive pair can be reduced to, and what happens the moment the anti-collapse term is deleted.

When two augmented images are declared a positive pair, the method is told that the transformation should not change identity. The declaration can be far thinner than that and still work.

In unsupervised SimCSE the entire positive-pair definition is a second forward pass. The same sentence is encoded twice under standard dropout, and two different dropout masks are the only thing separating the views. That is enough for an average Spearman correlation of 76.3% over seven STS tasks with BERT-base. The supervised variant, which takes entailment pairs from NLI as positives and contradictions as hard negatives, reaches 81.6%. The authors state the mechanism in the abstract, in 2021: “We find that dropout acts as minimal data augmentation and removing it leads to a representation collapse.”

The number did not stay inside one group. The team behind DiffCSE reproduced the unsupervised model in-house a year later at 76.16%, and took 76.25% as the baseline their own method would “raise the averaged Spearman's correlation from 76.25% to 78.49%”.

So the pairing rule is the specification, and it is a weak label smuggled into the data pipeline. In recommender systems, co-consumed items might be treated as related. False positives collapse distinctions that matter. False negatives push related items apart. Instance discrimination, where every other item is treated as a negative, can work surprisingly well at scale, but it becomes a problem when many samples share semantics. Supervised contrastive variants use labels to construct more meaningful positive sets — exactly what supervised SimCSE does when it promotes entailment pairs to positives. Temperature changes how strongly the loss emphasizes the hardest similarities. Batch size or a memory bank changes the number and diversity of negatives. None of this is only an optimization detail. It is the geometry.

Both families learn view-consistent representations. They prevent trivial solutions differently, and the difference has been argued out in public with numbers on both sides.

BYOL showed that explicit negatives are not required. Its online network predicts the target network representation of another augmented view, while the target is updated as “a slow-moving average of the online network”. DeepMind reported the result in 2020: “BYOL reaches 74.3% top-1 classification accuracy on ImageNet using the standard linear evaluation protocol with a standard ResNet-50 architecture and 79.6% with a larger ResNet.” The larger network is a ResNet-200 (2×), 250M parameters. The figure then survived a rival lab. Chen and He at Facebook AI Research reproduced every competitor in-house in 2021, and their Table 4 recovers exactly 74.3% for BYOL at 800 epochs — 66.5%, 70.6% and 73.2% at 100, 200 and 400 — against 70.4% for their improved SimCLR reproduction and 71.3% for SimSiam.

Whether the bootstrap really escapes negatives was then contested. Take the normalisation out of BYOL's MLPs and the method stops working. Abe Fetterman and Josh Albrecht, then at Untitled AI and now Imbue, trained a ResNet-18 with BYOL on STL-10 and published the sweep on 24 August 2020. Batch normalisation in both MLPs gave 57.7% linear-evaluation accuracy. No normalisation gave 28.3%, layer norm gave 29.4%, and the random baseline is 28.8%. Removing batch statistics left the encoder indistinguishable from an untrained one. Batch statistics, they argued, act as implicit negatives.

Two months later DeepMind answered with a batch-independent scheme, group normalisation plus weight standardisation, reaching 73.9% against vanilla BYOL's 74.3% ImageNet top-1 with ResNet-50. The abstract does not hedge: “Our finding disproves the hypothesis that the use of batch statistics is a crucial ingredient for BYOL to learn useful representations.” The dispute was settled the way such disputes are settled. Somebody re-ran the thing without the ingredient.

Each choice should be justified by downstream semantics, and the sensitivity is usually larger than intuition suggests. The rate at which a target network is updated is not a default to inherit. It is a tuned hyperparameter with measured cliffs on both sides.

MoCo's momentum ablation reports ResNet-50 accuracy of 55.2%, 57.8%, 59.0% and 58.9% for momentum m = 0.9, 0.99, 0.999 and 0.9999 with a 4096-key queue. At the bottom of the range it does not degrade, it fails: “When m is too small (e.g., 0.9), the accuracy drops considerably; at the extreme of no momentum (m is 0), the training loss oscillates and fails to converge.” BYOL's Table 5a shows the mirror image on its own scale. Target decay 0.999 gives 69.8%, 0.99 gives 72.5%, 0.9 gives 68.4%. An instantaneously updated target (τ = 0) gives 0.3%, and a frozen random target (τ = 1) gives 18.8% ± 0.7. One number, moved a little, spans the distance between a working encoder and chance.

Treat the embedding as a product with observable neighborhoods, and make the collapse check a measurement rather than an impression.

SimSiam reaches 67.7% ImageNet linear accuracy at 100 epochs. Delete a single element of the architecture, nothing else changed, and Chen and He report what happens: “Solely removing stop-gradient, the accuracy becomes 0.1%, which is the chance-level guess in ImageNet.” The failure is visible before any probe is trained, because the std of the l2-normalised output falls to zero. Every input has been mapped to the same point. Chaoning Zhang and colleagues reproduced the collapse independently in 2022, in symmetric variants on CIFAR-100 with a ResNet-18 — SimSiam at 66.62% top-1 against 1% for Mirror SimSiam, Naive Siamese and Symmetric Predictor — while refuting SimSiam's own alternating-optimisation explanation of why the stop-gradient works.

That is the shape of a good audit step: a statistic that separates a working encoder from a dead one by two orders of magnitude, computed before anything downstream is fitted. Variance and norm statistics first. Then nearest neighbors across slices, then linear and few-label probes, then augmentation-sensitivity tests, then retrieval at the depth the product will actually show. Retest under domain shift last, because neighborhoods reorganize on new devices and populations.

Most contrastive failures begin with assumptions about which views should agree or disagree. Those assumptions are not abstractions. They are a dataset somebody can open.

LAION-5B was assembled by crawling Common Crawl and keeping image–alt-text pairs whose CLIP similarity was high enough. A similarity threshold was the pair rule, and effectively the only gate. On 23 December 2023 David Thiel at the Stanford Internet Observatory published an audit of it: “Through this process, we identified 3,226 dataset entries of suspected CSAM, much of which was confirmed as CSAM by third parties.” LAION withdrew the dataset. On 30 August 2024 it published Re-LAION-5B — 5,526,641,167 pairs, with 2,236 links removed using hash lists from the Internet Watch Foundation and the Canadian Centre for Child Protection, a removal set that “subsume[s] 1008 links found by the Stanford Internet Observatory report in Dec 2023”.

The precedent is the point. A pair generator is a data pipeline with a public record. An outside party can inspect what the rule admitted, and the maintainer can republish with a changed count and a dated explanation. Do the same at your own scale, before tuning the encoder. Define what two positive views must share. Find pairs labelled different that are the same entity. Stress the augmentations for erased target signal. Measure how batch sampling changes negative difficulty and class coverage. Evaluate the frozen geometry with retrieval and probes before any end-to-end fine-tuning. Track feature variance, rank, and target-network behaviour as collapse indicators.