Understanding Self-supervised Contrastive Learning through Supervised Objectives

AI summary

Overview

Research area: Self-supervised representation learning, contrastive learning theory, and its relationship to supervised learning objectives.

Technical level: Advanced. The paper is built around formal problem formulations, theorems with proofs in the appendix, and an empirical study. Readers need comfort with cosine similarity, expectation notation, Jensen's inequality, and InfoNCE-style losses to follow the derivations, though the empirical section is readable by a broader audience.

Scope: The paper reframes self-supervised contrastive learning as an approximation of a supervised prototype-based representation learning objective, derives an InfoNCE-like loss from that approximation, and empirically validates the resulting design insights (augmentation, similarity measure, class balance, and balancing parameters).

What This Paper Is About

Self-supervised learning has succeeded empirically but has advanced largely through architectural tricks rather than formal objectives, unlike supervised learning which is grounded in empirical risk minimization. This paper asks whether self-supervised representation learning can be understood as an approximation to a supervised representation learning problem, where the supervision is implicitly constructed rather than given. Starting from a supervised objective defined over similarities to class prototypes, the author replaces the unavailable true prototype with a surrogate built from augmented views of the same image, and then shows how the resulting upper bounds lead to losses resembling InfoNCE / NT-Xent.

Key Contributions

1. A theoretical framework that formulates self-supervised representation learning as an approximation of supervised representation learning, from which a contrastive loss closely related to InfoNCE is derived.
2. A perspective on common contrastive learning practices, specifically representation normalization (cosine similarity) and the use of balanced datasets, showing how they correspond to components of the framework.
3. The introduction of prototype representation bias, defined as the expected distance between the true prototype representation and the surrogate prototype representation, with an observed correlation to downstream accuracy.
4. The proposal of a balanced contrastive loss as a natural extension of InfoNCE, along with the observation that improved balancing of the attracting and repelling components leads to better performance.

Main Findings

• Derivation of an InfoNCE-like loss: The triplet loss with pseudo-labels is shown to be an approximation to an InfoNCE-type loss with samples. The total derived loss takes a form similar to NT-Xent in SimCLR.
• A decoupled formulation: In the derived loss, the positive pair is explicitly excluded from the denominator's first summation, yielding a formulation that coincides with the decoupled contrastive loss proposed by Yeh et al. (2022), who empirically showed summing over [m] \ {i} performs better than over [m].
• Prototype representation bias and augmentation: Lower prototype representation bias tends to result in higher accuracy. SimCLR's default augmentation ("base") achieved both the highest accuracy and the smallest bias; removing Gaussian blur or color distortion, or adding random cutout, random rotation, or Gaussian noise, did not improve accuracy.
• Normalization and cosine similarity matter: Comparing similarity measures produced top-1 accuracies of 65.98 for cosine similarity with normalization, 0.43 for dot product without normalization, and 10.63 for negative Euclidean distance without normalization.
• Balanced classes help: With training sets containing the same number of images (115,846, which the paper states is 9% of the ImageNet training set) but different class distributions, the uniform class distribution gave 20.82 top-1 accuracy versus 13.65 for the long-tailed distribution.
• Siamese and asymmetric encoders: The asymmetric attracting objects in the formulation—f_θ(t(x)) and E_T f_θ(T(x))—explain why both symmetric and asymmetric (teacher/student) designs coexist in the literature, since the expectation term has lower variance.
• Balancing parameter grid: Evaluating (α, λ) with α, λ ∈ {1,2,4,8}, the balanced contrastive loss peaked at (α, λ) = (4, 2) while the generalized NT-Xent variant peaked at (2, 2). In both cases the best accuracy was not at λ = 1. The paper notes ImageNet has 1,000 classes and chance-level top-1 accuracy is 0.1%.

Methodology in Plain English

The author first writes down what supervised representation learning would look like if an oracle supplied an ideal "prototype" point for each visual concept: pull each image's representation toward its own class prototype (attraction) and push it away from the closest prototype belonging to a different class (repulsion). Since no oracle exists, the prototype for a label is estimated as the average representation over images of that label; the true prototype used in the repelling term is left as is, but the attracting target is swapped for a surrogate—the average representation over augmented views of the same single image.

Two upper bounds are then derived. The attracting term's bound (Theorem 4.4) uses cosine similarity, l₂-normalized representations, and a technical assumption that the augmented representation and the averaged representation lie in the same hemisphere. The repelling term's bound (Theorem 4.6) uses the balanced-label assumption and a log-sum-exp approximation of the maximum, with α controlling how sharply high-similarity negatives are weighted and ν a minimum-norm quantity over other classes. Combining the bounds and rearranging gives a per-sample loss resembling NT-Xent, and replacing expectations with sample averages gives a practical loss.

For the experiments, the author uses SimCLR with temperature τ = 0.5, ImageNet as the dataset, and ResNet-50 as the backbone, evaluating top-1 accuracy via linear evaluation while holding all settings fixed except the factor under study. The factors varied are the data augmentation strategy (six scenarios: base, minus Gaussian blur, minus color distortion, plus random cutout, plus random rotation, plus Gaussian noise), the similarity measure (three variants), the class distribution (uniform versus long-tailed), and the balancing parameters α and λ over a grid.

Why This Matters

Impact on research: The paper supplies a principled derivation connecting supervised prototype-based objectives to contrastive losses, which lets widely used design choices (cosine similarity with normalization, balanced datasets, Siamese architectures, temperature-like parameters) be read as consequences of an approximation argument rather than as heuristics. The introduction of prototype representation bias offers a concrete diagnostic quantity that correlates with downstream accuracy.

Real-world applications:

• Training vision encoders when labeled data is scarce or expensive, where knowing which augmentations reduce prototype representation bias can guide augmentation design.
• Large-scale retrieval or embedding systems that rely on contrastive pretraining and are sensitive to normalization and class balance in the training corpus.
• Domains where annotation is labor-intensive, subjective, or error-prone—the paper explicitly motivates self-supervised learning via the cost and generalization limits of human annotation.
• Any pipeline that pretrains on naturally imbalanced or long-tailed data, where the reported gap between uniform (20.82) and long-tailed (13.65) distributions signals a practical consideration.

Industry relevance: The empirical study is conducted at a scale that is directly relevant to practitioners: ImageNet, ResNet-50, SimCLR, linear evaluation. Hyperparameter recommendations such as (α, λ) = (4, 2) for the balanced contrastive loss and (2, 2) for the generalized NT-Xent variant are actionable, and the 0.1% chance-level reference on 1,000-class ImageNet indicates the observed gains are meaningful.

Future Directions

1. Extending the framework beyond the assumptions used here—the balanced-label assumption is explicitly experimentally motivated but restrictive, and the "same hemisphere" technical assumption may not hold for aggressive augmentations.
2. Investigating why enriching augmentation with random cutout, random rotation, or Gaussian noise did not improve accuracy, and whether the mismatch between surrogate and true prototype representations can be reduced by other means.
3. Developing prototype representation bias into a practical model-selection or augmentation-selection criterion, beyond its demonstrated correlation with accuracy.
4. Exploring whether further performance gains are achievable through better balancing of attracting and repelling forces, which the paper suggests as a route to improving contrastive losses.

Target Audience

Researchers and graduate students working on self-supervised learning, contrastive losses, and representation learning theory; practitioners who train contrastive models at scale and need reasoning behind augmentation, normalization, and data-balancing choices; and readers interested in formal connections between supervised and self-supervised objectives. The paper was reviewed on OpenReview (https://openreview.net/forum?id=cmE97KX2XM), is licensed CC BY 4.0, and includes code in the supplementary material and all proofs in the appendix.

Authors’ abstract

Self-supervised representation learning has achieved impressive empirical success, yet its theoretical understanding remains limited. In this work, we provide a theoretical perspective by formulating self-supervised representation learning as an approximation to supervised representation learning objectives. Based on this formulation, we derive a loss function closely related to popular contrastive losses such as InfoNCE, offering insight into their underlying principles. Our derivation naturally introduces the concepts of prototype representation bias and a balanced contrastive loss, which help explain and improve the behavior of self-supervised learning algorithms. We further show how components of our theoretical framework correspond to established practices in contrastive learning. Finally, we empirically validate the effect of balancing positive and negative pair interactions. All theoretical proofs are provided in the appendix, and our code is included in the supplementary material.