Your Self-Supervised Projection Head Captures Object Co-Occurrence Statistics

AI summary

Overview

Research area: Self-supervised visual representation learning (SSL) and its analysis — specifically, what the projection head of standard SSL pipelines actually encodes, and how object co-occurrence statistics shape representation learning.

Technical level: Intermediate. The paper is empirical and analytical rather than introducing a new architecture; readers benefit from familiarity with contrastive/distillation SSL pipelines (backbone, projector, augmentation views) and standard evaluation tools such as Representational Similarity Analysis (RSA) and linear probing.

Scope (one sentence): The paper tests, across many pretrained SSL models and three purpose-built segmented-object datasets, whether common SSL methods implicitly encode natural object co-occurrence statistics, and shows that this encoding is concentrated in the projection head and benefits downstream categorization.

What This Paper Is About

Standard SSL vision methods train a backbone plus a multi-layer projection head to produce embeddings that are invariant to crops, colour changes, and other augmentations. The usual explanation is that augmentations preserve label information, but the authors hypothesize a second, hidden mechanism: random crops of natural scenes often show different objects that happen to co-occur (e.g., food and a fork), so SSL may implicitly learn which categories appear together.

The goal is to determine whether such co-occurrence statistics are actually encoded, which layers encode them, whether they align with human similarity judgments, and whether controlling them during pre-training causally changes downstream categorization performance. To isolate object-level representations from scene context, the authors build datasets in which each image contains a single segmented object on a clean background.

Key Contributions

1. An analysis framework linking SSL representations to object co-occurrence. The authors construct three paired co-occurrence/segmented-object datasets (COCO-THINGS: 736 images, 53 categories; ADE20K-THINGS: 918 images, 77 categories; LVIS-LVIS: 118,068 images, 1,002 categories) and introduce two metrics: RSA Kendall's tau against empirical co-occurrence matrices, and a "context classification accuracy" based on context assignments of categories.

2. Evidence that the projection head, not the backbone, is where co-occurrence statistics live. Across MoCoV3 (ViT-B/16 and ResNet50), BYOL (ResNet50), VICReg (ResNet50) and DINO (ViT-B/16), an internal projection layer consistently beats the final backbone layer at encoding co-occurrences, and removing the non-linear projection head degrades co-occurrence encoding.

3. A link to human semantics. Projection-head layers align better than backbones with human inter-category similarity judgments from the THINGS odd-one-out data (≈4M judgments), and layers that encode co-occurrences better also match humans better.

4. Causal evidence via controlled co-occurrence. By artificially defining co-occurrence contexts over ImageNet-1k classes during pre-training, the authors show that co-occurrence structure speeds up category learning, with a peak gain of up to +5.52% ImageNet-1k accuracy over the no-context baseline at p_c = 0.3.

Main Findings

• Crops often mix categories. Using a pretrained ResNet50 as a proxy on 1M sampled crop-augmented image pairs from the ImageNet-1k train split, augmented views are semantically inconsistent approximately 15% of the time under standard hyperparameters (e.g., minimum crop size 0.08). A lower bound is obtained by counting when a crop's predicted label falls outside the other crop's top-5 predictions.

• The backbone encodes categories, the projection head encodes context. The final backbone layer discriminates object categories best, while an internal projection layer best captures co-occurrences. For example, DINO-ViT-B on COCO-THINGS reaches RSA Kendall's tau of 9.0 (backbone), 21.5 (mP.0), 33.4 (mP.1) and 13.9 (mP.2), and context accuracy of 43.5 / 45.7 / 50.3 / 40.7 respectively.

• Early-to-intermediate projection layers are best; the final linear layer is weak. Across datasets and methods, the best co-occurrence encoding generally appears in early or intermediate projection layers, while the last linear layer tends to perform poorly. Two exceptions are noted and left unexplained: BYOL-R50 on LVIS-LVIS (RSA Kendall's tau) and MoCoV3-ViT-B on COCO-THINGS (context classification).

• DINO-ViT-B is the strongest co-occurrence encoder. It is the best model on these co-occurrence metrics with a single exception (Kendall's tau on LVIS-LVIS), and it is also the best ImageNet classifier among the models studied, hinting at a link between co-occurrence encoding and representation quality.

• The non-linear projection head is necessary for this encoding. Replacing it with a linear layer reduces co-occurrence encoding across datasets, methods and metrics, with a single exception (ADE20k-THINGS context accuracy for SimCLR).

• Deeper projection heads shift the optimal layer deeper. With controlled numbers of projection layers (1 to 4), the best layer for co-occurrence encoding moves deeper as the head deepens, with one exception (BYOL-4 on COCO-THINGS RSA Kendall's tau). With a single linear layer, the projection head does not necessarily beat the backbone.

• Co-occurrence statistics correlate with human similarity judgments. Object co-occurrence and human similarity RSMs correlate positively (Kendall's tau 0.266, Spearman 0.374). At least one projection-head layer matches human similarity better than the backbone in every model tested (e.g., DINO-ViT-B: backbone 30.5, P.0 28.5, P.1 31.1, P.2 28.0).

• Co-occurrence encoding predicts human-alignment and accuracy. Across 101 representation layers (14 publicly pretrained SSL models plus 12 SSL models the authors pretrained on ImageNet-1k for 100 epochs), layer-wise co-occurrence encoding correlates with human-similarity alignment: r = 0.52 (p = 1.72×10⁻⁸) for COCO-THINGS, r = 0.47 (p = 6.14×10⁻⁷) for ADE20k-THINGS, and r = 0.39 (p = 6.19×10⁻⁵) for LVIS-LVIS. Co-occurrence encoding also correlates with ImageNet-1k accuracy: r = 0.78 (p = 7.95×10⁻⁶) for COCO-THINGS, r = 0.48 (p = 1.81×10⁻²) for ADE20k-THINGS, and r = 0.79 (p = 4.84×10⁻⁶) for LVIS-LVIS.

• Controlled co-occurrences causally help, up to a point. Sampling the second image of a positive pair with probability p_c from a different class in the same hand-defined context yields a "sweet spot" at p_c = 0.3, with up to +5.52% ImageNet-1k accuracy over the p_c = 0 baseline; accuracy consistently drops for p_c > 0.3, indicating that same-image augmentation alignment remains essential. Increasing the number of contexts n_c (evaluated with p_c = 0.3, at n_c = 300 in the p_c sweep) also improves representations over the no-context baseline. DINO collapsed at p = 0.7 and p = 0.9, so raw accuracies are shown for those points.

• Semantic hierarchy across depth. In-context one-shot category classification accuracy decreases with projection depth for every dataset and method (e.g., DINO-ViT-B on LVIS-LVIS: backbone 85.4, P.0 84.1, P.1 80.7, P.2 79.6, P.3 74.2), supporting a transition from categorical to contextual organisation in deeper layers.

• Hierarchy within the projection head is not cleanly resolved. Testing five levels of contextual organisation (3, 10, 30, 100, 300 LVIS clusters) shows the projection head consistently beats the backbone, but there is no consistent trend as to which layer is best per level — suggesting extra mechanisms would be needed to enforce explicit hierarchical encoding.

Methodology in Plain English

Isolating objects from scenes. Because ImageNet-1k lacks segmentation annotations, the authors start from THINGS (26,107 naturalistic images spanning 1,854 object concepts, each showing one clearly visible object) and cut out the objects using Lang-SAM, a language-conditioned pipeline combining Grounding-DINO for concept-specific bounding boxes and SAM-2 for segmentation; poorly segmented images were manually filtered out. They also crop segmented objects larger than 32×32 pixels from the LVIS validation split.

Measuring what "should" be encoded. Natural co-occurrence statistics are estimated by counting how often pairs of categories appear together in annotated scene datasets: the COCO training and validation splits, the ADE-20K training split, and the LVIS training split. Only classes overlapping with the segmented datasets are retained, producing the three dataset pairs.

Comparing representations to statistics. For each layer, they average representations over all segmented images of the same category, compute a pairwise cosine representational similarity matrix between category representations, and correlate its upper triangle with the co-occurrence matrix using Kendall's tau. As a complementary metric, they assign categories to contexts — 10 hand-defined spatial contexts (e.g., "Kitchen", "Savanna", "Bedroom") for COCO-THINGS, and 10 spectral-clustering clusters run three times on the log co-occurrence matrix for the other two datasets — and measure how often a category representation is closer to a same-context category than to categories from other contexts (1,000 tuples for the THINGS-based sets, 10,000 for LVIS-LVIS).

Controlling co-occurrences during training. To move from correlation to causation, ImageNet-1k classes are randomly partitioned into n_c contexts. During pre-training, the second image of each positive pair is drawn with probability p_c from a different class within the same context. Varying p_c and n_c isolates the effect of co-occurrence structure alone. SimCLR and DINO were pretrained for 100 epochs under these conditions.

Models and compute. Main experiments use MoCoV3 (ViT-B/16, ResNet50), BYOL (ResNet50), VICReg (ResNet50) and DINO (ViT-B/16), all pretrained on ImageNet-1k; additional pretrained models (listed in Appendix Table 6, including ViT-S/8, ViT-B/8, ViT-S/16, ResNet50x2 and ResNet200x2 variants) are used for correlation analyses. The authors' own pretraining used 4 NVIDIA A100 GPUs for BYOL and 8 AMD Instinct MI210 GPUs for SimCLR/DINO, taking 20 to 30 hours for BYOL/SimCLR and approximately 72 hours for DINO; metric computation takes under 1 hour on four NVIDIA A100 GPUs.

Why This Matters

Impact on research. The paper challenges the dominant framing that SSL augmentations work mainly by preserving label information. It reframes the projection head — usually discarded after pre-training because it is "too specialised" for augmentation invariance and suffers dimensional collapse — as a store of semantic context knowledge. This connects SSL to the co-occurrence-modelling principle behind word embeddings and large language models, and suggests that co-occurrence structure in the pre-training data is a first-class design variable rather than an incidental artefact.

Potential real-world applications (not evaluated in this paper):

• Retrieval and recommendation: embeddings that encode "what tends to appear with what" support scene-aware image search and context-conditioned product or content recommendation.
• Robotics and embodied agents: co-occurrence-aware features provide priors about which objects belong in which environments, useful for scene understanding and affordance reasoning.
• Pre-training data and pipeline design: deliberately curating or re-sampling image pairs by co-occurrence context could yield better features for the same compute budget.
• Model auditing and interpretability: co-occurrence probing offers a concrete diagnostic for detecting dataset-induced biases (e.g., a category always learned through its typical surroundings).

Industry relevance. Most industrial vision systems are built on SSL or SSL-like pretraining, and the projection head is typically trained and thrown away. The paper's argument that these layers carry useful semantic structure, and that co-occurrence-aware pair sampling improves accuracy at a controlled budget, is directly actionable for teams designing pretraining recipes. The reported +5.52% ImageNet-1k gain from a simple change to positive-pair sampling is a low-cost intervention.

Future Directions

• Explain the exceptions and the role of architecture. The authors note that factors such as neural architecture and SSL method affect co-occurrence encoding, and that two specific results remain unexplained (BYOL-R50 on LVIS-LVIS for RSA, MoCoV3-ViT-B on COCO-THINGS for context accuracy).

• Test modern foundation models. DINOv2 and similar models do not release projection-head weights, so the authors explicitly call for future work to confirm these findings where those weights are available.

• Design pretext tasks that explicitly exploit co-occurrence structure. The controlled experiments suggest deliberately structured pair sampling helps; extending this into principled pretext-task design is posed as an open avenue.

• Enforce a real hierarchy inside the projection head. Since the paper finds no consistent mapping between projection depth and level of contextual organisation (3 to 300 clusters), additional mechanisms appear necessary to obtain explicit hierarchical encoding.

Target Audience

SSL researchers and graduate students who want to understand why standard pipelines work rather than only how well they score; practitioners designing pretraining recipes who are deciding whether to keep, discard, or restructure projection heads; computational neuroscientists and cognitive scientists interested in the parallels between SSL, human object co-occurrence learning, and semantic similarity judgments; and interpretability researchers looking for concrete, dataset-grounded probes of what self-supervised features actually represent.

Authors’ abstract

Self-Supervised Learning (SSL) has achieved impressive success in learning semantic visual representations, yet the underlying principles driving this success remain underexplored. In this work, we hypothesize that common SSL pretext tasks implicitly model object co-occurrence statistics, a fundamental cue for visual learning. To test this, we curate three datasets of segmented objects from existing vision benchmarks using a state-of-the-art segmentation model. Through experiments across many SSL models, we reveal a hierarchical encoding of semantic information: while the visual backbone captures object categories, the projection head specializes in encoding co-occurrences between object categories. As a result, the projection head outperforms the backbone in aligning with human judgments of inter-category similarity. Furthermore, by controlling co-occurrence patterns during pre-training, we demonstrate that encoding object co-occurrences can significantly accelerate the emergence of category-level representations. Our findings uncover a previously hidden learning principle in SSL and suggest a path toward designing more effective pretext tasks by explicitly leveraging object co-occurrence structure.