Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers?

AI summary

Overview

Research area: Computer vision interpretability, bridging cognitive science theories of "object binding" with the internal representations of pretrained Vision Transformers (ViTs).

Technical level: Advanced. The paper leans on representational geometry, quadratic probing, and ablation studies, though its central claim is stated in accessible terms.

One-sentence scope: The paper tests whether large pretrained ViTs implicitly encode a pairwise "IsSameObject" signal (whether two image patches belong to the same object), how that signal is structured and used, and whether removing it harms the model.

What This Paper Is About

Humans effortlessly group the scattered visual features of a scene into coherent objects, an ability cognitive scientists call object binding. Prior work assumed ViTs could not do this without being explicitly built for it (for example, via Slot Attention), and critics have argued that pure connectionist architectures lack the machinery for symbolic, object-level grouping. This paper asks whether object binding instead emerges naturally from large-scale pretraining, and answers by probing whether ViT patch embeddings encode whether any two patches belong to the same object.

Key Contributions

1. Demonstrates that object binding naturally emerges in large pretrained ViTs, challenging the cognitive-science assumption that binding is not possible given their architecture.
2. Shows that ViTs encode a low-dimensional signature of IsSameObject (whether two patches belong to the same object) on top of their feature representations.
3. Shows that this ability is acquired through specific pretraining objectives rather than being a universal architectural artifact, based on comparisons across DINO, CLIP, ImageNet-supervised, and MAE models.
4. Provides functional evidence that the binding signal guides self-attention and serves the pretraining objective: ablating it degrades downstream segmentation and increases the DINO self-distillation loss.

Main Findings

• Binding is decodable and accurate. A quadratic similarity probe decodes IsSameObject from patch embeddings across ViT layers, reaching over 90% accuracy; the headline figure is 90.20% for DINOv2-Large at layer 18.
• The representation is quadratic, not linear. From middle layers onward, quadratic probes consistently outperform linear probes, diagonal quadratic probes, and pointwise object-class/object-identity probes. This is consistent with the quadratic form used by self-attention itself.
• The signal is distributed, not localized. Full quadratic probes beat diagonal quadratic probes, meaning the information is spread across dimensions rather than isolated in a few specialized channels.
• Binding is an acquired ability, not an architectural given. Table 1 reports peak probe accuracy and improvement over the 72.6% majority baseline: DINOv2-Small 86.7% (+14.1 pp, peak layer 1.00), DINOv2-Base 87.5% (+14.9 pp, 0.82), DINOv2-Large 90.2% (+17.6 pp, 0.78), DINOv2-Giant 88.8% (+16.2 pp, 0.77), ImageNet-supervised ViT-L 84.2% (+11.6 pp, 0.39), CLIP ViT-L 82.9% (+10.3 pp, 0.65), and MAE ViT-L 76.3% (+3.7 pp, 0.13). MAE is markedly weaker than every other model tested.
• Layer-wise dynamics. DINOv2 models peak in later layers (normalized peak indices 0.77–1.00), the supervised ViT peaks earlier (0.39), and MAE's peak is very early (0.13).
• Binding emerges mid-network and then shifts to class-level grouping. Using a test image with three red cars (two identical, one a different brand) and a red boat, early layers attend to similar surface features, mid-layers discern the local object each patch belongs to, and higher layers progressively treat all red cars as the same. A distance-based baseline reaches 77.16% accuracy.
• The binding subspace is low-dimensional. Applying PCA to residuals between corresponding patches of four identical red-car images yields three linearly separable clusters in the first three principal components.
• The signal actively guides attention. Attention weights correlate with IsSameObject scores in middle layers, with Pearson r = 0.163 and r = 0.201 in two reported cases. The authors note the correlation is modest because attention serves many roles beyond binding.
• Ablation confirms functional importance. Under uninformed ablation (randomly shuffling binding vectors across patches), segmentation mIoU falls from 44.14 (ratio 0) to 41.03 (0.5) to 39.20 (1), and instance mIoU from 35.14 to 31.39 to 28.19. Under informed injection of ground-truth instance information, segmentation mIoU reaches 44.91 (α = 0.5) and 43.59 (α = 0), with instance mIoU of 36.37 and 37.02 respectively. Uninformed ablation also raises the DINO self-distillation loss from 0.6182 (ratio 0) to 0.6591 (0.5) to 0.6749 (1), meaning ablation works against the pretraining objective.

Methodology in Plain English

The authors treat each image as a grid of patches and ask, for every pair of patches, whether they come from the same object. They define this predicate, IsSameObject, and test whether it can be read out of the model's internal activations using lightweight classifiers called probes.

Four probe designs are compared. A linear probe sums weighted features. A diagonal quadratic probe uses only per-dimension products, testing whether binding lives in a few specialized channels. A full quadratic probe (with factorization enforced for symmetry, since IsSameObject is symmetric in its inputs) captures pairwise interactions across all dimensions, mirroring the query–key dot product in attention. Pointwise probes instead map each patch to an object class or object identity distribution and compare them, testing whether the model just recognizes coarse categories rather than binding actual instances.

Probes are trained on the ADE20K dataset with images resized and cropped to 512×512 and padded to 518×518, using 14×14 patches for a total of 1,369 patches per image. To control cost, 64 patches per image are sampled each epoch and supervision is applied to the resulting pairs. Training uses the Adam optimizer with a learning rate of 0.001, a step scheduler with step size 8 epochs and gamma decay 0.2, 16 epochs, and batch sizes of 256 or 128, all in float32 on an NVIDIA RTX 4090. To compare models fairly, inputs to CLIP, MAE, and the supervised ViT are resized and cropped so their patch grids align with DINOv2's, which gives every model the same 72.6% majority baseline.

Beyond probing accuracy, the authors visualize IsSameObject predictions layer by layer on a deliberately hard test image, use PCA on residuals between identical objects to measure dimensionality, correlate IsSameObject scores with attention weights, and run two ablations on layer 18: randomly shuffling binding vectors (uninformed) and injecting ground-truth instance masks (informed). They then retrain segmentation heads on a subset of ADE20K and also track the DINO teacher–student distillation loss.

Why This Matters

The paper reframes a long-standing cognitive-science claim. Critics argued ViTs lack the recurrent, dynamic, or symbolic machinery needed for binding; this work shows binding emerges anyway, from the learning objective rather than from architecture. It also suggests that fixing binding failures in vision models may not require bolting on explicit object-centric modules such as Slot Attention, which introduce scaling and training challenges, but could instead come from strengthening intrinsic mechanisms through training objectives or minimal architectural changes.

Real-world applications:

• Instance segmentation. Producing per-object masks for overlapping and disconnected instances, which the paper illustrates with overlapping deer and with patches of the same deer separated in the image.
• Visual reasoning. Locating and counting objects with particular feature combinations, where patches triggered by one feature must be bound to the rest of their object.
• Vision-language and generative models. Diffusion models suffer attribute leakage (for example, both a dog and a cat ending up with sunglasses and a sun-hat), and vision-language models struggle to differentiate multiple objects with feature conjunctions; better binding would reduce these failures.
• Compositional generation and manipulation. Object-centric representations enable composing and editing scenes, a benefit previously available only through explicit object-centric modules.

Industry relevance: Pretrained ViT backbones are widely deployed, so understanding that a usable object-level signal already exists inside them means practitioners may be able to extract object structure with lightweight probes rather than training separate object-centric architectures. The finding that DINO-style self-distillation and CLIP-style image-text alignment induce binding while MAE-style reconstruction does not gives concrete guidance for choosing pretraining objectives.

Future Directions

1. Understanding why objectives produce binding. The authors explicitly call for further analysis of how different pretraining objectives induce object binding, since the mechanism behind the DINO/CLIP/supervised versus MAE gap is not established.
2. Establishing causality. The paper states it does not establish a causal relationship between object binding and downstream task performance, and says further analysis is needed.
3. Broadening evaluation beyond segmentation. Downstream evaluations focus only on segmentation; whether emergent binding benefits other vision tasks such as visual reasoning is left open.
4. Strengthening binding without explicit modules, and modeling object interactions. The conclusion suggests tailoring training objectives or making minimal architectural modifications instead of relying on Slot Attention, and studying how bound object representations interact with one another, potentially through low-dimensional "object files."

An additional limitation the authors note is that the analysis assumes the trained probe cleanly splits each embedding into "feature" and "binding" components, a simplification that would benefit from more empirical exploration. The paper studies binding at the patch level; more general forms of binding are not explored.

Target Audience

This paper is most useful for researchers in computer vision interpretability and representation learning, especially those working on self-supervised pretraining objectives and ViT backbones. It also speaks directly to cognitive and computational neuroscientists interested in whether the binding problem can be solved by connectionist systems, and to practitioners who build object-centric or segmentation pipelines and want to know whether object structure is already latent in off-the-shelf models. Readers without a background in probing methods or quadratic forms will find the conceptual claims accessible, but the methodology sections require some familiarity with transformer internals. Code is available at https://github.com/liyihao0302/vit-object-binding.

Authors’ abstract

Object binding, the brain's ability to bind the many features that collectively represent an object into a coherent whole, is central to human cognition. It groups low-level perceptual features into high-level object representations, stores those objects efficiently and compositionally in memory, and supports human reasoning about individual object instances. While prior work often imposes object-centric attention (e.g., Slot Attention) explicitly to probe these benefits, it remains unclear whether this ability naturally emerges in pre-trained Vision Transformers (ViTs). Intuitively, they could: recognizing which patches belong to the same object should be useful for downstream prediction and thus guide attention. Motivated by the quadratic nature of self-attention, we hypothesize that ViTs represent whether two patches belong to the same object, a property we term IsSameObject. We decode IsSameObject from patch embeddings across ViT layers using a quadratic similarity probe, which reaches over 90% accuracy. Crucially, this object-binding capability emerges reliably in DINO, CLIP, and ImageNet-supervised ViTs, but is markedly weaker in MAE, suggesting that binding is not a trivial architectural artifact, but an ability acquired through specific pretraining objectives. We further discover that IsSameObject is encoded in a low-dimensional subspace on top of object features, and that this signal actively guides attention. Ablating IsSameObject from model activations degrades downstream performance and works against the learning objective, implying that emergent object binding naturally serves the pretraining objective. Our findings challenge the view that ViTs lack object binding and highlight how symbolic knowledge of "which parts belong together" emerges naturally in a connectionist system.