Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Data

AI summary

Overview

Research area: Multimodal representation learning (CLIP-style vision-language models) and mechanistic interpretability of spatial reasoning. Technical level: Intermediate — the paper uses a deliberately minimal synthetic setup, but its core analysis relies on decomposing attention logits into weight/bias and token/positional terms. Scope: A controlled 1D (and small 2D) image-text testbed that isolates when and how left-right relational understanding emerges in Transformer-based CLIP encoders.

What This Paper Is About

Vision-language models are strong at recognizing objects but weak at relational and spatial reasoning — they often behave like "bag-of-words" models that cannot tell "X is on the left of Y" from "Y is on the left of X." This paper builds a deliberately tiny, controllable CLIP-style testbed with 1D images and simple captions to ask whether left-right understanding is actually learned, what kind of training-data diversity drives it, and which internal mechanism produces it.

Key Contributions

1. A controllable testbed for analyzing the emergence of left-right spatial understanding in CLIP-style Transformers, built on a synthetic 1D image-text dataset with one- or two-object scenes.
2. A data-diversity result: CLIP-style contrastive training does learn left-right relations and generalizes to unseen object pairs, and label diversity — not layout diversity — is the primary driver of that generalization in this setup.
3. A mechanistic account: decomposing per-head pre-softmax attention logits shows that interactions between positional and token embeddings induce a horizontal attention gradient that breaks left-right symmetry; ablating this contribution substantially reduces left-right discrimination.
4. Evidence the mechanism is not a 1D artifact: the same horizontal gradient appears in a 2D toy setting, in a setting with both left and right captions, in 3-object images (Appendix N), and in an autoregressive VLM setting (Appendix O).

Main Findings

• Left-right relations are learnable: With a CLIP-style contrastive loss, the model learns to match two-object images to relational captions and generalizes to ordered label pairs never seen in two-object training.
• Label diversity dominates layout diversity: Training on a larger variety of object categories improved all three generalization types (single-object positional, seen-pair configuration, unseen-pair). Variation in how objects were spatially arranged had little effect on performance.
• Weight decay helps: Weight-decay regularization enhanced generalization, which the authors describe as reminiscent of observations in the grokking literature (Power et al., 2022).
• Generalization emerges in stages: Analysis of generalization accuracy and training loss shows single-object positional, two-object coordination, and unseen object pair generalization are achieved consecutively (Appendix D).
• Heads specialize by direction: In the reduced 1-layer vision encoder, head 2 showed the strongest attention bias toward the right-side object, while the remaining heads showed left-biased patterns. Similar specialization appeared in the text encoder, where head 3 was the most strongly left-biased in EOT-to-word attention while heads 0, 1 and 2 showed rightward bias.
• A horizontal attention gradient comes from positional embeddings: The cross term E W_QK P^T (token embedding times positional embedding) produced a clear horizontal gradient in the attention logits, creating a systematic rightward bias. In models that failed to generalize, this gradient was absent (Appendix G).
• Ablating the gradient destroys spatial discrimination: Zeroing the positional-embedding-derived attention logit terms at inference caused left-right discrimination accuracy to drop to near 0.5 — the model could still recognize which objects appear together but could no longer encode their spatial relationship. The EP term had the strongest effect.
• The value pathway also matters: Ablating the position-dependent contribution from the VP term P W_V^T to the value vector also dropped accuracy to 0.5 (Appendix I).
• Rotation links the two encoders: Raw cosine similarity between image and text token embeddings for the same label showed little direct alignment. Fitting a rotation matrix on labels 1–15 and evaluating on labels 16–20 produced markedly stronger alignment, suggesting the two spaces align up to a rotational degree of freedom.
• Both captions need more capacity: When both left and right textual representations were used, a 1-layer text encoder did not generalize. Adding a second layer to the text encoder (while keeping the vision encoder at 1 layer) restored systematic generalization, and the EP term remained the dominant causal driver.
• The mechanism transfers beyond 1D: In a 4×4 image setting flattened to 16 tokens, the model reached 98% unseen-pair generalization accuracy (N_tot = 20, n_2 = 10), and the same positional-embedding horizontal gradient appeared.

Methodology in Plain English

The authors strip spatial reasoning down to its simplest possible form. An "image" is a 1D sequence of length 10, where each object occupies a single pixel, background pixels are 0, and different integers denote different object categories. A learnable [CLS] token is prepended at the first position. Captions are short token sequences: for one object, "[label] is in the image"; for two objects, "[label 1] is on the left of [label 2]" (and optionally the mirrored "[label 2] is on the right of [label 1]").

Both a vision encoder and a text encoder are Transformer-based. The vision encoder uses bidirectional self-attention; the text encoder uses a causal mask. The image representation is taken from the [CLS] output and the text representation from the [EOT] token. A linear projection is applied to the text representation, cosine similarity is computed against the image representation, and the standard CLIP contrastive loss (average of image-to-text and text-to-image cross-entropy) is used.

To test generalization, the authors hold out some object categories entirely from two-object training (labels N_pair+1 through N_tot; N_val = 5) and evaluate three things by image-to-text retrieval accuracy: matching single-object captions at unseen positions, matching seen label pairs at unseen position configurations, and matching entirely unseen ordered label pairs. They then vary label diversity and layout diversity to see which one matters.

For mechanism, they reduce the architecture to a single Transformer block and remove LayerNorm and MLP to isolate attention dynamics. They decompose the pre-softmax attention logits first into weight and bias terms, then expand the dominant X W_QK X^T term into four pieces: token-token (E W_QK E^T), two cross terms (E W_QK P^T and P W_QK E^T), and positional-positional (P W_QK P^T). They then zero specific terms at inference to test causal necessity.

Why This Matters

Impact on research: The paper moves beyond benchmark-style evaluations of VLM spatial failures to a mechanistic account of one specific capability. It identifies a concrete, ablatable pathway (positional-embedding-driven attention gradients) whose removal directly degrades left-right discrimination, and it connects to prior work on data diversity in compositional generalization (Uselis et al., 2025) while arguing that relational and attributive understanding may require different mechanisms.

Real-world applications:

• Autonomous driving, where left/right and in-front-of/behind judgments are safety-critical.
• Robotic manipulation, which depends on correctly interpreting which object is where before acting.
• Image retrieval and accessibility captioning, where "X left of Y" versus "Y left of X" changes the meaning of a scene description.
• Model design and diagnostics, using the identified attention-gradient signature as a check on whether spatial signals are present.

Industry relevance: The authors are affiliated with InfoTech, Toyota Motor Corporation, and the paper explicitly motivates the work by safety-critical automotive and robotics needs. The finding that label diversity drives generalization — and that positional embeddings are the key architectural ingredient — offers design guidance for teams training or auditing spatial encoders.

Future Directions

• Whether similar attention-gradient mechanisms emerge in large-scale VLMs trained on natural 2D images, where spatial structure and compositionality are richer (the authors show the mechanism in a 2D toy setting but call bridging to natural images an important goal).
• A more thorough comparison of spatial-relation competence across training paradigms, since only a preliminary autoregressive VLM check is reported (Appendix O).
• How multiple attention heads interact (Appendix P) and how relational signals propagate across layers (Appendix Q) remain open.
• Extending the approximate LayerNorm decomposition used to analyze non-linear models, since fully characterizing the role of non-linear components remains a significant challenge (Appendix R).

Target Audience

Researchers working on multimodal representation learning, mechanistic interpretability, and compositional or spatial reasoning in vision-language models. It will also be useful to practitioners who need to audit or design encoders for safety-critical spatial tasks, and to readers comfortable with attention math who want a minimal, fully controlled case study rather than a benchmark leaderboard.

Authors’ abstract

Spatial understanding remains a key challenge in vision-language models. Yet it is still unclear whether such understanding is truly acquired, and if so, through what mechanisms. We present a controllable 1D image-text testbed to probe how left-right relational understanding emerges in Transformer-based vision and text encoders trained with a CLIP-style contrastive objective. We train lightweight Transformer-based vision and text encoders end-to-end on paired descriptions of one- and two-object scenes and evaluate generalization to unseen object pairs while systematically varying label and layout diversity. We find that contrastive training learns left-right relations and that label diversity, more than layout diversity, is the primary driver of generalization in this setting. To gain the mechanistic understanding, we perform an attention decomposition and show that interactions between positional and token embeddings induce a horizontal attention gradient that breaks left-right symmetry in the encoders; ablating this contribution substantially reduces left-right discrimination. Our results provide a mechanistic insight of when and how CLIP-style models acquire relational competence.