Proving and Improving the Platonic Representation Hypothesis with JEPAs

AI summary

Overview

Research area: Self-supervised representation learning and multimodal alignment — specifically the Platonic Representation Hypothesis (PRH), the conjecture that models trained on different modalities converge to a shared representation of the world as they scale.

Technical level: Advanced. The core argument relies on spectral graph theory (normalized adjacency matrices, eigengaps, spectral embeddings), orthogonal Procrustes alignment, and careful measure-theoretic framing of a "world," its "renderings," and "nuisance" variables. The empirical section is more accessible but assumes familiarity with JEPA-style training, ViT/BERT backbones, and cross-modal retrieval metrics.

Scope: The paper converts the PRH from an empirical observation into a theorem for Joint-Embedding Predictive Architectures (JEPAs), identifies which training objectives produce "good views" versus "bad views," and validates the resulting predictions from 145k captions up to the texts of 940M web images.

What This Paper Is About

The PRH, proposed by Huh et al., observes that vision and language models become more similar in their internal representations as they grow, and conjectures that they are converging on a shared statistical model of reality. Until now this was an observation without a proof, a precise condition for when it holds, or a recipe for making alignment better. Balestriero and LeCun ask a sharper question — when do models trained separately on different modalities provably learn the same representation? — and answer it by showing that the answer depends almost entirely on what a model's positive pairs are.

Key Contributions

1. The Platonic Representation Theorem. For an idealized JEPA setting (population optima, injective renderings, minimizers unique up to rotation), any number of modalities trained separately with the same positive-pair distribution over world states learn the same embedding up to one rotation each. When positives differ, the misalignment is bounded by how much the kernels drawn by those positives differ, divided by the spectral gap.

2. A good-view / bad-view taxonomy. "Good views" are two observations of the same scene (two photographs, two captions); "bad views" are corruptions of a single observation (two crops, two maskings). Bad views make an encoder spend embedding dimensions on its modality's nuisance — word choice, viewpoint, lighting — which no rotation can align across modalities. Good views spend none.

3. Next-token prediction is a bad-view JEPA. The softmax log-partition plays the role of the anti-collapse term, and the positives are a prefix and its continuation within a single document. This is offered as the explanation for why language models align with vision only partially and mostly through scale.

4. Empirical validation plus a scaling law of alignment. Across identical texts and encoders, good views align best in nearly every comparison run; alignment follows a scaling law in the fraction of good views and the samples seen, which predicts 85 new runs before they are trained, as accurately as rerunning them with another seed.

5. Beating far larger language models. A 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model evaluated, up to 72B parameters, and better than CLIP's text encoder.

Main Findings

• Good views lead on every Flickr metric. Against a separately trained image JEPA on 1,000 Flickr test pairs, the good-view text JEPA (caption views) reaches mutual 10-NN .062 ± .003, R@1 .044 ± .002 and R@5 .133 ± .004; the bad-view JEPA (masked spans) reaches .047, .027 and .095; next-token prediction reaches .042, .021 and .077. Chance is .010, .001 and .005.

• The gain is the objective, not merely the information. Next-token prediction trained on pairs of captions of the same image barely moves (mutual 10-NN .042 to .043, R@1 .021 to .023), whereas the same positives as JEPA views help substantially.

• Nuisance is measurable. Using the five captions per Flickr image as five nuisances, at k = 32 the scene explains a fraction ρ of .804 of the good-view JEPA's embedding, against .655 for the bad-view JEPA and .608 for next-token prediction (each within ±.005 across seeds). The corresponding theoretical lower bounds on best-rotation error are 12.6, 22.1 and 25.1, and the measured errors (33.5, 39.4 and 40.3) respect and follow that order.

• Alignment lives in the leading spectral directions. After the best rotation, image–text correlation is 0.70 on the first principal direction of the text space and decays to about 0.1 by the 60th, while it is near zero everywhere at initialization.

• The mismatch bound holds empirically. Across 300 randomly generated finite worlds (8–39 states, embedding dimension 1 to 5, kernels mixed with an unrelated kernel at weight 10⁻⁴ to 1), the post-rotation error lies below the theoretical bound in every world, at 18% of it at most.

• Sample efficiency. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment; concretely, with 21.6M samples a good-view JEPA matches next-token prediction at 900M samples (.109 and .111, described in the figure as 14 times fewer text tokens).

• Scale is a poor substitute. On the texts of 50M web images a 33M-parameter text JEPA reaches .125 on the PRH paper's benchmark, against .034 on Flickr captions, level with Qwen2.5-3B and -7B. Trained on the texts of 940M web images, a 109M-parameter text JEPA reaches .184, above Qwen2.5-72B (.154) and above CLIP's text encoder (.124) on two benchmarks and two metrics; it also leads with MAE, I-JEPA and CLIP's image encoders, e.g. .230 against .210 with CLIP ViT-H. The authors note this 109M encoder read 116B text tokens and beats models 660 times larger trained on over 100 times more data.

• Captioning from one rotation. A single linear map with orthonormal rows, the only part fitted on image–caption pairs, joins frozen DINOv2 and DINOv3 image encoders to a language model fine-tuned on text alone, reaching CIDEr .838 on COCO.

• Pair-free alignment. On the benchmark of Schnaus et al., which aligns vision and language encoders from unpaired data, the 109M text JEPA beats every zero-shot accuracy reported across 21 vision–language pairs and five settings; it beats Qwen3-8B rerun under identical conditions by 17 points of zero-shot ImageNet-100 accuracy, e.g. 64% against at most 56% top-5 on ImageNet-100.

• Weaker positives need more data. On a synthetic world of five classes of six states, the reach that weak bad views need falls as samples per state grow, along Δ ∝ log N / N; reach large enough to cross scenes caps alignment at any sample size, while good views align exactly from one sample per state.

Methodology in Plain English

The authors first build an abstract model: a "world" is a finite set of states, and each modality observes it through an injective rendering — pixels for a photograph, tokens for a caption. A JEPA learns by minimizing a prediction term over positive pairs plus an anti-collapse term that keeps embeddings from collapsing. Because an injective rendering only changes coordinates, each modality's training problem is the same problem written in different coordinates, so the solutions must coincide up to a rotation.

They then make this precise for JEPAs with second-order anti-collapse (whitened embeddings, squared-error prediction): such a JEPA learns the top eigenvectors of the graph its positive pairs draw on the world, unique up to rotation when the spectrum has a gap. They extend this to the realistic case where positives differ, deriving a bound in terms of kernel mismatch and eigengap, and then introduce "nuisance" variables to formalize why corruptions of a single observation capture modality-specific information that cannot be rotated away.

Next they show algebraically that next-token prediction has the same shape — context encoder, token table, inner-product prediction, log-partition anti-collapse — but is identified only up to an invertible linear map rather than a rotation.

For experiments they hold everything fixed except the positives: the same LeJEPA objective (λ = 0.02), the same ViT-S/16 image encoder (2 global 224² and 6 local 96² crops), the same 12-layer, 384-width BERT text encoder trained from scratch, on Flickr30k with the Karpathy split (29k training images, five captions each), 200 epochs, batch 256, AdamW at 5·10⁻⁴ and weight decay 0.05, over three seeds. Three text variants differ only in their positives: masked spans, caption views, and causal next-token prediction. Alignment is scored without fitting anything (mutual 10-NN) and with a single rotation fitted on validation pairs (retrieval). Larger-scale runs extend the same recipe to web-image text corpora.

Why This Matters

The paper reframes the PRH from a hopeful observation into an engineering principle: what determines cross-modal alignment is the structure of positive pairs, not sheer scale. That gives practitioners a concrete lever — define views as two observations of one scene — instead of prescribing more parameters and more tokens.

Real-world applications:

• Cheap cross-modal retrieval and search: a small text encoder that aligns with a frozen image encoder can serve as a retrieval index without training on paired data.
• Image captioning without paired supervision: the single fitted rotation (CIDEr .838 on COCO) means captioning systems can be assembled from separately trained, frozen components.
• Pair-free model stitching: aligning independently trained encoders from unpaired data (as on the Schnaus et al. benchmark) supports modular system building, including for rare modalities where paired data is scarce.
• Data-efficient multimodal pretraining: 42 times fewer samples for the same alignment translates directly into compute and data savings.

Industry relevance: the result challenges the assumption that scaling language models is the efficient route to grounded representations. A 109M-parameter encoder beating models up to 72B on alignment, and beating an 8B model on zero-shot ImageNet-100 (64% vs at most 56% top-5), is a strong argument for revisiting objective design before scaling budgets.

Future Directions

• Good views at scale for images. The authors explicitly leave this open: good image views need several observations of each scene (e.g., video frames), and an image JEPA trained on MSR-VTT's 7k clips is far weaker than DINOv2. Whether good image views can be scaled is unresolved.
• World-model JEPAs. The analysis assumes an identity predictor. JEPAs used as world models predict one embedding from another, often conditioned on an action, and the paper states its analysis does not cover this case.
• Uniqueness under full-distribution anti-collapse. Proposition 1 covers second-order whitening with a spectral gap, but LeJEPA's SIGReg target — an isotropic Gaussian — is only argued to have the same second moments; uniqueness under its full-distribution constraint is left as an assumption.
• Better positive-pair design across modalities. The theory says nuisance preservation is the failure mode; designing augmentations or data pipelines that provably avoid it — and testing whether the kernel-mismatch bound can be tightened — is the natural next step.

Target Audience

Researchers in self-supervised learning and multimodal representation learning who want a theoretical handle on cross-modal alignment; practitioners building retrieval, captioning, or embedding systems who need data-efficient alternatives to scaling language models; and theoretically inclined readers interested in spectral graph arguments applied to representation learning. Readers without linear algebra and spectral graph background will find Sections 2.1–2.4 demanding, though the experimental sections and figures are readable on their own.

Authors’ abstract

Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.