Predictive Closure: A Set-Level Characterization of JEPA's Feature Selection

AI summary

Overview

Research area: Machine learning — self-supervised representation learning, specifically joint-embedding predictive architectures (JEPA) and the question of which features their learned latents encode.

Technical level: Intermediate. The argument rests on an information-theoretic reformulation (mutual information, entropy constraints, a decomposition into "aleatoric uncertainty" and "predictive leakage"), but the experiments are small and described in plain terms, so the empirical half is accessible to a general ML reader.

Scope: The paper proposes and tests a set-level theory — "predictive closure" — of how JEPA selects which observation features to represent, arguing that selection depends on the joint predictive relationships within a candidate feature set rather than on each feature's own predictability.

What This Paper Is About

Earlier work argued that JEPA encodes features that are easy to predict and discards unpredictable ones, raising the worry that a feature like a traffic light — whose next color is essentially random — would never make it into the representation. This paper argues that such a feature can still be represented, because its current value may help predict other features that JEPA does encode (the light's color helps predict whether a cyclist will ride or stop). The goal is to characterize feature selection as a choice between competing feature sets, and to test that characterization in controlled experiments.

Key Contributions

1. A set-level characterization ("predictive closure"). The authors reformulate the JEPA training objective and show that an ideally trained encoder represents a feasible feature set — one whose joint entropy matches the latent's entropy constraint — that maximizes the predictive information between the set's current and next values. Selection therefore depends on joint properties of sets, not on a per-feature ranking.

2. A decomposition into two interpretable terms. The objective is rewritten as minimizing the sum of aleatoric uncertainty (how uncertain the set's next values remain even given the full current observation) and predictive leakage (how much the current observation predicts the set's next values beyond what the set's own current values predict). This yields concrete, testable predictions.

3. Reconciliation with prior work and new predictions. The paper shows the view collapses to earlier per-feature characterizations (slow features, predictable features, subspace selection) when features evolve independently, and explains specific prior empirical findings from Sobal et al. (2022), Enigma Team (2026), Karmann et al. (2026) and Pendharkar (2026). When features are predictively related, it makes new predictions: cross-subsidization, set switching, feasibility dependence, and indirect steering.

4. Controlled experiments confirming the predictions, including indirect steering. Using synthetic CIFAR-10 image collages with known and tunable predictive relationships, the authors test three selection predictions and two steering predictions, showing that targeting one feature (via an auxiliary reconstruction loss or via predictor side information) can strengthen the representation of a different, untargeted feature.

Main Findings

• Unpredictable features can be retained if they predict others (cross-subsidization). In a collage with a two-valued traffic light and two images, where the light's current value determines which image is resampled with probability q and the next light is an independent uniform draw, decoding accuracy for the light increases as q increases — even though the light's own future stays impossible to predict. The authors note this transition involves a nonlinear interaction that linear-predictor frameworks such as Littwin et al. (2024) cannot capture.

• Changing one set's internal relationships can displace a disjoint set (set switching). In a 2×2 collage with image pairs A/B and C/D, raising only the probability p_D that C is copied to D strengthens decoding of C/D and weakens decoding of A/B, even though A/B's own predictive relationships are unchanged and neither pair helps predict the other.

• A set can be penalized simply for exceeding latent capacity (feasibility dependence). Growing the competing cycle to N ∈ {2, 4, 8, 16} images while keeping A/B fixed and average within-cycle copying probabilities unchanged reduces mean decoding accuracy for the cycle's features at every increase in N, in all three optimization seeds. At N = 16, the fixed A/B pair is better represented in all three seeds — an endpoint reversal that also holds for the worst and best positions in each cycle. In one of three sensitivity-analysis configurations, the increase in A/B representation falls within the error bars.

• An auxiliary reconstruction objective bleeds into predictive partners. Adding a loss to reconstruct a coarse version of the current C image (only C is a target; the decoder is discarded before probing) makes D — never reconstructed — decoded almost as accurately as C at the largest weight, while A/B becomes less represented. The authors note the changes are not uniformly monotonic, the baseline already favors C/D at high p_D, and each cell uses a single optimization seed.

• Predictor side information also redistributes representation. Supplying the current C image through a separate side encoder concatenated with the main latent (no auxiliary loss, no masking from the main input) increases representation of D for p_D ≥ 0.75 and decreases it at p_D = 0.60, while A's representation increases throughout the sweep. The paper notes predictive closure does not account for A and B separating, since it values them only as a pair.

• Steering can be indirect. Both steering interventions target only C, yet both can strengthen D. The authors point out that an auxiliary loss can target either a feature that helps predict the desired feature's next values or one whose next values the desired feature helps predict, whereas side information can supply only the first kind.

• A note on measurement limits. Low decoding accuracy from the main latent alone does not establish that a feature is unavailable to the predictor, since information may be recoverable jointly from the main and side representations.

Methodology in Plain English

The paper has a theory half and an experiment half.

For the theory, the authors start from JEPA's standard training objective: a prediction loss on consecutive latents plus a regularizer that keeps the latent distribution from collapsing. Because the regularizer pins the latent's entropy, every encoder is subject to the same "capacity budget," so the prediction loss is the only thing left to optimize. They then argue — with a proof sketch and a full derivation in the appendix — that under a Gaussian prediction-error assumption, minimizing squared prediction error is equivalent to maximizing the mutual information between consecutive latents, which equals the predictive information of whichever feature set the latent can represent. Selecting the encoder is therefore the same as selecting a feasible feature set with maximal predictive information.

For the experiments, they need sequences whose predictive relationships are known and can be changed one variable at a time, which natural data does not allow. So they construct synthetic sequences of CIFAR-10 image collages, train JEPA systems on them, freeze the encoder, and use probes to measure how accurately specific target features (a traffic-light state, or binary indicators of target-image presence) can be decoded from the latent on held-out observations. Everything uses a three-dimensional latent — deliberately small so that features must compete for capacity — with a training objective combining prediction loss and SIGReg regularization at coefficients of 0.9 and 0.1 respectively. Architecture and training budget are held fixed across conditions so that any change in what is represented can be attributed to the manipulated property. Unless otherwise stated, each configuration uses three optimization seeds, with error bars showing standard errors across seeds; the steering experiments use one seed each.

Why This Matters

Impact on research. The paper reframes a live debate about JEPA from "which features are predictable enough to keep" to "which sets of features jointly justify their capacity cost." That shift matters for interpretability and safety of self-supervised world models: if a task-relevant feature can be absent from the latent for reasons that have nothing to do with its own predictability — because it competes with a disjoint set, or because its predictive partners lost out — then judging a representation feature-by-feature will systematically mispredict what the model knows. The paper also supplies a vocabulary (aleatoric uncertainty vs. predictive leakage) that connects JEPA design to older information-bottleneck and predictive-information ideas.

Real-world applications (as implications the paper draws or that follow directly from its framing):

• World models for planning and control. Because the latent acts as the world model for each observation, features absent from it cannot inform planning regardless of prediction accuracy — relevant to any system using JEPA-style representations downstream.
• Action-conditioned representation learning. If actions are treated as predictor side information, predictive closure suggests action-conditioned JEPA favors features the agent has an impact on, which is directly relevant to robotics and reinforcement learning. The authors note the consequences of genuine action conditioning remain to be tested.
• Targeted representation engineering. Practitioners who need a specific feature represented but cannot write a loss for it can instead target a predictive partner — for example, supervising velocity to strengthen the representation of position — or supply an extracted feature as side information.
• Multi-sensor and industrial monitoring. The paper's own examples — adding sensors for an independent, highly predictable process displacing original features, or wind measurements favoring the representation of a flag's state — illustrate both a risk and a design lever for systems that fuse multiple information sources.

Industry relevance. Auxiliary objectives for steering JEPA representations have already been proposed (LeCun, 2022; Destrade et al., 2025; Zhu et al., 2026; Vujinovic and Kovacevic, 2026). This paper warns that such steering is not local: it can boost or suppress predictive partners and competitors that were never targeted, so steering should be evaluated against all features needed downstream. That is a practical caution for teams building JEPA-based systems where unmonitored representational drift is costly.

Future Directions

• Effect sizes in realistic settings. The authors state explicitly that their experiments test the direction of each predicted effect, and that measuring its size in large JEPA systems trained on natural data remains for future work.
• Genuine action conditioning. The side-information experiment uses image C as a stand-in for an action; the paper says the consequences of real action conditioning remain to be tested.
• Explaining effects the theory does not cover. In the side-information experiment, A and B separate in ways predictive closure does not account for, since it values them only as a pair — a gap the framework would need to close.
• Connecting to biological representation learning. The conclusion suggests the set-level view may be relevant to models that link representation to prediction (Palmer et al., 2015; Chalk et al., 2018; Singer et al., 2023), which is raised but not tested here.

Target Audience

Researchers and engineers working on self-supervised representation learning and JEPA-style world models, especially those who need to reason about what such a model does and does not encode. It suits readers comfortable with mutual information and entropy arguments who want a compact theoretical framing plus small, interpretable experiments. Practitioners designing auxiliary losses, side-information inputs, or action-conditioned predictors will find the steering results and the warning about non-local effects most directly actionable. Readers looking for large-scale benchmark numbers will not find them here — the paper is explicit that it tests direction, not magnitude, on synthetic collages.

Authors’ abstract

In joint-embedding predictive architectures (JEPA), an encoder and a predictor are trained together to produce a representation that minimizes the error in predicting the representation of the next observation. Because the representations of both the current and the next observation are learned, it is unclear which features they will include. Earlier characterizations describe the features JEPA selects in terms of their individual predictability, raising the concern that unpredictable features, like a traffic light, will not be encoded into the representation. However, the prediction error is also affected by the relationships between features, e.g., the light's color helps predict whether the traffic will stop. We propose the predictive closure characterization, which accounts for these relationships. Based on a reformulation of the JEPA training objective, we show that selection depends on properties of feature sets. This view agrees with earlier works when features evolve independently, but makes new predictions when they do not, for instance that a feature like the traffic light can still be retained. We test this characterization in controlled experiments, which show that unpredictable features can become better represented when they help predict other features, and that changing one feature's predictive relationships can affect how well others are represented. We also demonstrate implications for steering feature selection: targeting one feature with an auxiliary objective, or supplying it to the predictor as side information, can strengthen the representation of another feature whose future values it helps predict, so steering can be indirect.