Probing Spatial State Representations in Visual Foundation Models

AI summary

Overview

• Research area: Computer vision / representation learning — specifically, what spatial information visual foundation models (image encoders, video encoders, 3D reconstruction models, video generators, world models, and multimodal models) actually encode.
• Technical level: Advanced. The paper assumes familiarity with probing methods, frozen-feature readouts, retrieval metrics, camera poses, and representation learning benchmarks.
• Scope (one sentence): A frozen-feature probing study of ten visual foundation models that tests four functional levels of spatial capability — spatial structure, object persistence, predictive spatial update, and actionability — under streaming egocentric video of static indoor scenes.

What This Paper Is About

A moving observer sees only part of the world at any moment, yet must remember where things are once they leave view, anticipate how the view will change when they move, and use that knowledge to act. The paper asks whether visual foundation models internally represent this kind of persistent spatial state, rather than merely being good at recognizing what is currently visible in a single image. The authors break the question into four graded capabilities and test ten frozen models on them using lightweight probes, matched controls, and layer/context sweeps.

Key Contributions

1. An operational definition of spatial state. The authors define spatial state representations functionally through four levels rather than prescribing a 3D map, scene graph, coordinate system, or architecture:

   • L1 Spatial structure — metric geometry (depth and camera pose), instance identity, and view consistency.
   • L2 Object persistence — retaining the bearing of an object after it leaves the field of view.
   • L3 Predictive spatial update — direct future-feature prediction and recurrent path integration under ego-motion.
   • L4 Actionability — goal-conditioned action prediction, tested in both a candidate-selection and a continuous-output setting.

2. A frozen-feature probing framework across ten representations. Ten models spanning image encoders, video encoders, 3D models, multimodal models, video generators, and world models are probed without updating backbone parameters: DINOv2, V-JEPA2, VideoMAE, Fast3R, Wan, CogVideoX, MiniMax-H3, LingBot-video, Aether, and Qwen2.5-VL. The framework organizes analysis along two axes — network depth and streaming prefix length (prefixes of 8, 12, 16, and 24 frames).

3. Matched controls that separate decodability from input use. Rather than reporting only readout accuracy, the study adds controls that alter motion, visual history, transformation order, or goals while holding the probe and target fixed (e.g. correct versus wrong motion in pose-matched candidate pools; full history versus current frame alone; copying the last feature as a temporal baseline; shuffled goals at fixed weights).

4. Evidence that readout accuracy, input use, and goal-specific utility dissociate. The paper shows these three notions come apart across models, layers, and prefix lengths, and identifies two distinct "ingredient chains" running through the four levels — motivating multidimensional rather than single-score evaluation of spatial representations.

Main Findings

• No representation leads on all tasks. Fast3R has the strongest camera-pose readout (AUC@30 0.3870) and lowest object-persistence error (32.79°); DINOv2 leads direct future-feature prediction (global R@1 0.2373); V-JEPA2 leads path integration (0.1588). DINOv2 and V-JEPA2 reverse order between direct prediction (0.2373 versus 0.0625) and path integration (0.0900 versus 0.1588).

• Strong visible-scene readout does not imply object persistence. VideoMAE localizes previously seen objects more accurately (36.45°) than DINOv2 (65.38°) and Qwen2.5-VL (68.59°), even though both outperform it on depth, instance identity, and view consistency.

• Object persistence tracks camera-pose readout. The three most accurate localizers (Fast3R, VideoMAE, V-JEPA2) are exactly the three models with the highest AUC@30, and the cross-model rank correlation between persistence and camera pose is ρ = 0.76, compared with ρ ≤ 0.43 for depth, instance identity, and view consistency. Depth, instance identity, and view consistency rank models almost identically (ρ ≥ 0.91 pairwise), whereas camera pose does not (ρ ≤ 0.62). In all four multi-layer backbones swept (VideoMAE, V-JEPA2, Wan, CogVideoX), the block where persistence peaks is the block where metric geometry peaks.

• Persistence and prediction produce unrelated model orderings. Direct prediction and path integration rank models similarly to each other (ρ = 0.93 over the nine models under the common prediction protocol) but neither ordering relates to persistence (|ρ| < 0.1). Fast3R, best on every L1 metric except instance identity, ranks eighth on direct prediction and seventh on path integration.

• Different readouts favor different layers. V-JEPA2 peaks at block 23 for direct prediction, path integration, and instance identity but at block 17 for object persistence; VideoMAE peaks at block 23 for localization and instance identity and at block 17 for both prediction tasks; Wan’s best block is 15 for five of six readouts although its designated main-table layer is block 20; CogVideoX peaks at block 10 or earlier for metric geometry, persistence, and both prediction tasks.

• Layer choice can flip cross-model orderings. At block 17 and prefix 8, V-JEPA2 reaches a localization error of 28.8°, below VideoMAE’s best cell of 29.4°, whereas at the designated layers VideoMAE leads by about 9°.

• Longer prefixes do not uniformly help. Among 56 model–metric endpoint pairs for the eight models compared, 45 favor the shorter prefix p8, eight favor p24, and three tie; 33 and five of these respectively have paired 95% intervals excluding zero. All eight object-persistence pairs worsen with the longer prefix, each beyond its interval. VideoMAE’s localization error rises from 29.55° to 54.80°, and DINOv2’s path-integration R@1 falls from 0.1256 to 0.0713. Every pair favoring p24 is a visible-scene readout (depth, camera pose, or view consistency), with the single exception of CogVideoX path integration. Because a longer prefix also moves the endpoint, these comparisons do not isolate added history.

• Decodability is not input use. Motion gain (correct minus wrong motion) is 18.84 percentage points for V-JEPA2, 10.51 for VideoMAE, 4.35 for CogVideoX, and 2.90 for Wan with an interval spanning zero. History benefit (full history minus current frame) is 10.26° for V-JEPA2 and 10.01° for Wan, both intervals above zero, but 2.08° for VideoMAE and −3.41° for CogVideoX with intervals spanning zero. Notably, VideoMAE — the most accurate localizer among these four in the main table — shows no measurable benefit from history beyond the current frame.

• Copying can beat learned prediction. In the one-step protocol, Wan achieves 15.22% with learned prediction versus 11.96% by copying the last observed feature, whereas V-JEPA2 achieves 13.04% versus 19.93% for copying — copying exceeds learned prediction for V-JEPA2.

• Native prediction preserves recoverable structure but not the readout interface. A reader trained on real V-JEPA2 future features and applied unchanged to native predictions sees normalized-depth MAE rise from 0.182 to 0.246 at horizon two; retraining an equal-capacity reader on the predictions improves MAE to 0.223, better than the 0.241 layout prior but worse than the real-feature readout. Horizon four shows the same pattern, and frozen transfer remains better than copied context but slightly worse than the prior.

• Goal sensitivity is not goal utility. In the candidate-selection setting, direct selection reaches 75.9–79.9% accuracy against a 67.8% action-only baseline, and shuffling goals reduces accuracy by 12.2–20.7 percentage points. In the continuous-output setting, FULL achieves positive progress on 50.2–59.8% of trajectories and beats GOAL-ONLY by 6.0–13.8 percentage points, but does not improve over CURRENT-ONLY for any backbone. Substituting a difficulty-matched goal from another scene rotates predicted actions by 29.8–45.6°, yet correct-minus-shuffled-goal effects on positive progress (−2.3 to +1.2 percentage points) and goal reduction (−0.006 to +0.001) all have 95% intervals containing zero.

• Two ingredient chains run through the four levels. Object persistence and actionability build on camera-pose readout — among the four models evaluated at L4, Fast3R leads both action settings (79.9% candidate-selection accuracy, 59.8% positive progress), V-JEPA2 is second in candidate selection (78.1%) and tied with DINOv2 in positive progress (53.8%), and Wan is last (75.9% and 50.2%), matching the persistence order of 32.79°, 45.31°, then 62.38° and 65.38°. Feature prediction instead builds on visible-scene readout: it correlates with depth, instance identity, and view consistency (ρ = 0.52–0.63) but not camera pose (ρ ≤ 0.25) or persistence (|ρ| < 0.1). In each chain the lower level bounds the higher one without guaranteeing it.

Methodology in Plain English

The authors treat each model as a fixed feature extractor. They feed it increasingly long causal video prefixes (8, 12, 16, or 24 frames) drawn from posed ScanNet++ indoor trajectories, encode each prefix independently with all backbone weights frozen, and then train small, shallow "probe" networks to read specific quantities out of those features.

Each capability has its own probe design. Depth and camera pose are decoded directly from the prefix features. Instance identity is measured by a cross-entropy loss over per-pixel embeddings with prototypes from ground-truth instance groups. View consistency matches correspondences between features from two viewpoints. For object persistence, a target seen earlier is summarized into an object query; by the last frame it has left the view, and the probe must predict its bearing relative to the final camera — neither the visibility mask nor the target location is given. For prediction, a probe receives the prefix plus the camera motion to a future frame and must retrieve that future frame’s feature (direct prediction, pooling offsets of 1, 2, and 4), or applies successive motions in a recurrent loop (path integration). For actionability, a probe receives the current history plus an independently encoded goal frame and either scores two candidate 9D camera motions or predicts a 9D motion directly.

The critical layer of the design is the matched controls. Because a probe can be accurate for reasons unrelated to the capability under test, the authors repeat evaluations while swapping in the wrong motion, removing history, replacing history with the current frame alone, zeroing or shuffling visual tokens, copying the previous feature instead of predicting, or substituting a mismatched goal at fixed weights. They also sweep network depth and prefix length to check whether conclusions depend on the arbitrary choice of extraction point. Statistical uncertainty comes from 95% scene-clustered bootstrap intervals over 69 validation scenes (1,000 shared resamples, so comparisons are paired); the main table uses 276 validation windows per prefix and training seed 42.

Why This Matters

The paper argues that evaluating spatial representations only by what can be decoded from them is insufficient — a representation can expose information that the model does not actually maintain, update, or use. This reframes benchmarking for any model claimed to have spatial or world-modeling ability.

Real-world applications:

• Robot navigation and embodied agents: knowing whether a frozen perception backbone supplies a usable persistent state and reliable goal-conditioned actions determines whether it can be dropped into a navigation stack.
• Augmented and virtual reality: headset systems need to remember object locations after they leave the camera frustum and to predict how the view changes under self-motion.
• Autonomous driving and mobile mapping: streaming egocentric perception with camera motion is the same regime studied here, and the layer/prefix findings bear on how much temporal context to feed a frozen backbone.
• Model selection and system design: because different readouts favor different layers and prefix lengths, the results argue against picking one extraction point for all downstream spatial tasks.

Industry relevance: the study gives practitioners a diagnostic template — including negative controls — for deciding whether a foundation model’s internal features are genuinely useful for a spatial task, rather than relying on training-objective performance or a single linear probe. The finding that longer context can hurt object persistence is directly actionable for systems that naively extend context windows.

Future Directions

• Beyond static indoor scenes. The authors explicitly scope out object dynamics, contact physics, and complete physical world modeling; extending the four-level framework to moving objects and physical interaction is an open problem.
• Closed-loop evaluation. The action experiments are supervised offline diagnostics, not a demonstration of a closed-loop policy; testing whether these representations support real goal-directed control with feedback remains open.
• Disentangling architecture, objective, and data. The authors note their cross-model comparisons do not isolate the effect of training objective, since architecture and training data differ as well; controlled pretraining comparisons would be needed.
• Fixing the prefix confound and the data-overlap caveat. Longer prefixes also move the endpoint, so history effects are not isolated; and Fast3R’s training data include ScanNet++, with some validation scenes among its training scenes, so its leading camera-pose, persistence, and action results may partly reflect familiarity. Designs that separate history from endpoint, and cleaner evaluation splits, would strengthen the conclusions.
• Better readout interfaces for predicted features. Since natively predicted features retain recoverable geometry but not the original readout interface, understanding what interface predicted features require is an open design question.

Target Audience

Researchers and engineers working on visual foundation models, world models, embodied AI, and 3D/streaming perception who want a rigorous, control-heavy methodology for evaluating what a representation actually contains. It is also useful for practitioners choosing a frozen backbone and extraction layer for a spatial downstream task, and for benchmark designers looking for evaluation protocols that go beyond single-number probing accuracy. Readers without background in probing methods, retrieval metrics, or camera-pose evaluation will find the experimental details dense despite the plain framing of the four levels.

Authors’ abstract

An observer sees only part of the surrounding world, yet must keep track of what lies beyond the current view. Spatial perception therefore involves more than recognizing individual images: it calls for a persistent spatial state that is updated under self-motion and used to guide action. Do visual foundation models provide representations that support such a state? We study this question through four levels of spatial capability---spatial structure, object persistence, predictive spatial update, and actionability---using frozen-feature probes on ten representations under streaming egocentric observations of static indoor environments. The resulting capability profiles vary across tasks, network layers, and prefix lengths: strong readout of the visible scene does not ensure accurate object persistence or future-feature retrieval. Matched controls separate readout accuracy from reliance on visual history and motion. Native future prediction retains recoverable spatial structure, but a spatial reader trained on real features does not transfer to predicted features unchanged. Finally, goal-dependent candidate selection does not establish reliable continuous action utility: predicted actions respond to goal changes, yet supplying the correct goal does not reliably improve progress. Across the four levels, object persistence and actionability build on camera-pose readout, whereas feature prediction builds on visible-scene readout. Together, these findings motivate evaluating spatial representations not only by what can be decoded from them, but by whether that information is maintained beyond observation, updated predictively, and used for goal-directed action.