The Pulse
Vision Models Read Indoor Space Better Than They Act on It
A paper submitted to arXiv on October 29 compares ten visual foundation models on spatial perception, prediction and goal-directed action. The models can reveal information about indoor scenes, but offline tests found that goal-sensitive ac
AI.info Team · Written with AI · fact-checked ·
A paper submitted to arXiv on October 29 finds a gap between what visual foundation models can reveal about indoor space and what they can reliably do with that information. The researchers tested ten frozen model representations, from DINOv2 and V-JEPA2 to video generators and a world model, on tasks ranging from reading scene geometry to predicting goal-directed movement. Strong performance on one task did not guarantee success on the next.
The study, “Probing Spatial State Representations in Visual Foundation Models,” treats spatial ability as more than recognizing the objects in a single image. It asks whether a model’s features preserve information about things that leave the camera’s view, anticipate how a scene changes as the observer moves, and support actions that make progress toward a goal.
Four tests, from visible scenes to action
The authors divide spatial capability into four levels: spatial structure, object persistence, predictive spatial update and actionability. Their probes read features from causal video prefixes—sequences that include past and current views, but not future frames. The models’ underlying parameters stay frozen while task-specific probes learn what information can be extracted.
The tests use posed indoor trajectories from ScanNet++, a dataset of 3D scenes. The researchers evaluate geometry and view consistency, whether a probe can locate an object after it leaves view, whether it can retrieve features of a future view given camera motion, and whether it can predict movement toward a goal. The setup measures information accessible in model features; it does not test a robot carrying out a closed-loop task.
“Finally, goal-dependent candidate selection does not establish reliable continuous action utility: predicted actions respond to goal changes, yet supplying the correct goal does not reliably improve progress,” the authors write. Baiqiao Yin, Zhongang Cai, Ruisi Wang, Qiwei Liang, Lei Yang and Ziwei Liu, “Probing Spatial State Representations in Visual Foundation Models”
Fast3R leads on persistence, not prediction
No model topped every measure. Fast3R had the strongest camera-pose readout and the lowest error when locating previously seen objects, at 32.79 degrees. DINOv2 led direct future-feature prediction, while V-JEPA2 performed best on path integration, which tests sequential updates along a camera trajectory. The authors report that object persistence tracked camera-pose readout across models more closely than it tracked depth, instance identity or view consistency.
More visual context did not automatically help. Across the eight models included in the prefix comparison, all eight had worse object-localization results with a 24-frame prefix than with an eight-frame prefix. VideoMAE’s localization error rose from 29.55 to 54.80 degrees. The authors caution that a longer prefix also changes the endpoint of the observation sequence, so the comparison does not isolate the effect of added history alone.
Goal-sensitive predictions did not reliably improve progress
In one action test, a probe chose between two candidate camera movements. Direct selection reached 75.9% to 79.9% accuracy, above the 67.8% action-only baseline; Fast3R scored highest. But a second test asked probes to predict continuous movement rather than choose from candidates. Positive progress ranged from 50.2% to 59.8% of trajectories, and adding the goal did not improve that rate over using the current view alone for any of the four tested representations.
Replacing the correct goal with a difficulty-matched goal from another scene changed predicted directions by roughly 30 to 46 degrees. Yet the change did not produce a reliable difference in progress or reduction in distance to the goal. The paper describes these results as offline diagnostics, not evidence of a working navigation policy.
What the benchmark does—and does not—show
The findings apply to frozen features, learned probes and static indoor trajectories. The study does not test moving objects, contact physics or a robot acting in a live environment. It also notes that some ScanNet++ validation scenes appeared in Fast3R’s training data, which could partly explain that model’s leading camera-pose, persistence and action-readout results.
For model developers, the result is a warning against treating a strong spatial score as proof of general spatial competence. Recognizing visible geometry, remembering an object beyond the frame, forecasting a future view and choosing useful movement are separate capabilities. In these tests, Fast3R’s 79.9% candidate-selection score did not erase the distinction: its continuous-action probe made positive progress on 59.8% of trajectories, and the benchmark remained an offline test rather than a demonstration of real-world navigation.