GeoState: Learning View-Decoupled Geometry States for 3D World Modeling

AI summary

Overview

Research area: Computer vision — 3D world models, geometry foundation models (GFMs), and autonomous-driving scene forecasting.

Technical level: Advanced. The paper assumes familiarity with multi-view geometry (camera intrinsics/extrinsics, Plücker rays, point maps), transformer-based latent world models, and the VGGT family of geometry foundation models.

Scope: The paper proposes GeoState, a fixed-length, view-decoupled geometry state built from frozen VGGT-Ω features, and shows that predicting future states in this space (with camera motion handled separately) improves long-horizon depth forecasting, cross-view query consistency, and zero-shot transfer across driving datasets.

What This Paper Is About

Recent 3D world models use features from geometry foundation models as the "state" they try to predict into the future. But those features are organized around the observing camera, so the state changes when the camera moves even if the physical scene does not — forcing the temporal model to predict two entangled things at once. GeoState's goal is to build a state that describes what the world is, leaving the camera to determine only how that world is observed, and then to predict the future directly in that decoupled state space.

Key Contributions

1. Problem identification: The authors identify viewpoint coupling in GFM features as a specific limitation when those features are used as predictive world states, since camera-induced variation is mixed with genuine scene dynamics.
2. View-decoupled state: They introduce GeoState, built by a Scene Tokenizer that aggregates multi-view GFM features from a shared anchor coordinate frame into a fixed-length scene state, plus a query-conditioned Geometry Decoder that introduces the query camera only at decoding time — so one state can be re-queried from many viewpoints without re-encoding the scene.
3. World model in state space: They formulate a 3D world model that predicts scene evolution directly in GeoState space via residual updates, while camera evolution is modeled separately, and the rollout keeps a single context-derived scale without per-frame scale alignment.
4. Empirical validation: They report gains over prior 3D world models on Waymo long-horizon depth forecasting, geometrically consistent cross-view querying, and zero-shot generalization to KITTI and nuScenes.

Main Findings

• Waymo long-horizon forecasting: With predicted future cameras, GeoState beats DINO-Foresight, VGGT-World, FR3D, and the Copy-Last baselines on AbsRel and δ₁ at every reported horizon: AbsRel 0.065 at t+0.2 s, 0.079 at t+0.6 s, 0.092 at t+1.0 s, and 0.127 at t+2.0 s (δ₁ 0.939, 0.921, 0.902, 0.848 respectively).
• Growing advantage with horizon: The gap over VGGT-World is reported as a 40.4% AbsRel reduction at t+2.0 s. These results are obtained while using one context-derived scale, and they hold beyond the 1.4 s horizon seen during training.
• Decomposing camera vs. scene effects: Copy-Last with predicted cameras (re-querying the last observed state at future cameras) gives AbsRel 0.071 / 0.088 / 0.100 / 0.132, clearly better than Copy-Last Fixed View (0.094 / 0.147 / 0.184 / 0.248) — showing much of the apparent "future change" is camera-induced. GeoState's predicted states nonetheless beat their camera-matched Copy-Last counterparts at every horizon under both ground-truth and predicted cameras, so the world model captures real scene evolution too.
• Ground-truth camera setting: GeoState with GT cameras reaches AbsRel 0.062 / 0.071 / 0.081 / 0.099 versus Copy-Last GT at 0.068 / 0.082 / 0.092 / 0.108.
• Cross-query geometric consistency: Under Small (0–0.5 m, 0–2°), Medium (0.5–1.0 m, 2–5°), and Large (1.0–1.5 m, 5–10°) virtual-camera perturbations, predicted states closely match observation-encoded states on symmetric reprojection AbsRel at every forecast horizon (reported in Figure 3; numeric values are not given in the text provided).
• Zero-shot transfer: Without target-dataset adaptation, GeoState with predicted cameras outperforms DINO-Foresight, VGGT-World, and the reported FR3D results on both metrics at all reported horizons on KITTI (AbsRel 0.095 / 0.112 / 0.155 at t+0.6/1.0/2.0 s) and nuScenes (0.136 / 0.172 / 0.213 at t+0.75/1.25/2.5 s).
• Compression is lossless enough: Reconstructing depth through the Scene Tokenizer and Geometry Decoder with ground-truth query cameras is slightly better than direct VGGT-Ω depth on Waymo (AbsRel 0.068 vs 0.074 at t+0.2 s; 0.072 vs 0.078 at t+2.0 s), showing the fixed-length state retains the needed geometric information.
• View decoupling makes the state more predictable: Against a baseline that compresses the same GFM features into a 512×512 latent without decoupling, reconstruction quality is comparable (AbsRel 0.075 vs 0.073 at t+0.6 s), but forecasting is much better for GeoState (predicted: 0.081 vs 0.092 at t+0.6 s; 0.095 vs 0.111 at t+1.0 s; 0.133 vs 0.144 at t+2.0 s).
• State supervision matters only at initialization: One-step state matching helps (AbsRel 0.071 vs 0.075 at t+0.6 s and 0.071 vs 0.077 when combined with rollout supervision), but keeping latent state matching during rollout hurts long horizons — the best configuration is one-step state supervision on, rollout state supervision off (0.071 / 0.081 / 0.099 AbsRel).
• Motion stratification: Improvements are more pronounced in Dynamic than in Static Waymo windows under both GT and predicted cameras (detailed results in Appendix A.8.2).

Methodology in Plain English

1. Encode a scene, not a view. Multi-view images go through a frozen VGGT-Ω model, which returns features from four internal levels, pixel-aligned 3D point maps with confidences, and estimated cameras. To keep a common coordinate anchor across time, the model is fed observations whose first input view is shared, but only the outputs for the current timestamp are kept.
2. Add explicit geometry. Because raw GFM features vary in scale across forward passes, each patch feature is fused with explicit cues: its 3D location and confidence, the source camera pose, and its viewing ray — all median-normalized in scale. A small MLP predicts a correction to the feature, and the sum forms a patch memory.
3. Compress to a fixed-size state. A Perceiver-style tokenizer with 512 learned scene queries (512 layers of width 512, eight Perceiver layers, eight heads) pools the variable-length patch memory into a state of size 512×512, independent of how many views were used.
4. Decode from any camera. A query camera is turned into a Plücker ray field in the same coordinate frame and scale, Fourier-encoded and patchified. Scene tokens pass through scene-only self-attention, then join the ray tokens in sixteen transformer blocks; at 416×624 resolution this gives 1,014 query tokens plus 512 scene tokens (1,526 total). Multi-level features from blocks 4, 8, 12 and 16 are adapted to the interface expected by the frozen VGGT-Ω DPT head, which outputs normalized depth, converted back to metric using the context-derived scale.
5. Predict the future in state space. A masked factorized transformer (the FR3D architecture) takes four historical states on a sliding window and predicts a residual update to the latest state, with a zero-initialized residual head so the model starts equivalent to Copy-Last. Rollout is autoregressive; each predicted state is decoded with either a user-specified camera or a separately predicted one.
6. Training. Tokenizer and decoder are pretrained on roughly 10k multi-view scenes from DL3DV, TartanAir, RealEstate10K, CO3D and WildRGB-D (200k steps, learning rate 1e-5), then adapted on a static-content Waymo subset (50k steps, 5e-6), using log-depth plus depth-gradient losses. The world model trains in two stages — 20k steps of one-step prediction with state + depth supervision, then 35k steps of seven-step rollout with depth-only supervision — on about 140k windows from 800 segments, using FRONT, FRONT_LEFT and FRONT_RIGHT context views at 0.2 s intervals. Tokenizer, decoder and GFM stay frozen while gradients flow through the decoder to the predicted states.
7. Evaluation. Depth is scored with AbsRel and δ₁ against LiDAR; scale is calibrated once from LiDAR at the last observed FRONT frame and held for the whole rollout, whereas VGGT-World uses per-target-frame scale alignment, FR3D uses per-target-frame scale-and-shift, and DINO-Foresight is fit per target frame. Cross-view consistency uses a symmetric reprojection AbsRel (E_sym). Test sets are 100 held-out Waymo windows, 113 eligible KITTI drives, and all 150 nuScenes validation scenes.

Why This Matters

Research impact. The paper reframes a subtle but consequential issue: a representation that is excellent for perception is not automatically a good state space for dynamics. It offers a concrete recipe — separate content from observation inside the representation, not just in the architecture — and shows the payoff in predictability, which is relevant to anyone using pretrained geometry features as latent states.

Real-world applications:

• Autonomous driving: longer-horizon, geometrically grounded forecasts of surrounding structure that remain valid under different camera placements.
• Robotics and embodied agents: predictive scene states that can be queried from a planned future viewpoint to support perception-aware planning.
• Simulation and content creation: replaying the same predicted scene state from user-chosen virtual cameras, which the paper demonstrates via cross-query consistency tests with perturbed cameras.
• Multi-camera / fleet deployment: since the state is fixed-size and camera-agnostic, one encoded scene can serve many camera rigs without re-encoding.

Industry relevance. Removing per-frame scale alignment matters for deployment pipelines, where test-time scale fitting is often unavailable or fragile; and the zero-shot transfer to KITTI and nuScenes suggests value for teams that cannot retrain per vehicle platform.

Future Directions

• Embodied settings: the authors explicitly propose extending the framework to embodied scenarios where predictive scene states support perception, planning and interaction in dynamic environments.
• Better camera prediction: the pipeline currently relies on a separately trained camera model (Appendix A.2) or externally supplied cameras; how much end-to-end joint modeling of camera and scene would help is left open.
• Understanding residual state-matching degradation: rollout-time latent matching hurt performance, attributed to context-dependent GFM feature variations that do not correspond to geometric change (Appendix A.7) — a fuller characterization could yield better state targets.
• Dynamic-scene stress testing: since gains are larger in Dynamic than Static windows, extending the evaluation and the state design to more heavily dynamic scenes (and to the nuScenes/KITTI zero-shot regimes in more depth) is a natural next step.

Target Audience

Researchers and graduate students working on 3D world models, latent dynamics models, and neural scene representation; engineers building autonomous-driving forecasting or occupancy/depth prediction systems; and practitioners using geometry foundation models (DUSt3R, VGGT, VGGT-Ω) who want to repurpose their features for temporal prediction rather than single-frame perception. Readers need a solid grounding in multi-view geometry and transformer architectures to follow the method details.

Authors’ abstract

Recent 3D world models increasingly use features from geometry foundation models (GFMs) as predictive world states, benefiting from their rich geometric priors. However, the view dependence of these features entangles scene dynamics with camera-induced variation, making the resulting state evolution harder to predict. We propose GeoState, a view-decoupled geometry representation for predictive 3D world modeling that separates scene-state evolution from camera-dependent observation. Specifically, a scene tokenizer aggregates multi-view GFM features into a view-decoupled scene state, which a geometry decoder queries using camera rays. An autoregressive world model then predicts future scene states, while camera motion is modeled separately. The predicted states can therefore be decoded under either predicted or externally specified cameras, while retaining a single context-derived scale throughout the rollout without per-frame scale alignment. Experiments show that GeoState outperforms prior 3D world models on Waymo, supports geometrically consistent cross-view querying, and generalizes zero-shot to KITTI and nuScenes. Project page: https://hongchang-liu.github.io/GeoState-project-page/