Learning the 4D World through Point Trajectory Forecasting

AI summary

Overview

Research area: Computer vision and generative world modeling — specifically multi-view video forecasting, 4D scene understanding, and vision-language-action (VLA) robot control.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, transformer architectures, camera geometry (intrinsics, SE(3) poses, Plücker rays), and robot imitation learning.

Scope: The paper introduces VPM (Video-Point Model), a multi-view world model built on Cosmos 3 that jointly forecasts future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points, plus the VPM-Data training corpus, the VP-WorldBench evaluation suite, and an action-enabled extension for robot manipulation.

What This Paper Is About

World models that predict future video can produce visually convincing frames without necessarily describing coherent 3D motion — a serious problem when several cameras must agree about the same evolving scene. The authors' core hypothesis is that forcing a model to also predict where identifiable physical points will move through time gives it a shared, cross-view description of motion that anchors appearance changes in every camera. The goal is therefore to jointly forecast future multi-view videos and persistent 3D point trajectories from posed video histories plus a language instruction, and to test whether that point supervision also improves downstream robot control.

Key Contributions

1. VPM, a camera-conditioned multi-view video–point world model. Built on Cosmos 3, it packs latent video tokens from all cameras into a single shared sequence, encodes per-token camera geometry via Plücker-ray embeddings with camera-dependent attention rotations, adds a fourth "view" axis to rotary positional encoding ("4D RoPE"), and attaches a trajectory head that predicts persistent 3D point positions in a fixed reference frame. Point supervision is applied both to the head's outputs and directly to the generator's attention features through a cross-view correspondence objective.

2. VPM-Data, a simulation corpus with aligned visual and geometric supervision. 10K training clips (8,000 tabletop manipulation, 2,000 urban dynamics) pairing synchronized multi-view videos with persistent 3D trajectories, camera geometry, visibility annotations, and language captions. Each object is annotated with 256 surface points and the robot's arms and hands with 512 additional points, propagated through motion and occlusion.

3. VP-WorldBench, a forecasting benchmark spanning simulation and the real world. It covers simulated tabletop manipulation (500 clips, 3 views, 20 fps) and urban dynamics (500 clips, 4 views, 15 fps), plus real-world tabletop manipulation (100 clips, 2 views) and indoor scenes (100 sequences, 48 synchronized RGB cameras). It includes "hard" subsets where individual cameras lose sight of the target for at least 30 consecutive frames, and evaluates visual fidelity and 3D motion accuracy on the same held-out events.

4. An action-enabled extension and evidence that point supervision transfers to control. Robot actions are added as an additional token modality sharing the point-supervised backbone, denoised jointly with future video. Across LIBERO, LIBERO-Plus, RoboCasa, and real-robot pick-and-place, the trajectory-supervised variant improves success rates over its video-only counterpart.

Main Findings

• Point supervision improves video forecasting on every split and metric. On VP-WorldBench, VPM (V+T) reaches FVD 67.35 / PSNR 22.08 / MEt3R 0.016 / δ³ᴰ 50.05 on simulated robot scenes; 81.92 / 19.16 / 0.050 / 36.80 on simulated general scenes; 255.19 / 20.57 / 0.064 / 64.75 on real robot scenes; and 56.48 / 22.10 / 0.007 / 33.92 on real general scenes. The video-only VPM (V) scores 78.52 / 21.41 / 0.022 / 43.62; 101.37 / 16.95 / 0.057 / 28.71; 296.83 / 18.42 / 0.074 / 50.83; and 62.85 / 20.96 / 0.010 / 26.54 respectively.

• Multi-view processing alone already helps. VPM (V) improves over its Cosmos3-Nano backbone (e.g., simulated robot FVD 104.74 → 78.52; real general FVD 66.24 → 62.85) even before point supervision is added, and both variants are trained with the same budget so the comparison isolates the effect of trajectories.

• Gains hold zero-shot on external 4D generation benchmarks. On ReCamMaster (100 clips), VPM (V+T) achieves FVD 79.63 / PSNR 17.06 versus 103.12 / 15.09 for VPM (V) and 125.70 / 13.99 for Cosmos3-N. On CMU Panoptic (100 clips), it reaches FVD 128.64, PSNR 23.24, MEt3R 0.029, δ³ᴰ 42.03. On RoboCasa Kitchen it scores PSNR 22.83, SSIM 0.8801, LPIPS 0.0537, MEt3R 0.020, δ³ᴰ 53.70, beating Robot4DGen and X-WAM.

• Point supervision is what closes the 3D-consistency gap. VPM (V) trails the geometry-aware Robot4DGen on multi-view consistency, whereas VPM (V+T) surpasses it — the authors point to this as evidence that trajectory learning, not just multi-view input, drives geometric coherence.

• Point supervision raises robot manipulation success. VPM (V+T+A) achieves the best success rates on every benchmark: LIBERO 97.2, LIBERO-Plus 84.6, RoboCasa 76.2, and real-world pick-and-place of Banana 79.2, Cup 70.8, Cube 54.2 — versus 96.8 / 82.0 / 73.4 / 58.3 / 64.6 / 43.8 for the action-enabled model without point supervision (VPM (V+A)), and versus 96.8 / 68.0 / 62.1 / 41.7 / 47.9 / 12.5 for π0.5.

• All three modalities combine best. In the modality-wise ablation, adding trajectories (T) yields the largest gains in video fidelity and 3D motion accuracy on every split, adding actions (A) yields modest improvements, and training with V+T+A performs best overall — e.g., simulated robot FVD 58.97, PSNR 22.43, MEt3R 0.015, δ³ᴰ 53.14, and real general δ³ᴰ 37.03.

• Predicted trajectories alone carry actionable information. Converting forecast trajectories into commands via inverse kinematics, without action tokens, completed unseen real-robot tasks zero-shot; the authors note these rollouts are not yet consistent enough for quantitative evaluation.

Methodology in Plain English

The team starts from an existing pretrained video world model (Cosmos 3, specifically the Cosmos3-Nano backbone) whose architecture pairs a language "reasoner" tower with a video "generator" tower. They make three architectural changes so it can handle multiple cameras at once:

• One shared sequence for all cameras. Instead of processing each view separately, tokens from every camera are packed into a single sequence on a shared temporal grid, so attention can freely mix information across viewpoints.
• Telling the model where each camera is. Every video token gets a camera embedding derived from the Plücker ray of its image region (origin plus direction), which is added to the features and also used to rotate attention queries and keys per layer. A fourth positional axis encoding the camera's identity is added to the usual three spatial/temporal axes, so tokens from different cameras at the same pixel and time are distinguishable. The view frequencies are initialized to zero so the model starts out equivalent to the original 3D encoding.
• A trajectory head wired into the generator. From an intermediate self-attention layer (the 18th), the model reads out per-view feature grids, correlates a queried point's source feature against candidate locations in each view and timestep, and refines the estimate over 4 iterations with attention factorized over space, time, and views. For every view it predicts a 2D position, a depth, and a confidence, which are unprojected into a fixed reference frame to give a 3D point. Crucially, during training these features come from partially denoised future tokens, not clean ground truth, so the head learns to read motion out of the model's own predictions.

Supervision has two parts: a tracking loss (truncated Huber on 2D positions and frame-to-frame displacements, truncated Huber on log-depth, and binary cross-entropy on confidence, summed over the 4 refinement iterations) and a cross-view correspondence loss that treats matching as classification over spatial cells and directly penalizes the attention maps for failing to link the same physical point across views and time.

Training proceeds in three stages on 32 NVIDIA GH200 Grace Hopper GPUs: (1) adapt the generation pathway, camera embeddings, and view frequencies with flow matching on all multi-view data, including clips without trajectories; (2) freeze everything except the trajectory head and view frequencies and train with the tracking loss; (3) jointly optimize the video backbone and trajectory head with the combined flow-matching, tracking, and correspondence objective. The reasoner tower stays frozen throughout. The authors build their own simulation data — tabletop pick-and-place with a Fourier GR-1 T2 robot in Unreal Engine 5, and urban traffic/crowd scenes — exporting images and geometric annotations from the same simulation timestep so views stay aligned.

For robot control, actions are turned into an extra token modality (9-dimensional vectors, following Cosmos 3) that shares the backbone and is noised and denoised independently of the video. At inference, video and action tokens both start from noise and are denoised jointly, conditioned only on observations and the instruction. Evaluation is deliberately strict: no method, including VPM, receives future camera parameters, so camera motion must be inferred from history. Models get 16 frames of history and predict the next 64 frames at 480p; quality is measured with FVD and PSNR, geometric consistency with cross-view MEt3R, and motion accuracy by extracting 3D tracks from generated videos with MVTracker and comparing to ground truth using depth-adaptive thresholds (δ³ᴰ).

Why This Matters

Impact on research. The paper isolates a clean, testable claim — that predicting where physical points go is a better learning signal for video world models than appearance alone — and backs it with controlled ablations on matched training budgets. It also connects two literatures that usually stay apart: geometric/motion-aware video generation and video-based robot policy learning. By showing that 3D trajectory supervision lifts both fidelity and manipulation success, it argues that 4D scene representation is not just an evaluation nicety but a training signal that transfers to action.

Real-world applications:

• Robot manipulation: the action-enabled variant outperforms both VLA policies (π0.5, GR00T N-series) and world-action models on LIBERO, LIBERO-Plus, RoboCasa, and three real pick-and-place tasks on a Franka Research 3 with a Robotiq 2F-85 gripper.
• Simulation-to-real training data: VPM-Data offers an automated way to produce synchronized multi-view video with dense 3D point annotations and captions, avoiding manual labeling.
• Autonomous systems in dynamic scenes: the urban dynamics domain (pedestrians and vehicles, with hard cases where cameras lose the target for 30+ frames) targets exactly the occlusion-heavy forecasting that matters for monitoring and driving.
• Free-viewpoint video and 4D content creation: zero-shot results on ReCamMaster and CMU Panoptic show the model generating novel views with more coherent 3D motion.
• Home/indoor scene understanding: the 48-camera real indoor benchmark evaluates people handling objects and interacting with furniture.

Industry relevance. Model-based robot policies that internally simulate outcomes are a major commercial direction. VPM's message — that a cheap, geometry-grounded auxiliary objective improves both generation and control — is directly actionable for teams already fine-tuning video foundation models, because it requires no new sensor modality at inference, only extra supervision during training.

Future Directions

• Scaling trajectory supervision to real multi-view video. Current supervision is simulated and annotation-heavy; deriving 3D tracks from real captures at scale would remove the dependence on a simulator.
• Denser, non-rigid content. The authors note their annotations cover only the robot and annotated objects, and that skeletal annotations omit independent cloth and hair motion.
• Relaxing calibration requirements. VPM assumes known camera intrinsics and poses at inference; removing that assumption is listed as future work.
• Containing the cost of many views. All camera streams share one token sequence, so compute grows with the number of views — a real constraint given the 48-camera indoor benchmark.
• Turning trajectory forecasts into policies directly. Zero-shot inverse-kinematics rollouts from predicted trajectories were promising but "not yet consistent enough for quantitative evaluation," leaving open whether trajectory forecasting could serve as a policy on its own.

Target Audience

Researchers and engineers working on video generation, 4D scene reconstruction, multi-view geometry, and vision-language-action models will get the most from this paper. It is also valuable for robot learning practitioners interested in auxiliary objectives that improve policy robustness, and for anyone building evaluation suites for spatiotemporal generation — the VP-WorldBench design, with paired visual and geometric metrics on identical held-out events and hard occlusion splits, is a reusable template. Readers without a background in flow matching, camera geometry, and transformer attention will find the method sections demanding.

Authors’ abstract

Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Model (VPM), a world model that conditions on multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points. We extend a pretrained world model with camera-conditioned multi-view processing that integrates complementary observations across cameras, and add a trajectory head that predicts how physical points move through time. Each trajectory tracks a single point across all views, and supervising these trajectories alongside video generation therefore anchors appearance changes in every camera to a shared description of 3D motion. To support training and evaluation, we build VPM-Data, a simulation corpus pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark spanning simulated and real environments. We further extend VPM to robot control by adding robot actions as a token modality in the generator. Action tokens share the point-supervised backbone and are denoised jointly with future videos, thereby grounding the policy in the same 3D-aware representation. On VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart, and in simulated and real-robot experiments, it also raises the manipulation success of the action-enabled model.