WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

AI summary

Overview

Research area: Computer vision and embodied AI — generative world models that forecast future video together with explicit 3D object motion, with a downstream evaluation in robotic manipulation.

Technical level: Advanced. The paper assumes familiarity with diffusion/rectified-flow generative models, Mixture-of-Transformers architectures, video VAEs, rotary positional embeddings, and 3D point-trajectory metrics.

Scope: WorldFlow is a single generative model, pretrained on roughly 1.44 million object-level clips, that takes one RGB frame, a language instruction and sparse 3D query points, and jointly denoises a future video plus metric 3D trajectories for those query points; it is benchmarked on a new 1,678-sample suite (WorldFlowBench) and on 12 real-robot manipulation tasks.

What This Paper Is About

Video world models generate convincing futures but leave the underlying 3D motion implicit in pixels, while action-based world models depend on robot-specific action labels that are expensive to collect and hard to transfer across embodiments. Direct 3D motion predictors avoid both problems but are typically trained with flow-only supervision, which the authors argue is a weak learning signal for physical dynamics.

WorldFlow's goal is to fuse the two: keep the dense visual supervision of a pretrained video generator, and add an explicit, query-conditioned 3D flow output that is denoised in the same backbone at the same time.

Key Contributions

1. A joint video-and-flow world model. WorldFlow extends the Cosmos-Nano framework with a unified Mixture-of-Transformers backbone that denoises video latents and 3D flow tokens together under one rectified-flow objective, so explicit trajectory supervision and dense visual supervision are learned simultaneously rather than in separate stages.

2. A continuous 3D flow tokenization scheme. Each query point's frame-to-frame displacement becomes a continuous token, normalized by a domain-specific scalar, augmented with absolute and centroid-relative initial-point geometry, and given rotary positional coordinates on the same temporal/spatial grid as the video tokens (13×9×15 video-token grid alongside 48×64 flow tokens in the reported configuration).

3. A large heterogeneous corpus. Approximately 1.44 million object-level samples (1,438,379 standardized clips in the supplementary inventory), equivalent to 1,224 hours of 49-frame sequences at 16 FPS, drawn from four collections (Sim7, Real3, SpatialCog, MolmoMotion-1M) with source-specific quality control and unified representations.

4. WorldFlowBench. A 1,678-sample held-out benchmark from 11 data sources — 396 from Sim7, 882 from Real3, and 400 from SpatialCog — covering simulated interaction, real-world motion, and spatial-relation reasoning.

Main Findings

• Lowest ADE and FDE on all three evaluation subsets among evaluated learning-based methods. WorldFlow reports ADE/FDE of 0.051 m / 0.094 m on Sim7, 0.081 m / 0.113 m on Real3, and 0.060 m / 0.078 m on SpatialCog, with PWT (mean hit rate over 1, 2 and 5 cm thresholds) of 0.564, 0.311 and 0.449 respectively.

• Largest margin on the spatial-reasoning set. Relative to the strongest external learned baseline on each subset, ADE drops by approximately 46.9% on Sim7, 4.7% on Real3 and 54.2% on SpatialCog. The 0.060 m SpatialCog ADE is described as a 54.2% reduction over the strongest external baseline.

• Real3 is the tightest margin. Only a 4.7% ADE reduction is claimed there, and the PWT of 0.311 is below the best value of 0.332 from Cosmos3-Nano + 4RC. The paper additionally reports a zero-flow Static reference that is excluded from ranking; its Real3 numbers (0.062 m ADE, 0.086 m FDE) are lower than WorldFlow's.

• Joint generation beats flow-only training. Against a flow-only variant with the same backbone, data and recipe, the full model reduces ADE by 48.5%, 19.8% and 55.9% on Sim7, Real3 and SpatialCog, with lower FDE on all three. On SpatialCog, FDE falls from 0.248 m to 0.078 m, a 68.5% reduction. PWT improves on Sim7 and SpatialCog and decreases slightly on Real3 (0.320 to 0.311).

• Direct flow beats generating video and then tracking it. With the video generator and conditioning held fixed, directly generated trajectories outperform Track4World and 4RC applied to WorldFlow's own generated videos on all three subsets. On SpatialCog, direct flow gives 0.060 m ADE versus 0.083 m for Track4World and 0.098 m for 4RC.

• Auxiliary motion losses help consistently. Adding direction, displacement-magnitude and endpoint objectives to the flow-matching loss improves ADE, FDE and PWT on all three subsets; the largest ADE gain is on Sim7, from 0.063 m to 0.051 m (19.0%), and PWT rises from 0.527 to 0.564 (Sim7), 0.299 to 0.311 (Real3) and 0.436 to 0.449 (SpatialCog).

• Real-robot gains. Across 12 manipulation tasks (pick and place, stacking, relative placement), three randomized trials each and 36 executions per method on a 7-DoF xArm with an XHand and a ZED Mini camera, WorldFlow reaches 97.2% (35/36) flow-prediction success versus 25.0% (9/36) for μ0, and 66.7% (24/36) robot-execution success versus 11.1% (4/36). The paper attributes the gap between prediction and execution to grasp failures, trajectory-optimization failures and unstable stacking placement.

Methodology in Plain English

The model starts from a pretrained video generator (Cosmos-Nano) and keeps its language "reasoner" intact. A user supplies three things: an observed frame, a text instruction, and a set of 3D query points on the target object. The observed frame is encoded into clean video latents by a VAE, and the query points are encoded into geometry embeddings using both their absolute positions and their positions relative to the centroid of the query set.

Motion is represented not as absolute positions but as frame-to-frame 3D displacements, divided by a per-domain scale factor so that slow tabletop scenes and fast human activity share a comparable numerical range. These displacements are turned into a sequence of continuous tokens, one per query point per future frame, and each token is anchored to the same temporal index and image patch as the corresponding video token via shared rotary positional coordinates. That shared coordinate system is what lets the two modalities talk to each other inside attention.

At generation time, both the future video latents and the flow tokens start as noise and are denoised together by the shared transformer, with full attention between video and flow tokens and causal attention to the language context. Training uses a weighted rectified-flow loss summed over the two modalities with a shared per-sample noise level, plus three auxiliary losses applied to the reconstructed clean flow: displacement error, direction error (weighted by ground-truth displacement magnitude), and relative endpoint error (skipping points with small net displacement).

Data is assembled from simulations (Sim7), real robot and human video (Real3), a self-collected SpatialCog set, and the released MolmoMotion-1M annotations. Captions come from Gemini-2.5 Pro, object masks from SAM 2, and trajectories from Track4World; MolmoMotion-1M reuses its released annotations after format and validity checks. Training runs in two stages on 64 NVIDIA H800 GPUs: 32,000 steps on the SpatialCog training split starting from Cosmos-Nano checkpoints, then 51,100 steps over the mixed corpus in three curriculum phases (simulation-focused, mixed real-world adaptation, SpatialCog refinement) with a global batch size of 128.

Why This Matters

Impact on research. The paper makes a concrete architectural claim rather than a vague one: explicit 3D motion should be a generative modality inside a video world model, not a post-hoc tracking step applied to generated pixels and not a flow-only side task. The flow-only and generate-then-track ablations test exactly that claim with a controlled setup, which gives other groups a template for isolating the value of geometric supervision. WorldFlowBench also adds a spatial-reasoning split (SpatialCog) that most motion benchmarks lack.

• Manipulation from a single frame. The real-robot pipeline shows a language-conditioned motion predictor feeding a grasp planner and a locally rigid patch-fitting optimizer, so a robot can be driven toward a predicted object outcome without training a policy for each task.

• Cross-embodiment motion references. Because the output is metric 3D displacement in the camera frame rather than joint commands, the same predictor is intended to inform different robot platforms — the paper's data mix deliberately spans human hands, dexterous hands, and parallel-jaw-style manipulation.

• Data-efficient trajectory labeling. Automated captioning, segmentation and tracking pipelines turn existing video into training signal, which matters for any team that cannot afford motion-capture-grade 3D annotations.

• World-model pretraining for robotics. The recipe of pretraining a video generator and then adding a geometric output head offers a route to physical commonsense that does not require action labels at scale.

Industry relevance. The work sits where robot foundation models, synthetic data generation and video-generation infrastructure converge. Companies building manipulation stacks, simulation-to-real pipelines, or general-purpose video world models can read this as evidence that shipping explicit 3D output alongside video is worth the architectural complexity — and that a single-frame, no-motion-history interface is viable for real deployment.

Future Directions

• Automatic query selection. The current formulation requires query points to be supplied in advance, typically derived from a segmented object mask. The authors list automatic query selection as future work, which would remove the dependence on an upstream grounding-and-segmentation step (here Qwen3.8-27B plus depth).

• Cleaner trajectory supervision. The paper acknowledges that automatically generated trajectory annotations propagate noise into training. Tightening the data pipeline, or learning to be robust to noisy flow labels, is an open problem — especially given that on Real3 the zero-flow Static reference already reports low error.

• Multimodal futures. The model currently produces one denoised future per condition. The authors propose exploring multimodal futures, which is natural for a rectified-flow generator but would require new evaluation protocols beyond ADE/FDE against a single ground-truth trajectory.

• Joint flow and action generation. Extending the shared backbone to emit actions alongside video and 3D flow would let the same model serve as a policy, and would test whether explicit motion representations actually help policy learning rather than only motion forecasting.

Target Audience

Researchers and engineers working on video world models, 3D scene flow and point tracking, and robot learning — particularly those interested in replacing action-labeled world-action models with geometry-grounded alternatives. It is also relevant to practitioners building data pipelines that convert heterogeneous video into 3D supervision, and to anyone who needs a benchmark for instruction-conditioned future motion with spatial reasoning. Readers without a background in diffusion or flow-matching generative models will find the architecture and training sections dense, though the problem framing and the benchmark tables are accessible.

Authors’ abstract

Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2\% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9\% and 68.5\%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.