VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control

AI summary

Overview

Research area: Computer Vision — video world models, controllable video generation, 4D scene representation, video diffusion models.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching video generators, Diffusion Transformers (DiT), 3D Gaussians, point clouds, and camera intrinsics/extrinsics math.

Scope: VerseCrafter is a geometry-driven video world model that conditions a frozen Wan2.1-14B video diffusion backbone on a unified 4D geometric scene state to jointly control camera motion and multi-object motion in real-world videos.

What This Paper Is About

Video world models try to simulate how the real world moves, but today's methods can only control camera and object motion through 2D cues (point trajectories, optical flow, masks, boxes) or narrow 3D cues (depth maps, 3D boxes, SMPL-X bodies), which are view-dependent, rigid, or category-limited. The authors propose a single, shared 4D geometric scene state — a static background point cloud plus per-object 3D Gaussian trajectories — that describes both where the camera goes and how each object moves in one world coordinate frame. Rendering this state into control maps lets a pretrained video diffusion model generate realistic, view-consistent videos that follow the specified dynamics.

Key Contributions

1. 4D Geometric Control representation. A unified scene state combining a static background point cloud with per-object 3D Gaussian trajectories in a shared world coordinate frame. Each Gaussian's mean traces the motion path, while its covariance captures the object's spatial extent and orientation — a soft, category-agnostic alternative to rigid 3D bounding boxes or parametric body models such as SMPL-X.

2. VerseCrafter model. A geometry-driven video world model that renders 4D Geometric Control into multi-channel 4D control maps and injects them into a frozen Wan2.1-14B backbone through a lightweight GeoAdapter, a ControlNet-inspired adapter branch. The backbone's architecture and weights are unchanged; only GeoAdapter is trained.

3. VerseControl4D dataset. A real-world dataset with automatically derived prompts and rendered 4D control maps, containing 35,000 training samples and 1,000 validation samples, addressing what the authors describe as a key data bottleneck.

4. Unified inference modes. One framework supports camera-only control, object-only control, and joint control, with background and object channels rendered separately to disentangle camera motion from object motion.

Main Findings

• Joint camera and object control (Table 1, VerseControl4D): VerseCrafter reaches an Overall Score of 88.10, versus 85.47 for Yume, 83.66 for Perception-as-Control, and 83.55 for Uni3C. It records RotErr 0.890, TransErr 3.103, and ObjMC 2.507, compared with Uni3C at RotErr 1.361, TransErr 7.731, ObjMC 5.883; Perception-as-Control at 5.006 / 8.767 / 6.556; and Yume at 7.560 / 8.735 / 7.959.

• Qualitative joint-control comparison (Fig. 4): Perception-as-Control and Uni3C show noticeable human deformation; Yume roughly follows the text-described motion but lacks precise camera control. Uni3C, relying on SMPL-X, is limited to single-person motion and struggles with categories such as vehicles.

• Camera-only control on static scenes (Table 2): VerseCrafter achieves Overall Score 86.80, RotErr 0.650, and TransErr 2.587, versus FlashWorld (85.33 / 1.792 / 3.257), ViewCrafter (84.04 / 2.101 / 9.868), and Voyager (78.12 / 3.557 / 3.880). ViewCrafter and Voyager show distorted facades, drifting structures, or inaccurate camera motion; FlashWorld tends to produce blurred scene boundaries.

• 3D Gaussian trajectories beat boxes and points (Table 3, Fig. 6): Replacing Gaussians with 3D bounding boxes gives Overall 85.45 and ObjMC 4.520; replacing them with 3D point trajectories gives Overall 85.57 and the weakest object-motion consistency at ObjMC 6.896. The full Gaussian model gives Overall 88.10 and ObjMC 2.507. Points and boxes often cause scale drift and misaligned motion.

• Depth channels matter (Table 3, Fig. 7): Removing depth from the background and trajectory controls ("Ours (w/o depth)") yields Overall 85.64, RotErr 1.177, and ObjMC 4.929. Without depth, the model produces incorrect foreground-background ordering and occlusion boundaries drift over time; with RGB+depth it recovers consistent parallax and occlusion.

• Decoupled controls beat merged controls (Table 3, Fig. 8): Merging background and 3D Gaussian trajectory maps into one stream ("Ours (BG & FG Merged)") gives Overall 85.72 and degrades object-motion accuracy to ObjMC 3.726, versus 2.507 for the decoupled design.

• Dataset composition: About 26% of training samples come from Sekai-Real-HQ and 74% from SpatialVID-HQ; 20% of samples depict static scenes; the validation set includes 250 static-scene samples for camera-only evaluation.

• Training and inference cost: Training runs on 16 96-GB GPUs with a global batch size of 16, using Adam at a learning rate of 2e-5, 100 warmup steps, 2,500 iterations at 480P followed by 2,500 iterations at 720P, totaling about 380 hours of wall-clock time. Inference uses 50 denoising steps and a classifier-free guidance scale of 5.0; generating an 81-frame 720P clip on 8 96-GB GPUs takes about 1152 seconds with peak per-GPU memory of about 90 GB.

• Model configuration: Final resolution 720P, 40 Wan-DiT layers, hidden dimension 5120, GeoAdapter injected at blocks [0, 5, 10, 15, 20, 25, 30, 35], pretrained backbone Wan2.1 T2V-14B.

Methodology in Plain English

The pipeline starts from a single input image and a text prompt. Monocular depth and camera intrinsics are estimated with MoGe-2, and object masks come from Grounded SAM2, with the user selecting objects by text prompt or clicks. The input view defines the reference world frame, so the reference camera pose is identity rotation and zero translation. Every pixel is back-projected using its depth to build a 3D point cloud, which the masks split into per-object point clouds and one static background point cloud.

Each object's point cloud is summarized as a single 3D Gaussian, then extended over time into a 3D Gaussian trajectory: the means trace where the object goes, and the covariances capture how its size and orientation change. Users can edit these trajectories in a 3D editor like Blender by dragging and keyframing an ellipsoid proxy; the final conditioning maps are always rendered from the underlying Gaussians, not the ellipsoid UI.

From this 4D Geometric Control state, the system renders per-frame maps in the target camera views: background RGB and depth from projecting the static point cloud, 3D Gaussian trajectory RGB and depth from projecting the object Gaussians, and a soft merged mask marking where the diffusion model should synthesize or overwrite content. For the first frame, the background RGB is replaced by the input image and the mask is set to zero, preserving the first frame. Background and object channels are rendered separately so camera motion and object motion stay disentangled.

The four RGB/depth maps are encoded by the frozen Wan Encoder; the mask is reshaped and interpolated to latent resolution; everything is concatenated channel-wise into a spatio-temporal geometry tensor. A lightweight GeoAdapter branch processes this tensor and, at every 5th Wan-DiT block, its output is linearly projected and added to the backbone as residual modulation. Text prompts are encoded by umT5 and injected into both the Wan-DiT and GeoAdapter blocks. Only GeoAdapter's parameters are updated; each GeoAdapter block is initialized from the weights of its paired DiT block to stabilize training.

To build training data, the authors start from two world-exploration datasets, Sekai-Real-HQ and SpatialVID-HQ, detect shots with PySceneDetect, and keep 81-frame sub-clips matching the Wan2.1 temporal length. Quality filtering keeps clips with 1 to 6 controllable objects (found via Grounded-SAM2 with prompts like "person . human . car . animal"), discards clips where any object mask covers more than 20% of the image area, removes human masks that touch borders or have aspect ratios outside [2,4], and applies aesthetic and luminance filtering. Captions come from Qwen2.5-VL-72B; geometry comes from a MegaSAM pipeline with its monocular and metric depth modules replaced by MoGe-2 and UniDepth V2 for more accurate, temporally consistent depth.

Evaluation uses VBench-I2V for video quality; RotErr and TransErr following CameraCtrl for camera control; and ObjMC from MotionCtrl for object motion, computed as the average Euclidean distance between estimated and ground-truth 3D Gaussian means over all controlled objects and frames, using the same annotation pipeline to re-estimate trajectories from generated videos.

Why This Matters

Impact on research. The paper argues that 2D control signals are view-dependent and fragile under large camera changes, and that many 3D controls are category-specific, rigid, or tied to reconstruction-heavy pipelines. By making the 4D geometric scene state compact, editable, category-agnostic, and shared across camera and objects, VerseCrafter reframes controllable video generation as conditioning on an explicit world state rather than on projections of it. It also contributes a large real-world dataset (VerseControl4D) in a space where several prior works lean on synthetic game environments with precise annotations.

Real-world applications:

• Content creation and film pre-visualization, where directors specify camera moves and object or character motion in a 3D editor and get realistic video back.
• Robotic simulation and planning, where future observation sequences can be predicted under specified camera and object trajectories.
• Autonomous driving and urban scene simulation, using the outdoor and urban videos in the training pool.
• Interactive world exploration and navigation, supported by the dataset's static-scene samples that train camera-only world exploration.

Industry relevance. The approach keeps a large pretrained 14B video backbone frozen and adds only a small adapter, which is attractive for teams that want geometric controllability without retraining a foundation model. Affiliations include Fudan University, Shanghai Innovation Institute, HKU ARC Lab, and Tencent PCG, and the work is supported by the Shanghai Municipal Science Technology Major Project (2025SHZDZX025G02).

Future Directions

• Extending beyond the current object set. The quality filter keeps clips with 1 to 6 controllable objects and objects under 20% of image area, and the paper notes that Uni3C is limited to single-person motion; a natural question is how far the Gaussian-trajectory representation scales to denser, more crowded scenes.
• Automating trajectory authoring. Trajectories are currently specified or refined by hand through an ellipsoid proxy in a 3D editor; automatic or text-driven trajectory generation would remove a manual step.
• Longer horizons and memory. The paper describes prior work extending temporal horizons with memories or long-sequence models, and VerseCrafter trains and generates at 81 frames — extending to longer sequences is an open direction.
• Reducing inference cost. Generating an 81-frame 720P clip takes about 1152 seconds on 8 96-GB GPUs with about 90 GB peak per-GPU memory, which the authors do not resolve.

Target Audience

Researchers and engineers working on video diffusion models, video world models, controllable video generation, and 4D scene representation. It is most useful to readers already comfortable with diffusion/flow-matching backbones and 3D geometry (point clouds, Gaussians, camera poses), and to practitioners who want a geometric control interface for a frozen pretrained video generator. Readers looking for a beginner-level introduction to video generation will find the methodology sections demanding.

Authors’ abstract

Video world models aim to simulate dynamic, real-world environments, yet existing methods struggle to provide unified and precise control over camera and multi-object motion, as videos inherently capture dynamics in the projected 2D image plane. To bridge this gap, we introduce VerseCrafter, a geometry-driven video world model that generates dynamic, realistic videos from a unified 4D geometric world state. Our approach is centered on a novel 4D Geometric Control representation, which encodes the world state as a static background point cloud and per-object 3D Gaussian trajectories. This representation captures each object's motion path and probabilistic 3D occupancy over time, providing a flexible, category-agnostic alternative to rigid bounding boxes and parametric models. We render 4D Geometric Control into 4D control maps for a pretrained video diffusion model, enabling high-fidelity, view-consistent video generation that faithfully follows the specified dynamics. To enable training at scale, we develop an automatic data engine and construct VerseControl4D, a real-world dataset of 35K training samples with automatically derived prompts and rendered 4D control maps. Extensive experiments show that VerseCrafter achieves superior visual quality and more accurate control over camera and multi-object motion than prior methods.