Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation

AI summary

Overview

Research area: Computer vision — 4D scene generation, single-image-to-4D synthesis, diffusion models, novel-view synthesis, and 3D point tracking.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, flow matching, variational autoencoders, Gaussian Splatting, and monocular depth estimation.

Scope: The paper introduces MoGe4D, a geometry-conditioned framework that models a scene as dense 4D point trajectories predicted from a single input image, together with a new 60,000-sample training dataset (TrajScene-60K) and a renderer for novel-view video synthesis.

What This Paper Is About

Generating a dynamic, viewable 4D scene from one static photograph is hard because the model must invent plausible future motion while keeping the 3D structure consistent over time and across viewpoints. Existing methods split the problem into two disconnected stages — either generating videos first and reconstructing 4D afterward, or reconstructing a static 3D asset and animating it afterward — which causes geometric drift, artifacts, or overly constrained motion. This paper reworks the second paradigm by tightly coupling geometry and motion: it predicts dense per-pixel 4D point trajectories in a single diffusion process, conditioned on an initial geometric prior inferred from the input image.

Key Contributions

1. TrajScene-60K dataset: A large-scale 4D scene dataset of 60,000 video samples containing dense 4D point cloud trajectories, videos, and text, built to address the scarcity of high-quality scene-level 4D training data.
2. Geometry-conditioned dense-trajectory formulation: A formulation for single-image 4D synthesis that predicts scene trajectories from an initial geometric prior, rather than relying on loosely decoupled generate-then-reconstruct or reconstruct-then-generate stages.
3. 4D-STraG: A diffusion-based 4D Scene Trajectory Generator that predicts geometry-consistent, motion-plausible trajectory fields, using a depth-guided motion normalization strategy to reduce scale ambiguity and a Motion Perception Module (MPM) to inject motion-aware priors.
4. 4D-ViSM: A 4D View Synthesis Module that renders the generated 4D representation into videos under arbitrary camera trajectories, plus extensive experiments showing strong perceptual quality, geometry-aware consistency, and efficient 4D generation.

Main Findings

• Dataset scale: TrajScene-60K provides over 3 million frames and approximately 12 billion 3D point annotations with dense occlusion-aware tracking, per-frame depth, and language descriptions, at 596 × 336 resolution. It was curated from the WebVid-10M corpus, extracting about 200,000 candidates through a two-stage automated filtering pipeline, then retaining 60,000 high-quality samples.
• VBench video quality (Table 1): MoGe4D reports a Dynamic Degree of 1.0000 against 4Real (0.7708) in Group I, with Aesthetic Quality 0.5613 vs 0.4938 and Imaging Quality 0.6230 vs 0.5095. In Group II it reaches Aesthetic 0.4820 and Imaging 0.5828 vs GenXD (0.4077, 0.5209) and DimensionX (0.4634, 0.5545). In Group III it reports Aesthetic 0.4820 and Imaging 0.5939 vs Free4D (0.3607, 0.3562) and Gen3C (0.3812, 0.4814). Against 4Real, MoGe4D's Subject Consistency (0.8752) and Background Consistency (0.9364) are lower than 4Real's (0.9329, 0.9709).
• VLM-based 4D consistency (Table 2): Using Qwen2.5-VL-72B-Instruct on 8 uniformly sampled frames rated 1–5 across five criteria, MoGe4D averages 3.80 (Group I), 3.83 (Group II), and 3.76 (Group III), versus 4Real at 3.35, GenXD at 2.64, DimensionX at 2.80, Free4D at 1.23, and Gen3C at 2.22. The largest reported margins are in Motion-Geometry Coupling and Temporal Texture Stability.
• Geometry-aware errors (Table 3a): MoGe4D reports trajectory errors of 0.058 (vs GenXD 0.236, DimensionX 0.465) and 0.042 (vs Free4D 0.252, Gen3C 0.197), against ground truth 0.000. Reprojection errors are 0.614 and 0.639, versus ground truth 0.301 and 0.284.
• Inference efficiency (Table 3b): On a single NVIDIA A100, MoGe4D synthesizes 49 frames at 512 × 368 in 6 minutes, reported as 5× faster than Free4D (16 frames at 512 × 368 in 30 min). GenXD is faster at 2 minutes but produces 12 frames at 256 × 256. For reference, 4Dfy takes 10 h and 4Real 1.5 h.
• Ablations (Table 4, Group III VBench): Removing depth-guided normalization drops Consistency to 0.8604 and Aesthetic to 0.4672; removing depth latents degrades both Consistency and Dynamic scores; removing MPM reduces Dynamic from 0.9000 to 0.8500; replacing patch-level features with a global [CLS] token slightly improves Consistency (0.8743 vs 0.8702) but degrades Dynamic (0.8840 vs 0.9000) and Aesthetic (0.4754 vs 0.4820). Reducing training data to a 1k subset (0.8567 / 0.8920 / 0.4791) or using unfiltered 60k samples (0.8549 / 0.8850 / 0.4654) both cause drops. Adding random noise during training yields only negligible degradation (0.8685 / 0.8967 / 0.4771). The full model reports 0.8702 / 0.9000 / 0.4820.
• Versus sequential pipelines: Comparing against Wan2.1-I2V followed by either DELTA tracking or VGGT reconstruction, the paper reports that sequential pipelines suffer error accumulation, producing spurious background drift and geometric fragmentation, while conditioning directly on the initial point cloud geometry avoids this.
• Limitations: MoGe4D struggles with extreme photometric conditions such as harsh backlighting and highly unstructured dynamics such as chaotic crowds, because monocular depth estimators degrade and introduce noise into the initial geometric prior.

Methodology in Plain English

The input is one image plus a description. The goal is a set of N = H × W points moving over T frames, which can be rendered from any camera path.

Step 1 — Training data. The authors start from WebVid-10M. CogVLM2 writes a detailed caption for each video, and DeepSeek-V3 keeps only those with clearly countable entities and self-initiated, non-rigid, or articulated motion, discarding clips dominated by crowds, wind, background jitter, or camera motion (egomotion would be confused with real object motion). The DELTA model then extracts occlusion-aware 4D trajectories using monocular depth, and a three-criterion filter removes invalid or anomalous depth values, samples with excessive scene-depth standard deviation, and geometrically inconsistent samples. 60,000 samples survive at 596 × 336; these are rendered from the original viewpoint with Gaussian Splatting, with inpainting masks for void regions.

Step 2 — Generating motion (4D-STraG). The model predicts only relative motion ΔP_t = P_t − P_0, where P_0 is the first frame. Because small 3D motion near the camera looks large on screen and the same motion far away looks tiny, the authors divide each motion component by the viewing-frustum size at that point's initial depth (using α_x = f_x/W and α_y = f_y/H), producing a scale-invariant representation. A lightweight trajectory encoder converts displacements into an RGB motion map, and a trajectory decoder recovers the trajectories. The image, noise, and depth latents are concatenated along the feature dimension and fed to a Diffusion Transformer trained with a flow-matching objective, so the model learns deterministic flows from noise to data.

Step 3 — Injecting motion awareness (MPM). A pretrained motion feature extractor, OmniMAE, produces motion-aware patch-level features from the static image. These drive Motion-aware Adaptive Normalization (MAdaNorm), which generates token-wise scaling and bias parameters that modulate intermediate DiT features — fine-grained, spatially specific control rather than global conditioning.

Step 4 — Rendering novel views (4D-ViSM). Each frame of the moving point cloud is rasterized as a set of 3D Gaussians, and temporally linking them yields a lightweight dynamic renderer that avoids optimizing a full dynamic 4D Gaussian field. Because novel views leave holes, a finetuned Wan2.1 video model fills the missing regions; during training it follows Wan2.1's mask strategy with mask value 0.5 for regions without projected points.

Step 5 — Inference. Depth is estimated with UniDepthv2 (matching the estimator used by DELTA during training), the VAE encodes image and depth, MPM extracts motion features, the DiT generates and decodes relative motion latents, these are de-normalized and fused with the initial point cloud, and 4D-ViSM renders the final video.

Training used Wan2.1-14B with full-parameter training for 4D-STraG (tracking components and VAE decoder for 5k steps, then DiT for 2k steps) and LoRA finetuning of Wan2.1-14B for 10k steps for 4D-ViSM, with AdamW at a learning rate of 2 × 10⁻⁵ on four NVIDIA H20 GPUs.

Why This Matters

Impact on research: The paper argues that coupling geometry and motion at the representation level — dense 4D point trajectories — is more robust than cascading separate generation and reconstruction models, and it supplies a dataset intended to make scene-level 4D generation trainable at scale. The VLM-based scoring protocol plus trajectory and reprojection error metrics also addresses gaps the authors identify in existing 2D-focused benchmarks.

Real-world applications:

• Virtual reality and augmented reality content creation from a single photograph.
• Immersive content production, such as turning a still image into a dynamic scene viewable from chosen camera paths.
• Novel-view video synthesis for scenes where multi-camera capture is impractical.
• Rapid previsualization or scene prototyping where only one reference image exists.

Industry relevance: Generating a 49-frame, 512 × 368 dynamic scene in 6 minutes on a single A100 GPU is framed as practical, and the reported 5× speed advantage over Free4D at equivalent resolution matters for pipelines that need longer-duration synthesis. The code is released at https://github.com/Zhangyr2022/MoGe4D and the project page is https://ivg-yanranzhang.github.io/MoGe4D/.

Future Directions

• Integrating more robust visual foundation models to handle extreme physical conditions such as harsh backlighting, where monocular depth estimation degrades and pollutes the initial geometric prior.
• Extending the approach to highly unstructured dynamics such as chaotic crowds, which the paper identifies as a failure case.
• Developing more universal dynamic priors, and lightweight 4D representations for practical deployment.
• The authors also position the efficiency of the pipeline as enabling longer-duration video synthesis as a direction to pursue.

Target Audience

Researchers and graduate students working on 4D generation, image-to-video diffusion, novel-view synthesis, and dynamic 3D reconstruction; practitioners building AR/VR or immersive content pipelines; and anyone interested in dataset construction for spatiotemporal 3D learning. Readers should be comfortable with diffusion transformers, flow matching, and point-based rendering.

Authors’ abstract

Generating interactive and dynamic 4D scenes from a single static image remains a core challenge. Most existing generate-then-reconstruct and reconstruct-then-generate methods decouple geometry from motion, causing spatiotemporal inconsistencies and poor generalization. To address these, we present MoGe4D (Motion and Geometry-Aware image-to-4D Synthesis), a geometry-conditioned framework for single-image 4D synthesis that models a scene as dense 4D point trajectories. Instead of treating geometry and dynamics as two disconnected stages, our method starts from an initial geometric prior inferred from the input image and predicts future time-varying trajectories in a diffusion process, improving spatiotemporal coherence while preserving structural stability. To support this task, we first introduce TrajScene-60K, a large-scale dataset of 60,000 video samples with dense 4D point trajectories, addressing the scarcity of high-quality training data for scene-level 4D generation. Built on this, our diffusion-based 4D Scene Trajectory Generator (4D-STraG) predicts geometry-consistent and motion-plausible trajectory fields conditioned on the input image, with a depth-guided motion normalization strategy to reduce scale ambiguity and a Motion Perception Module (MPM) to inject motion-aware priors. We further propose a 4D View Synthesis Module (4D-ViSM) to render the generated 4D representation into videos under arbitrary camera trajectories. Experiments show that MoGe4D produces high-quality 4D scenes with strong temporal coherence, favorable geometry-aware consistency, and compelling novel-view synthesis from a single image. Code: https://github.com/Zhangyr2022/MoGe4D.