Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models

AI summary

Overview

Research area: Computer vision, specifically generative 3D/4D scene synthesis from images and video, combining video diffusion models with explicit 3D Gaussian Splatting representations.

Technical level: Advanced. The paper assumes familiarity with diffusion models, latent-space transformers, 3D Gaussian Splatting, and differentiable rendering.

Scope: The paper proposes Diff4Splat, a feed-forward framework that turns a single image, a specified camera trajectory, and an optional text prompt into an explicit, renderable dynamic 4D scene in about 30 seconds without per-scene optimization. (The submitted header titles the work "Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models"; the paper content itself is titled "Diff4Splat: Repurposing Video Diffusion Models for Dynamic Scene Generation.")

What This Paper Is About

Generating a dynamic 3D scene from one image normally forces a choice between two bad options: a slow multi-stage pipeline that generates a video first and then spends GPU-hours reconstructing 3D geometry, or a fast feed-forward model that only produces flat 2D video frames with no explicit geometry or motion. Diff4Splat closes that gap by training a video diffusion model to directly predict a deformable 3D Gaussian field in a single forward pass, so appearance, geometry, and motion all come out of one network.

Key Contributions

1. A new paradigm for dynamic 3D scene generation. The diffusion model is trained to synthesize a deformable 3D Gaussian field directly in a single forward pass, rather than generating video and then lifting it to 3D, resolving the tension between feed-forward efficiency and explicit 3D representation.
2. A Video Latent Transformer architecture. The paper designs a latent reconstruction model (called the Latent Dynamic Reconstruction Model, LDRM, in the methodology) that bridges the representational gap between 2D latent features and a structured 4D representation, predicting a deformable 3D Gaussian field from video diffusion latents conditioned on camera and temporal embeddings.
3. A large-scale 4D dataset. The authors built a data pipeline that annotates both synthetic and real-world videos with metric-scale geometry and motion labels, amassing approximately 130,000 high-quality 4D training scenes, intended to be released for future research.
4. An explicit representation with extra capabilities and speed. The explicit 3D Gaussian representation enables deterministic rendering from a given camera path, depth rasterization, and real-time interaction, while collapsing the generation-plus-reconstruction pipeline into roughly 30 seconds — described as a 60-fold acceleration over state-of-the-art optimization-based methods.

Main Findings

• Speed. Diff4Splat generates a dynamic scene in approximately 30 seconds on inference, with training taking roughly 7 days on 32 A100 GPUs in BF16 mixed precision over 100,000 iterations. The paper contrasts this with optimization-based pipelines that require 18 minutes (AC3D + Shape of Motion), 35 minutes (AC3D + SaV), and 45 minutes (AC3D + Mosca), and notes that methods such as DimensionX demand several GPU hours for a single video while Mosca is described in the introduction as requiring half an hour.

• Appearance and aesthetic quality (Table 1). Diff4Splat records FVD 210.153, KVD 2.316, CLIP-Score 23.123, CLIP-Aesthetic 5.231, and QA-Quality 2.813. It achieves the best FVD, CLIP-Score, and CLIP-Aesthetic in that table, while AC3D + Mosca leads on KVD (2.012) and QA-Quality (2.842). For reference, CameraCtrl scores FVD 478.192 and AC3D scores FVD 339.431, and both run in 20s and 28s respectively.

• Geometric integrity (Table 2). Diff4Splat attains 5114.22 average matches (MASt3R local correspondences), 88.32 subject consistency, and 89.89 background consistency in 30 seconds. It leads on average matches and subject consistency; AC3D + Mosca is ahead on background consistency at 90.43, and CameraCtrl trails at 2015.82 matches.

• Camera pose fidelity (Table 3). Measured by Average Relative Pose Error, Diff4Splat achieves 0.012 translation error and 0.008 rotation error, versus 3.001 and 0.810 for the implicit AC3D baseline. The explicit representation additionally supports novel view synthesis, depth, rasterization, and real-time interaction, none of which the AC3D baseline provides.

• Motion loss matters (Table 4). Removing the motion loss degrades FVD from 210.153 to 351.382, KVD from 2.316 to 3.351, QA-Quality from 2.813 to 2.145, average matches from 5114.22 to 4821.56, subject consistency from 88.32 to 82.45, and background consistency from 89.89 to 85.12, at the same 30-second reconstruction time.

• Deformation Gaussian field matters. Without the deformation module, the model struggles to distinguish camera movement from foreground object motion, producing ghosting artifacts, motion blur, spike artifacts, and degraded image quality, especially in frames with large motion.

• Progressive training matters. Training directly on dynamic scenes without static pre-training fails to initialize the 3DGS, causing unstable training and degraded 4D quality, or requires triple the training time (21 versus 7 days) for a comparable baseline.

• Training data. The curated 4D dataset combines seven synthetic sources (TartanAir, MatrixCity, PointOdyssey, DynamicReplica, Spring, VKITTI2, MultiCamVideo) and two real-world sources (RealEstate10K, Stereo4D). Scale statistics reported in the supplementary table include Stereo4D at 74K scenes and 14.8M frames, MultiCamVideo at 14K scenes and 11M frames, and RealEstate10K at 70K scenes and 6.36M frames. Stereo4D is the only listed dataset that is dynamic, has depth, has tracking, and is real.

• Metric scale recovery. Because real-world datasets such as RealEstate10K provide only relative camera poses from COLMAP, the authors re-estimate metric depth and camera extrinsics using Video Depth Anything and MegaSaM, anchoring relative depth to sparse metric-scale points. They report that the estimated scale factor shows a variance of less than 5% across different subsets of anchor points.

• Evaluation protocol. The camera-controllability benchmark is an evaluation set of 160 samples, built by applying five camera trajectories (spiral, forward, backward, upward, and downward) to 32 unique text-captioned scenes.

Methodology in Plain English

The system starts from a pretrained video diffusion transformer (CogVideoX, operating in the latent space of a 3D causal VAE with a 32×4×8×8 compression scheme, 32 blocks, and hidden dimensionality 4096) that is conditioned on the input image and on Plücker-embedding camera poses, and optionally on a text prompt.

Instead of stopping at video frames, the authors attach a 16-block transformer (the Latent Dynamic Reconstruction Model) that reads the diffusion model's latent tensor, converts the latents into tokens with a 2×2 patch size, concatenates matching camera-pose tokens, and interprets the result as a dynamic point cloud. A lightweight decoder then regresses the parameters of 3D Gaussians — position, scale, rotation quaternion, opacity, and spherical-harmonic color — and a 3D deconvolutional layer maps those attributes back onto source video pixels.

Motion is handled by predicting a per-Gaussian deformation at every timestep: a 3-channel displacement of the mean, a 4-channel rotation delta applied by quaternion multiplication, and a 3-channel scale adjustment, for a total deformation dimensionality of 10. Gaussians with predicted opacity below 0.005 are pruned before rendering.

Supervision combines four terms: a flow-matching loss applied only to the base video diffusion model to align its latent space with the 4D-annotated data, a photometric loss (MSE plus LPIPS with weight 0.5), a geometric loss based on depth correlation plus a total-variation smoothness term, and a motion loss using point tracks (ground truth for synthetic data, CoTracker tracks filtered by confidence for real data). Weights are 1.0 for photometric, 0.5 for geometric, and 2.0 for motion.

Training is progressive across three stages: 40K iterations of static geometry pre-training at 256×256 with the deformation module frozen, then 40K iterations of high-resolution refinement at 512×512 still with the deformation module frozen, then 20K iterations of dynamic fine-tuning with the whole model unfrozen and the full loss including motion. Text guidance is injected through cross-attention in each DiT block using image embeddings from a T5 model, and at sampling time a cosine-based dynamic classifier-free-guidance schedule is used with N = 30 inference steps and a maximum guidance scale of 7.5.

Why This Matters

The paper's central claim is that explicit 4D scene generation no longer needs to be slow. By folding generation and representation into one feed-forward pass, it makes dynamic 3D content creation practical on a timeframe (around 30 seconds) that two-stage pipelines of 18 to 45 minutes cannot match, and it produces a representation that can be rendered deterministically from a chosen camera path rather than an implicit one that can only be sampled.

Potential real-world applications, drawn from what the paper itself demonstrates or states:

• Novel view synthesis and free-viewpoint rendering of dynamic scenes.
• Depth map extraction from generated 4D content.
• Real-time rendering and interactive exploration of generated scenes.
• Immersive content creation, virtual reality experiences, robotics, and simulation, which the introduction names as domains with transformative potential.

Industry relevance: The explicit Gaussian representation is directly compatible with existing real-time Gaussian splatting renderers, and the roughly 30-second turnaround changes the economics of content pipelines that previously had to budget minutes or hours of GPU time per scene. The collected 4D dataset, the source code, and pretrained models are stated to be released upon publication, lowering the barrier for groups that cannot afford to build a comparable metric-scale 4D annotation pipeline themselves.

Future Directions

• Reconciling the metric trade-offs. Diff4Splat leads on FVD, CLIP-Score, CLIP-Aesthetic, average matches, and subject consistency, but AC3D + Mosca still leads on KVD, QA-Quality, and background consistency. Closing those remaining gaps is an open question the results raise.
• Extending beyond the evaluated camera trajectories. The benchmark uses five trajectories (spiral, forward, backward, upward, downward) applied to 32 scenes; whether the model generalizes to more complex, longer, or user-driven camera paths is not reported.
• Broadening comparison to feed-forward 4D methods. The paper notes that many recent feed-forward 4D generation methods are not open-sourced or have different input requirements, precluding direct fair comparison; a common benchmark would clarify where this approach stands.
• Dataset and scaling limits. The training set of approximately 130,000 scenes includes nine datasets with differing dynamic, multi-camera, depth, tracking, and real-world properties. How much each source contributes, and whether the approach scales with more real dynamic data (Stereo4D being the only listed real, dynamic, tracked, depth-annotated source), is left open.

Target Audience

Researchers and practitioners in generative 3D/4D vision, video diffusion modeling, and neural rendering will get the most from this paper, particularly those working on feed-forward reconstruction, camera-controllable video generation, or Gaussian splatting pipelines. It is also relevant to graphics and simulation engineers evaluating whether diffusion-based scene generation is fast and explicit enough to embed in interactive or production content workflows. Readers without grounding in diffusion models, transformers, and differentiable rendering will find the methodology sections demanding.

Authors’ abstract

We introduce Diff4Splat, a feed-forward method that synthesizes controllable and explicit 4D scenes from a single image. Our approach unifies the generative priors of video diffusion models with geometry and motion constraints learned from large-scale 4D datasets. Given a single input image, a camera trajectory, and an optional text prompt, Diff4Splat directly predicts a deformable 3D Gaussian field that encodes appearance, geometry, and motion, all in a single forward pass, without test-time optimization or post-hoc refinement. At the core of our framework lies a video latent transformer, which augments video diffusion models to jointly capture spatio-temporal dependencies and predict time-varying 3D Gaussian primitives. Training is guided by objectives on appearance fidelity, geometric accuracy, and motion consistency, enabling Diff4Splat to synthesize high-quality 4D scenes in 30 seconds. We demonstrate the effectiveness of Diff4Splat across video generation, novel view synthesis, and geometry extraction, where it matches or surpasses optimization-based methods for dynamic scene synthesis while being significantly more efficient.