AEON: Unifying Video and 3D World Models

AI summary

Overview

Research area: Computer vision and generative world modeling, specifically unifying video diffusion models with 3D/4D scene representations.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, variational autoencoders, Gaussian splatting, and distillation-based training.

Scope: AEON is a single framework that generates explicit, persistent 4D Gaussian Splatting worlds from text prompts, images, or videos by operating inside a pretrained video diffusion model's latent space.

What This Paper Is About

World models currently split into two camps: video generative models such as Sora, which produce realistic footage but drift in 3D and cannot be revisited, and 3D generative models such as Marble, which produce geometrically consistent scenes but are often static and less photorealistic. AEON's goal is to produce one representation that has both sets of strengths: photorealistic appearance, correct 3D geometry, modeled motion, scene persistence, and real-time rendering.

The chosen representation is 4D Gaussian Splatting (4DGS), a set of explicit graphics primitives that each carry position, shape, appearance, a reference time, a duration, and translational and angular velocity. The central technical problem the paper solves is how to make a video diffusion model, which normally outputs only pixels, output these 4D primitives instead.

Key Contributions

1. A unified world model built on 4D Gaussians. AEON represents the world as one persistent set of 4DGS primitives rather than a stack of frames or a static 3D scene, and the authors argue this is the first representation to satisfy all five properties they list (realistic, 3D consistent, dynamic, persistent, real-time explorable). Table 1 contrasts this against video world models (e.g., Sora) and 3D world models (e.g., Marble).

2. A 4D Gaussian Reconstruction Model (4DRM). Trained on monocular video, it uses four task-specific heads (depth, camera, Gaussian appearance, evolution) to jointly predict 4DGS attributes and camera poses in a single feed-forward pass, removing the need for external poses or Structure-from-Motion. It is initialized from MapAnything weights and built on a DINOv2-Large encoder.

3. A dual-path domain adapter trained by distillation. It translates latents from a frozen video diffusion model (Wan 2.1 / Wan 2.2) into the reconstructive 4D latent space. It combines a linear branch initialized by closed-form ridge regression with a Perceiver Bridge residual branch, fused through a learnable scalar gate initialized to zero, and injected at transformer layer 4 of the 24-block DINOv2-Large backbone.

4. A three-stage training pipeline and broad task coverage. Stage 1 trains the reconstruction model, Stage 2 performs generative distillation from a frozen teacher into a student plus adapter, and Stage 3 refines end-to-end through the differentiable rasterizer. The resulting system supports text/image/video-to-4DGS generation, 3D world generation, robotics simulation, physical state estimation, scene editing, and novel view and timestep synthesis, and reports state-of-the-art results on the benchmarks used.

Main Findings

• 4D generation: On 30 image-text-prompted scenes scored with VBench, AEON with Wan 2.2 reaches 0.913 I2V Subject, 0.875 Subject Consistency, 0.980 Motion Smoothness, and 0.531 Aesthetic Quality, versus Lyra at 0.865 / 0.864 / 0.974 / 0.459 and 4DNex at 0.796 / 0.790 / 0.968 / 0.342. AEON with Wan 2.1 scores 0.912 / 0.864 / 0.976 / 0.519, showing the method transfers across backbones without architectural change.

• Video-to-4D reconstruction: AEON records the best numbers on both benchmarks, with 14.487 PSNR, 0.421 SSIM, and 0.469 LPIPS on Dycheck, and 24.661 PSNR, 0.771 SSIM, and 0.243 LPIPS on DaVis, beating NeoVerse (14.269 / 0.363 / 0.501 and 23.929 / 0.735 / 0.283), StreamSplat, MoVies, 4DGT, and L4GM. It performs within a small margin of its own teacher 4DRM (14.413 / 0.418 / 0.462 and 24.886 / 0.775 / 0.241) despite working through the video VAE latent bottleneck.

• Lower input requirements than baselines: Unlike 4DGT and MoVies, AEON needs no external poses, and unlike StreamSplat it needs no monocular depth from Depth Anything V2.

• Novel trajectory rendering: Across 30 unseen in-the-wild videos and six VBench metrics, AEON is best on four metrics (Subject Consistency 94.64, Background Consistency 96.77, Aesthetic Quality 69.02, Imaging Quality 58.65) and competitive on the other two against TrajectoryCrafter, RecamMaster, and NeoVerse. The authors attribute this to rendering being exact rasterization of 4DGS rather than learned hallucination.

• Ablations confirm each design choice matters: Replacing the adapter with a plain Conv3D drops Dycheck PSNR from 14.487 to 12.086 and DaVis from 24.661 to 12.813. Removing distillation drops them to 12.390 and 16.718. Disabling the duration/temporal-opacity term drops them to 12.500 and 22.057. The full model is best on every metric.

• Training data strategy: Stage 1 uses six annotated datasets (DynamicReplica, SAILVOS3D, Spring, TartanAir, PointOdyssey, OmniWorld) with resampling factors from 20× to 1000× to offset size differences; Stage 2 adds a 2.4M-scene Panda-70M subset and needs only raw RGB, since the frozen teacher supplies supervision.

Methodology in Plain English

The pipeline has three moving parts and three training stages.

Part one: a model that can turn video into 4D. The researchers first build the 4D Reconstruction Model, which watches a normal monocular video and outputs a set of 4D Gaussians plus the camera positions. Four heads split the work: one predicts depth, one predicts camera intrinsics and pose, one predicts how each Gaussian looks (size, orientation, opacity, color), and one predicts how it behaves over time (when it appears, how long it lasts, how fast it translates and rotates). Depth is unprojected through the predicted camera to place each Gaussian in 3D space, which anchors everything else.

Part two: an adapter that speaks the diffusion model's language. A pretrained video diffusion model already knows how to imagine realistic worlds, but its internal representation is tuned for generating pixels, not for geometry. The adapter converts the diffusion model's latents into the kind of features the reconstruction model expects. It has two paths: a simple linear path whose weights are computed analytically up front as a solid starting point, and a more expressive Perceiver-based path that handles the complicated nonlinear part. A gate multiplies the expressive path by a value that starts at zero, so training begins from the safe linear solution and gradually lets the complex path contribute.

Part three: staging the training. Stage 1 trains the reconstruction model for 20 epochs on video clips with ground-truth poses and depth, using three loss groups: appearance (pixel error plus perceptual and structural terms), geometry (depth and normal error against annotations), and a regularizer that discourages unnecessary motion and short lifespans so the model does not degenerate into per-frame point clouds. Stage 2 freezes that model as a teacher and trains the adapter plus a student copy for 30 epochs, matching the student's Gaussians, dense geometry, and intermediate features to the teacher's. Stage 3 fine-tunes for 10 epochs directly on rendered image quality, comparing rasterized output against frames decoded from the video latent, with a decaying distillation regularizer to keep the geometry learned earlier.

The authors deliberately avoid a simpler alternative: decoding the video latent into frames and feeding those to the reconstruction model. They explain that the reconstruction model was trained on real videos, so VAE and generation artifacts would compound with reconstruction errors.

All experiments ran on 16 NVIDIA H200 GPUs with AdamW, a base learning rate of 5×10⁻⁶, cosine scheduling with 2000 warmup steps, weight decay of 10⁻⁴, and gradient clipping at global norm 1.0.

Why This Matters

For research, the paper reframes the video-versus-3D world model divide as an interoperability problem rather than a competition, and shows a concrete mechanism for extracting explicit geometry out of a frozen video diffusion model without retraining it. It also advances feed-forward monocular video-to-4DGS reconstruction by folding camera pose estimation into the same network.

Real-world applications the paper names or directly enables:

• Embodied AI and robotics simulation: the static component supplies occupancy and free space, dynamic primitives move by their own velocities, and real-time rasterization renders egocentric observations, so an agent can be placed in a generated world and its actions rolled out in a closed loop, replayably.
• Autonomous driving and game design: listed as motivating domains for world modeling generally.
• Scene editing and recomposition: filtering Gaussians by lifespan yields motion masks that cleanly separate static from dynamic content, and parts from different generated worlds can be freely recombined.
• Physical state estimation: depth maps, normal maps, motion masks, and optical flow can be read directly out of the generated 4DGS.

Industry relevance comes from the persistence and real-time exploration properties: a one-shot generated world can be re-entered, rendered from new viewpoints at new timesteps, and reused, which is a different product proposition from a generated video clip that can only be replayed.

Future Directions

• Extending beyond the diffusion model's sequence length. The authors propose a windowed scheme: render conditioning frames from the current 4DGS at the latest camera pose, generate the next window, decode it into new primitives, and fuse where they overlap. They expect drift to accumulate more slowly than in video-only autoregression because each step is anchored to explicit geometry, but this is presented as a path forward, not a demonstrated result.

• Filling unobserved regions. Regions never seen in the generated video cannot be recovered, so large viewpoint changes may expose holes. The authors suggest a generative completion module.

• Measuring the gap to the teacher. AEON trails its own teacher 4DRM slightly on both reconstruction benchmarks, which frames the residual latent-translation loss as a concrete target.

• Scaling on raw video. Because Stage 2 and Stage 3 need no poses or depth, the authors position training on large unannotated corpora as the natural scaling direction; the paper does not report how far this scaling was pushed.

Target Audience

Graduate students and research engineers working on world models, video generation, or 3D/4D reconstruction who want a concrete architecture for coupling a diffusion transformer to an explicit scene representation. It is also useful for practitioners in robotics simulation, autonomous driving, and content creation who need generated environments that are persistent, editable, and renderable in real time. Readers without a background in diffusion latent spaces and Gaussian splatting will find the method sections demanding, though the motivation and comparison tables are accessible.

Authors’ abstract

Current world models for explicit world simulation typically follow two distinct pathways: (1) video generative models (e.g., Sora), which simulate realistic appearances but fail to preserve 3D consistency; and (2) 3D generative models (e.g., Marble), which generate geometrically consistent scenes but often yield static, less realistic observations. We present AEON, a unified framework that bridges video-based and 3D-based world models and combines the benefits of both. The key insight of AEON lies in a generative model that operates within the text- or image-conditioned video latent space while producing renderable 4D graphics primitives---specifically, 4D Gaussian Splatting (4DGS)---as the explicit world representation. We train a 4D reconstruction model to predict both 4DGS and camera poses from monocular videos, serving as our initialized 4D decoder. Furthermore, we design a domain adapter that translates generative video latents into reconstructive 4D latents through a distillation process on the video VAE. AEON naturally supports video-based world modeling by rendering 4DGS at predicted camera trajectories, and constructs a persistent 4D environment with scene dynamics for realistic world simulation. Consequently, AEON empowers a diverse range of world modeling applications and achieves state-of-the-art performance on standard benchmarks. Project Page: https://weiqi-zhang.github.io/AEON