SeeU: Seeing the Unseen World via 4D Dynamics-aware Generation

AI summary

Overview

• Research area: Computer vision — 4D (3D space + time) dynamic scene understanding, novel-view synthesis, and physics-aware video generation.
• Technical level: Advanced. The paper combines 3D Gaussian Splatting, SE(3) rigid motion, B-spline trajectory parameterization, physics-informed losses, and a video diffusion generator (VACE).
• Scope: The paper introduces SeeU, a three-stage 2D → 4D → 2D learning framework that reconstructs a dynamic scene in 4D, learns continuous dynamics of the camera and foreground, and re-projects the evolved 4D world to 2D to generate content at unseen times and viewpoints.

What This Paper Is About

Images and videos are discrete 2D projections of a continuous 4D world, so most generative and predictive models that operate only on pixels lose 3D structure, compress motion, and entangle camera motion with scene motion. SeeU instead reconstructs a unified 4D representation from sparse monocular frames, learns continuous and physically plausible 4D dynamics on a low-rank motion basis, and then rolls the world forward or backward in time and re-projects it to 2D. The goal is to generate visually and physically consistent content for times and viewpoints that were never observed.

Key Contributions

1. A new 2D → 4D → 2D learning framework. The paper's stated core innovation is an information flow in which 2D inputs are lifted to a 4D representation, dynamics are learned in 4D, and 2D content is then generated from the evolved 4D world. The authors note this flow of information and the corresponding learning scheme is new in the literature.
2. The Continuous 4D Dynamics Model (C4DD). A lightweight, physics-aware module that learns control points of B-spline functions to represent shared motion bases and camera trajectories, with a data loss and a physics loss that penalizes second-order translational and rotational accelerations, weighted more heavily in extrapolated time spans.
3. Demonstrated applicability across three task families. SeeU is evaluated on unseen temporal generation (past frame inference, dynamic frame interpolation, future frame prediction), unseen spatial generation (novel camera poses and previously occluded regions), and video editing (object removal, object replacement, time-lapse generation).
4. SeeU45, a curated evaluation set. A set of 45 dynamic scenes assembled from the authors' own captures and public sources, spanning indoor/outdoor environments, diverse subjects, and multiple camera regimes and motion types.

Main Findings

• Unseen temporal generation improves over specialized baselines. On past frame inference, dynamic frame interpolation, and future frame prediction, SeeU reports PSNR / SSIM / LPIPS / C-LPIPS of 20.4682 / 0.5496 / 0.2484 / 0.0355 (past), 21.0726 / 0.5715 / 0.2271 / 0.0327 (interpolation), and 20.5353 / 0.5581 / 0.2431 / 0.0358 (future). Baselines include InterpAny (interpolation: 20.5406 / 0.5636 / 0.2420 / 0.0525), Wan 2.2 (future: 18.2989 / 0.5064 / 0.2559 / 0.0704), Cosmos (future: 20.0672 / 0.5557 / 0.2885 / 0.0355), Shape-of-Motion (past: 15.5464 / 0.4518 / 0.3876 / 0.0116), and VACE (past: 17.1430 / 0.4648 / 0.3673 / 0.0924). The reference C-LPIPS values are 0.0397 (past), 0.0356 (interpolation), and 0.0389 (future); C-LPIPS is not lower-better and should be close to the reference.
• Extrapolation windows are a small fraction of each sequence. The past and future extrapolation windows each cover 6.67% of the sequence duration.
• Temporal error grows with temporal distance. The temporal prediction error analysis spans the past 50% to the future 50% of the time range, and the observed accuracy decrease in extrapolated regions is approximately linear to the temporal distance.
• Unseen spatial generation achieves better geometric and semantic consistency. Across five camera motions (dolly-out, dolly-right, dolly-up, tilt-up, pan-right), SeeU reports lower Epipolar Error, higher Epipolar Inlier Ratio, and higher CLIP-V than ReCamMaster and GCD. For example, on dolly-out SeeU reports EE 0.1997 / EIR 0.7848 / CLIP-V 0.9690 versus ReCamMaster at 0.2382 / 0.6736 / 0.9366 and GCD at 0.2004 / 0.5129 / 0.9253; on pan-right SeeU reports 0.1891 / 0.8229 / 0.9574 versus ReCamMaster at 0.2146 / 0.7035 / 0.9250.
• B-spline parameterization matters. Replacing the B-spline parameterization with plain MLP layers ("C4DD w/ MLP") degrades results to PSNR 17.5372, SSIM 0.3934, LPIPS 0.4274, EE 0.3125, CLIP-V 0.7390, which the authors attribute to reduced temporal smoothness and physical consistency.
• The physics loss stabilizes dynamics. Setting the physics loss weight λ_phy to 0 ("w/o physics loss") yields PSNR 19.3593, SSIM 0.5274, LPIPS 0.2735, EE 0.2240, CLIP-V 0.9198, degrading frame consistency.
• Performance is robust to sparse input. Subsampling inputs to 5 / 10 / 15 / 20 frames produces PSNR 18.3583 / 20.1626 / 20.3870 / 21.0832, SSIM 0.4521 / 0.5378 / 0.5484 / 0.5522, LPIPS 0.3054 / 0.2507 / 0.2408 / 0.2387, EE 0.2852 / 0.2044 / 0.2003 / 0.1972, and CLIP-V 0.9284 / 0.9551 / 0.9581 / 0.9596, indicating stable performance as frame density decreases.
• The framework supports editing operations. Object removal is performed by deleting foreground Gaussians and inpainting revealed background; object replacement conditions on a new text prompt with replaced foreground content; time-lapse is produced by re-parameterizing foreground trajectories in time.

Methodology in Plain English

SeeU runs in three sequential stages.

Stage 1 — 2D to 4D. Starting from a monocular frame sequence, the method estimates camera intrinsics, extrinsics, and per-frame depth using MegaSaM, extracts moving-foreground segmentation masks using Track-Anything, and extracts 2D point tracks using TAPIR. These priors are fused with the RGB frames to build a dynamic scene representation of canonical 3D Gaussians (each with canonical mean, orientation, scale, opacity, and color) that evolve over time through per-frame rigid transformations in SE(3). This stage builds on Shape-of-Motion because it works with casual inputs of limited parallax and explicitly separates static regions from trackable dynamic elements.

Stage 2 — Discrete 4D to Continuous 4D. Because the foreground can contain many Gaussians (for example 80k), learning an independent trajectory per primitive is infeasible. SeeU therefore uses a low-rank motion parameterization in which each Gaussian's properties equal its initial state plus a shared set of global motion basis functions multiplied by a time-invariant coefficient vector per Gaussian. The discrete motion bases are initialized from 3D point trajectories with Procrustes analysis and refined by minimizing photometric reconstruction error. Because both motion-basis and camera-pose trajectories show simple, smooth temporal trends in natural sequences, they are modeled as continuous B-spline functions whose control points are learned jointly under a total loss combining a data term (matching observed discrete motion bases) and a physics term (penalizing second-order translational and rotational accelerations, with higher weights on extrapolated time spans).

Stage 3 — 4D to 2D. The learned dynamics evolve the scene to any timestamp and any camera pose, and the result is rendered as a 2D "video scaffold" encoding geometry, motion, and occlusion structure. The scaffold can be incomplete — never-observed regions, low-confidence projections, and thin structures or sharp depth discontinuities. A spatial-temporal in-context video generator (a computationally efficient fine-tuning of VACE) repairs these frames using three context priors: a structured prompt derived from a vision-language model caption, the projected frames as a geometric and photometric reference (masked pixels set to a constant gray value of 127), and per-frame inpainting masks marking uncertain or missing regions. A Context Encoder turns these priors into context embeddings injected into the pretrained video generator.

Evaluation setup. Experiments use SeeU45 (45 scenes: 10 captured by the authors, 35 from public sources including TAP-vid, I2-2000FPS, AgiBot World, and Animal Kingdom). All stages are trained with the Adam optimizer on a single NVIDIA A100 (80 GB) GPU: Stage 1 uses 80,000 Gaussian primitives for the dynamic foreground and 80,000 for the background, learns 10 motion bases, optimizes for 4,000 iterations, and takes about 1 hour for a typical 10-frame sequence at 960 × 540 resolution; Stage 2 trains C4DD with cubic (deg = 3) B-splines with 8 control points, λ_phy = 1 × 10⁻⁴, learning rate 1 × 10⁻⁵, batch size 64, with 1,000 epochs taking roughly 10 minutes; Stage 3 fine-tuning takes approximately 2 hours. Temporal metrics are PSNR, SSIM, LPIPS, and C-LPIPS; spatial metrics are Epipolar Error, Epipolar Inlier Ratio (computed from SIFT correspondences with cross-check and ratio test, with the fundamental matrix estimated by RANSAC), and CLIP-V.

Why This Matters

Impact on research. The paper argues that modeling dynamics in the native 4D world, rather than in projected 2D pixel or latent space, provides 3D awareness, physical consistency, and explicit disentanglement of camera, background, and foreground motion — three properties the authors identify as missing in end-to-end 2D approaches. It positions itself between reconstruction-based methods (dynamic NeRFs, 3D Gaussian Splatting, and dense 3D point tracking, which lack continuous dynamics modeling) and physics-aware generation methods (which either require manual simulation setup, depend on hand-specified rules, or distill physical cues from pretrained models).

Real-world applications.

• Video editing: object removal, object replacement conditioned on a text prompt, and time-lapse generation.
• Novel-viewpoint and free-viewpoint video: synthesizing content for camera poses not present in the capture, across dolly-out, dolly-right, dolly-up, tilt-up, and pan-right motions.
• Occlusion completion: filling in regions that were previously occluded in the original monocular observations.
• Temporal video repair and prediction: past frame inference, dynamic frame interpolation, and future frame prediction with physically plausible motion.

Industry relevance. The work is supported in part by the United States National Science Foundation (grants 2133032 and 2431505) and a research award from Samsung Research America, indicating interest from both public funding agencies and a consumer-electronics/imaging company. The method's reliance on a single A100 (80 GB) GPU and a Stage 2 training time of roughly 10 minutes suggests the continuous-dynamics component is comparatively cheap once the 4D reconstruction exists. The authors state that all data and code will be made public.

Future Directions

• Reducing dependence on the underlying modules. The stated limitation is that SeeU's applicability is bounded by its tracking, camera estimation, and 4D reconstruction components, so the paper focuses on inputs with pronounced, smooth, and temporally stable foreground motion. Extending to harder inputs is an open problem.
• Broadening the range of supported dynamics. The current scope is limited to smooth, temporally stable foreground motion; sudden disturbances, highly non-rigid motion, and abrupt events are not addressed by the smoothness-oriented B-spline and physics priors.
• Longer-horizon extrapolation. The error analysis shows accuracy decreasing approximately linearly with temporal distance in the extrapolated regions (past and future windows of 6.67% of the sequence each), raising the question of how far the learned 4D dynamics can be reliably rolled forward or backward.
• Integrating 4D dynamics learning with stronger generative backbones. Stage 3 is a fine-tune of VACE; how SeeU-style 4D scaffolds interact with other large-scale video generators is left open.

Target Audience

Researchers and graduate students working on 4D scene reconstruction, dynamic novel-view synthesis, video generation, and physics-aware or world-model research. The paper is also relevant to practitioners in computational photography, AR/VR, and video editing tooling who need to generate plausible content at unobserved times or viewpoints, and who are comfortable with 3D Gaussian Splatting, SE(3) transformations, B-splines, and diffusion-based video generation.

Authors’ abstract

Images and videos are discrete 2D projections of the 4D world (3D space + time). Most visual understanding, prediction, and generation operate directly on 2D observations, leading to suboptimal performance. We propose SeeU, a novel approach that learns the continuous 4D dynamics and generate the unseen visual contents. The principle behind SeeU is a new 2D$\to$4D$\to$2D learning framework. SeeU first reconstructs the 4D world from sparse and monocular 2D frames (2D$\to$4D). It then learns the continuous 4D dynamics on a low-rank representation and physical constraints (discrete 4D$\to$continuous 4D). Finally, SeeU rolls the world forward in time, re-projects it back to 2D at sampled times and viewpoints, and generates unseen regions based on spatial-temporal context awareness (4D$\to$2D). By modeling dynamics in 4D, SeeU achieves continuous and physically-consistent novel visual generation, demonstrating strong potentials in multiple tasks including unseen temporal generation, unseen spatial generation, and video editing. All data and code will be public at https://yuyuanspace.com/SeeU/