Flux4D: Flow-based Unsupervised 4D Reconstruction

AI summary

Overview

• Research area: Computer vision, specifically 4D (3D + time) reconstruction of large-scale dynamic driving scenes from camera and LiDAR data, with a focus on unsupervised and generalizable feed-forward models.
• Technical level: Advanced.
• Scope: The paper introduces Flux4D, a flow-based unsupervised framework that predicts 3D Gaussians and their velocities directly from unlabeled multi-sensor observations, and evaluates it on novel view synthesis, scene flow estimation, future prediction, and controllable camera simulation across PandaSet, Waymo Open Dataset, and Argoverse 2.

What This Paper Is About

Reconstructing a moving 4D world from vehicle sensor data is expensive because methods such as NeRF and 3D Gaussian Splatting require human annotations (3D tracklets or dynamic masks) to separate moving actors from the static scene, and they optimize a separate model for every scene, which takes hours. Existing self-supervised alternatives remove the annotations but need complex regularization, foundation-model priors, slow per-scene optimization, and are sensitive to hyperparameters. Flux4D's goal is to learn the static/dynamic decomposition directly from raw data by training a single feed-forward network across many diverse scenes, using only photometric losses plus an "as static as possible" regularization.

Key Contributions

1. An unsupervised, generalizable 4D reconstruction framework. Flux4D predicts 3D Gaussians and their motion dynamics from multi-sensor observations without any annotations, pre-trained supervised models, or foundational priors, supervised solely by photometric and static-preference losses.
2. A minimalist learning strategy that replaces hand-designed regularization with data scale. Training across many scenes acts as implicit regularization, allowing the network to automatically decompose static and dynamic elements where prior self-supervised per-scene methods require geometric constraints, cycle consistency, multi-stage training, or semantic features.
3. A LiDAR-initialized, 3D-consistent representation that scales. By initializing Gaussians from LiDAR points, Flux4D processes a large number (≥60) of high-resolution (1080px) input multi-view images and reconstructs dynamic scenes in seconds, rather than the sparse, low-resolution inputs used by prior generalizable methods.
4. Two enhancements producing the final model. Iterative refinement driven by 3D gradients and a polynomial motion model with pixel-wise velocity reweighting together produce Flux4D from Flux4D-base, improving appearance, geometry, and flow.

Main Findings

• Novel view synthesis on PandaSet (1s snippets, interpolation): Flux4D reaches 21.99 PSNR / 0.662 SSIM and 1.63 depth MAE on dynamic regions, and 23.84 PSNR / 0.675 SSIM / 1.07 depth MAE on full images, with 0.157 velocity RMSE. This beats unsupervised baselines such as EmerNeRF (17.79 PSNR dynamic-only), DeSiRe-GS (19.08), STORM (17.65), DepthSplat (16.87), and L4GM (17.36), and approaches supervised methods that use ground-truth 3D tracklets (NeuRAD 23.01, StreetGS 20.06, G3R 21.85 on dynamic regions).
• Speed: Flux4D reconstructs in 1.8s on a single RTX A5000 GPU (24GB), versus roughly 60 minutes for NeuRAD, roughly 28 minutes for StreetGS, roughly 100 minutes for EmerNeRF, and roughly 120 minutes for DeSiRe-GS. Flux4D-base runs in 0.13s, and generalizable baselines STORM (0.08s) and L4GM (0.32s) are faster still.
• Long-horizon reconstruction (8-second logs): Flux4D achieves 21.94 PSNR / 0.658 SSIM / 1.57 depth MAE on dynamic regions and 23.72 PSNR / 0.670 SSIM on full images, outperforming unsupervised per-scene methods EmerNeRF (18.65) and DeSiRe-GS (19.76) and again competitive with supervised methods (NeuRAD 22.99, StreetGS 21.63, G3R 20.60).
• Novel view synthesis on Waymo Open Dataset: Flux4D reports 26.62 PSNR / 0.82 SSIM / 0.18 LPIPS on full images, 26.86 / 0.83 on dynamic regions, and 26.09 / 0.80 on static regions, surpassing DrivingRecon (20.63 / 0.61 / 0.21 full image) by +5.99 dB PSNR and +0.21 SSIM, and well above LGM, PixelSplat, MVSplat, and L4GM.
• Future prediction: Extrapolating beyond observed frames, Flux4D reaches 19.07 PSNR / 0.489 SSIM / 2.57 depth MAE on dynamic regions with 0.162 velocity RMSE, outperforming unsupervised methods (EmerNeRF 15.40, DeSiRe-GS 15.69, STORM 16.61, Flux4D-base 17.88) and even the supervised baselines in this setting (NeuRAD 18.63, StreetGS 16.35, G3R 18.93).
• Scene flow estimation, despite not being designed for it: On PandaSet, Flux4D achieves 0.094 EPE3D, 0.775 Acc₅, 0.807 Acc₁₀, 0.123 angular error, and 0.165 EPE-3way at 0.31 s/frame, beating NSFP (0.183 EPE3D, 5.57 s/frame), FastNSF (0.194, 0.68 s/frame), and STORM (0.120 EPE3D, 0.01 s/frame). In bucketed evaluation it is strongest on smaller or less common categories such as wheeled VRUs (0.011 static, 0.866 dynamic) and other vehicles (0.021 static, 0.661 dynamic).
• Ablation of model components (dynamic-only): Starting from Flux4D-base at 18.62 PSNR / 0.454 SSIM / 2.06 depth MAE / 0.181 velocity RMSE, adding iterative refinement gives 21.32 / 0.636 / 1.66 / 0.167, and adding polynomial motion gives 21.45 / 0.641 / 1.55 / 0.167.
• Ablation of training strategy (dynamic-only): Full Flux4D is 21.99 / 0.662 / 1.63 / 0.157. Removing velocity reweighting drops to 21.45 / 0.641 / 1.55 / 0.167, and removing velocity regularization drops to 21.08 / 0.614 / 1.44 while velocity RMSE degrades sharply to 0.532, showing the static-preference prior is essential for accurate flow.
• LiDAR-free operation: Using an off-the-shelf monocular depth model, Flux4D reaches 21.71 PSNR / 0.668 SSIM on dynamic regions with EPE-3way 0.165, comparable to its LiDAR version (21.99 / 0.662, EPE-3way 0.165). Combining LiDAR and monocular-depth-lifted points gives the best full-image realism at 24.55 PSNR / 0.726 SSIM and EPE-3way 0.161.
• Scaling behavior: Analysis on PandaSet and WOD shows consistent improvements in photometric accuracy and motion estimation as the number of training scenes increases, whereas per-scene methods need explicit regularization or pre-trained models.
• A limitation acknowledged by the authors: Flow estimation for highly dynamic actors with complex motion patterns remains challenging; iterative processing of long-horizon logs creates visible inconsistencies at transition points; and the method assumes a simple pinhole camera model with clean LiDAR data.

Methodology in Plain English

Flux4D represents a scene as a set of 3D Gaussians, each carrying the usual attributes (center position, scale, orientation, color, opacity) plus two extra fields: a learnable instantaneous velocity and a fixed capture time. Positions are initialized from LiDAR points in each source frame, scales are set from the average distance to nearby points, colors are assigned by projecting the points into the corresponding camera image, and velocities start at zero. Extra random points on a distant spherical plane model sky and far regions, and random points within a 3D sphere increase robustness.

A neural network — a 3D U-Net with sparse convolutions — takes these initial Gaussians and their timestamps and outputs refined Gaussian parameters and predicted velocities. Each Gaussian is then moved from its capture time to any target time using a simple linear motion model (position plus velocity times elapsed time), rasterized into color and depth images, and compared against the real camera and LiDAR observations. The total loss is a reconstruction term (L1 plus structural similarity on images, plus an L1 depth loss against projected LiDAR) plus a velocity regularization term that penalizes motion magnitude, encouraging the scene to stay "as static as possible." This is trained across many scenes with weights of 0.8 for RGB, 0.2 for SSIM, 0.01 for depth, and 5e-3 for velocity regularization, for 30,000 iterations on 4× NVIDIA L40S (48G) GPUs, taking about two days.

Two refinements turn Flux4D-base into the final Flux4D. Iterative refinement, inspired by G3R, feeds back the rendered images and the computed 3D gradients of the Gaussians through a second network to correct color inconsistencies and sharpen detail within as few as two iterations. Motion enhancement replaces the single constant velocity with a polynomial velocity parameterization to capture acceleration, braking, and turning, and reweights the photometric loss pixel-wise by rendered flow magnitude (clipped to [0, 10]) so faster-moving, fewer-pixel regions matter more during training.

Why This Matters

Impact on research. The paper argues that a data-driven prior learned across many scenes can replace the elaborate regularization, geometric constraints, cycle consistency, multi-stage training, and external semantic features that prior self-supervised reconstruction methods depend on. It also shows that reconstruction supervision alone can be competitive on standard scene flow benchmarks, suggesting a path to unifying scene flow estimation and reconstruction in one framework.

Real-world applications:

• Scalable simulation environments for autonomous driving, where sensor data is turned into editable, photorealistic 4D scenes instead of being hand-authored.
• Controllable camera simulation, including actor removal, insertion, and manipulation, enabled by Flux4D's explicit 3D representation and instance mask extraction.
• Future-frame sensor prediction, described as analogous to next-token prediction but applied to dynamic 3D scenes, useful for forecasting and planning.
• Novel view rendering at high resolution (≥1080px) for validation of perception and planning systems.

Industry relevance. The work comes from Waabi with University of Toronto and UIUC, and it targets a concrete industrial bottleneck: annotation cost and per-scene optimization time. Reconstructing in 1.8s without labels or pre-trained vision models, and improving as training data grows, makes it plausible to process large volumes of unlabeled driving logs, which is directly relevant to simulation and autonomy development pipelines.

Future Directions

• Scaling to larger and more diverse datasets to address the difficulty of estimating flow for highly dynamic actors with complex motion patterns.
• A unified temporal representation for seamless long-term reconstruction, since the current iterative approach for long-horizon logs produces visible inconsistencies at transition points.
• Improved robustness to real-world sensor imperfections, since the method currently assumes a pinhole camera model and clean LiDAR, limiting use with rolling shutter cameras or noisy inputs.
• More advanced velocity models or implicit flow representations, which the authors call an exciting direction beyond the polynomial motion parameterization explored here.

Target Audience

Researchers and practitioners in computer vision, autonomous driving, and neural rendering who are interested in unsupervised or self-supervised 4D scene reconstruction, generalizable feed-forward reconstruction models, 3D Gaussian Splatting, scene flow estimation, and sensor simulation. It is most valuable to readers already familiar with NeRF, 3DGS, and differentiable rendering, though the high-level framing of the problem and its results is accessible to a broader technically literate audience.

Authors’ abstract

Reconstructing large-scale dynamic scenes from visual observations is a fundamental challenge in computer vision, with critical implications for robotics and autonomous systems. While recent differentiable rendering methods such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have achieved impressive photorealistic reconstruction, they suffer from scalability limitations and require annotations to decouple actor motion. Existing self-supervised methods attempt to eliminate explicit annotations by leveraging motion cues and geometric priors, yet they remain constrained by per-scene optimization and sensitivity to hyperparameter tuning. In this paper, we introduce Flux4D, a simple and scalable framework for 4D reconstruction of large-scale dynamic scenes. Flux4D directly predicts 3D Gaussians and their motion dynamics to reconstruct sensor observations in a fully unsupervised manner. By adopting only photometric losses and enforcing an "as static as possible" regularization, Flux4D learns to decompose dynamic elements directly from raw data without requiring pre-trained supervised models or foundational priors simply by training across many scenes. Our approach enables efficient reconstruction of dynamic scenes within seconds, scales effectively to large datasets, and generalizes well to unseen environments, including rare and unknown objects. Experiments on outdoor driving datasets show Flux4D significantly outperforms existing methods in scalability, generalization, and reconstruction quality.