Leveraging 2D Priors and SDF Guidance for Dynamic Urban Scene Rendering

AI summary

Overview

Research area: Computer Vision, specifically 3D scene reconstruction and novel view synthesis for dynamic urban driving environments.

Technical level: Advanced — the paper assumes familiarity with 3D Gaussian Splatting (3DGS), Neural Radiance Fields (NeRFs), Signed Distance Functions (SDFs), and multi-view geometry.

Scope: Presents UGSDF (Urban Gaussians via Signed Distance Functions), a method that combines 2D foundation-model priors with a dual SDF + 3DGS representation to render dynamic urban objects without relying on 3D motion annotations, LiDAR-only pipelines, or body templates like SMPL.

What This Paper Is About

Reconstructing and rendering dynamic urban scenes (cars, pedestrians, cyclists, trams) is hard because it normally requires expensive inputs: LiDAR scans, ground-truth 3D bounding boxes, motion tracklets, and hand-crafted templates such as SMPL for humans. This paper asks whether cheap 2D priors — depth from a depth network and point tracks from a point tracker — combined with a hybrid SDF + 3D Gaussian representation can replace those costly 3D requirements. The result is a unified optimization that couples geometry (SDF) and appearance/rendering (3DGS) so each representation cleans up the errors of the other.

Key Contributions

1. 2D-prior-based dynamic object modeling. Motion and 3D structure of dynamic objects are derived from off-the-shelf point trackers (CoTracker) and metric depth networks (UniDepth), eliminating the need for 3D tracklets, bounding boxes, or SMPL templates.

2. SDF-guided improvement of Gaussian distribution. A learned SDF deformation network guides where 3D Gaussians should be densified (near the SDF zero-level set) and pruned (far from it), improving geometry on thin, non-planar objects.

3. Gaussian-guided SDF ray sampling. Rasterized Gaussians are rendered to produce an aligned depth map (via UniDepth), which narrows the SDF's ray-sampling range so that surface reconstruction focuses near the true surface.

4. State-of-the-art results without 3D annotations. Achieves top rendering metrics on KITTI and Waymo/NOTR, sometimes outperforming template-based methods like OmniRe, and works even without LiDAR. Also enables scene editing tasks such as object removal, scene decomposition, and scene composition.

Main Findings

• Outperforms template-based baselines on Waymo/NOTR. UGSDF reaches PSNR 33.98 / SSIM 0.944 on scene reconstruction and PSNR 30.63 / SSIM 0.871 / LPIPS 0.129 on novel view synthesis, beating OmniRe (which uses tracklets and SMPL) without using 3D annotation.

• Strong on KITTI MOT. PSNR 29.55 / SSIM 0.934 for reconstruction and 28.63 / 0.926 / 0.123 for novel view synthesis, improving over OmniRe, 4DGF, and StreetGS by substantial margins.

• Even without LiDAR, results hold. The no-LiDAR variant nearly matches the full pipeline (e.g., 33.88 vs. 33.98 PSNR on reconstruction), showing 2D priors provide most of the geometric signal.

• Vehicles improve more than humans. On the NOTR per-category breakdown, UGSDF leads on vehicles (30.34 PSNR vs. OmniRe's 28.91) but trails slightly on humans (27.89 vs. 28.15), which the authors attribute to SMPL giving OmniRe a structural advantage on non-rigid bodies.

• SDF guidance is the most important component. The SG4GP ablation (removing SDF-guided Gaussian placement) causes the largest drop — 27.89 → 22.47 PSNR for humans and 30.34 → 22.27 for vehicles. Dense representations are essential for thin objects like cyclists and pedestrians.

• Atypical objects render well. Qualitative results show UGSDF handling trams and cyclists that other methods (StreetGS, S3Gaussians, OmniRe) fail to model without floater artifacts.

Methodology in Plain English

The pipeline starts with a sequence of posed RGB images, object masks (produced by prompting SAM2 with points tracked by CoTracker), and depth maps from UniDepth. Masks and tracks are used to lift object pixels into 3D point clouds, which are warped into a canonical frame (typically the first frame where the object is fully visible) to form an initial scaffold.

From this scaffold, two representations are initialized and trained jointly:

• 3D Gaussians in the canonical frame, animated over time by a small set of learnable basis trajectories (with sparsity enforced so only a few bases are used). This motion model is object-agnostic and learned from 2D tracks and depth rather than ground-truth motion.
• A signed distance function built from a multi-resolution feature grid plus two MLPs — a deformation network mapping observed points back to canonical space, and a topology-aware network that lifts points into a higher-dimensional space to handle shape changes.

The two representations feed each other in a loop. The Gaussians render depth, which is used to constrain where along each camera ray the SDF is sampled, focusing computation near the surface. In return, the SDF's zero-level set tells the system where to densify Gaussians (sparse regions near the surface) and where to prune them (points far from the surface, judged over time and using nearest neighbors, not just the point's own SDF value).

Training minimizes photometric and depth losses for both representations, plus standard SDF regularizers: a truncated near-surface loss, a free-space loss, an Eikonal term enforcing unit gradients, and a smoothness loss comparing normals at neighboring surface points. The whole system is optimized end-to-end without any 3D annotation.

Why This Matters

Research impact. The paper demonstrates that 2D foundation models (depth + tracking) plus a hybrid implicit/explicit representation can substitute for expensive 3D supervision in dynamic urban reconstruction. This is a meaningful step toward "annotation-free" scene reconstruction and challenges the assumption that template-based methods (SMPL, bounding-box tracklets) are necessary for high-fidelity dynamic rendering.

Real-world applications:

• Autonomous driving simulation — reconstructing real driving scenes from ordinary camera footage to build safety-critical test scenarios without LiDAR rigs or manual 3D labeling.
• AR/VR scene editing — objects can be removed, decomposed, or inserted (composition) in a captured urban scene, useful for content creation and virtual production.
• Digital twins for cities — building 4D city models from sparse camera data for urban planning and infrastructure monitoring.
• Robotics perception training — synthesizing realistic multi-view dynamic data to train detection and planning systems where ground-truth 3D is scarce.

Industry relevance. Reducing dependence on LiDAR and 3D annotations lowers the cost of deploying scene reconstruction at scale. Fleet companies already collect vast amounts of camera footage; methods like this let them convert that footage into editable 4D scenes with a fraction of the labeling budget.

Future Directions

• Robustness to tracking errors. The authors flag that inaccurate CoTracker tracks lead to poor Gaussian motion estimates; more robust or uncertainty-aware trackers are an obvious next step.
• Replacing the SMPL gap. UGSDF trails OmniRe on human modeling; a stronger learned body prior (or integration with a deformable template without explicit ground-truth) could close this gap.
• Generative priors. The authors suggest incorporating video generative models as additional priors to improve view synthesis under large camera deviations — which currently degrade all methods.
• Joint motion segmentation. Extending the framework to infer motion segmentation alongside rendering could make the pipeline more self-supervised and applicable to more scene types.

Target Audience

Researchers and graduate students working on neural rendering, 3D Gaussian Splatting, NeRFs, and dynamic scene reconstruction. It is also relevant to practitioners building autonomous driving simulators, AR/VR content pipelines, and urban digital twins who want to reduce reliance on LiDAR and 3D annotation. Readers should already be comfortable with multi-view geometry, implicit surface representations, and the basic 3DGS pipeline.

Authors’ abstract

Dynamic scene rendering and reconstruction play a crucial role in computer vision and augmented reality. Recent methods based on 3D Gaussian Splatting (3DGS), have enabled accurate modeling of dynamic urban scenes, but for urban scenes they require both camera and LiDAR data, ground-truth 3D segmentations and motion data in the form of tracklets or pre-defined object templates such as SMPL. In this work, we explore whether a combination of 2D object agnostic priors in the form of depth and point tracking coupled with a signed distance function (SDF) representation for dynamic objects can be used to relax some of these requirements. We present a novel approach that integrates Signed Distance Functions (SDFs) with 3D Gaussian Splatting (3DGS) to create a more robust object representation by harnessing the strengths of both methods. Our unified optimization framework enhances the geometric accuracy of 3D Gaussian splatting and improves deformation modeling within the SDF, resulting in a more adaptable and precise representation. We demonstrate that our method achieves state-of-the-art performance in rendering metrics even without LiDAR data on urban scenes. When incorporating LiDAR, our approach improved further in reconstructing and generating novel views across diverse object categories, without ground-truth 3D motion annotation. Additionally, our method enables various scene editing tasks, including scene decomposition, and scene composition.