Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles

AI summary

Overview

Research area: Computer vision for autonomous driving — joint 3D perception (depth, object detection, semantic occupancy) and feed-forward neural scene reconstruction (3D Gaussian Splatting) from multi-camera driving video.

Technical level: Advanced. The paper assumes familiarity with transformer backbones, camera/rig pose parameterizations in SE(3), DETR-style query decoding, deformable attention, and 3D Gaussian Splatting.

Scope: A single feed-forward multi-camera video model with a shared backbone and four task decoders, plus a long-sequence inference pipeline, an annotation pipeline, and a new 1,253-clip benchmark (PAI–Foundation4D).

What This Paper Is About

Autonomous driving systems need to both understand a recorded scene (how far away surfaces are, where other vehicles are, which space is occupied) and rebuild it so it can be re-rendered from new viewpoints for simulation. Today these are usually separate networks that reprocess the same camera images repeatedly. Foundation4D asks whether one shared model can produce depth, camera and ego motion, 3D boxes with velocities, semantic occupancy, and renderable 3D Gaussians from the same multi-camera video, in a single pass over a temporal window.

Key Contributions

1. A unified feed-forward model. Foundation4D shares multi-camera video features across task-specific decoders for geometry (depth, pointmaps, camera-to-rig calibration, ego motion), 3D detection with velocities, semantic occupancy, and Gaussian reconstruction. A factorized spatiotemporal attention scheme replaces VGGT-style global attention over all images, alternating within-image attention with cross-camera and cross-time attention.

2. A video-level inference pipeline. Window-level predictions are aligned into scene geometry and camera/object trajectories over long recordings. It uses pose graph optimization (PGO) with a motion prior for the trajectory and an offline, two-pass association scheme (Kalman filter plus Hungarian matching, followed by track reconnection and Rauch–Tung–Striebel smoothing) for object tracks.

3. A data annotation pipeline. It accounts for sensor timing, ego and actor motion, asynchronous sensing, rolling-shutter exposure, and visibility to generate dense depth images and semantic occupancy labels from driving recordings.

4. The PAI–Foundation4D benchmark. 1,253 clips from the Physical AI Autonomous Vehicles (PAI) dataset, with dense depth and semantic occupancy labels plus documented evaluation protocols so geometry, detection, and occupancy can be compared on the same driving scenes.

Main Findings

• Best depth accuracy among compared methods. With benchmark-specific fine-tuning, Foundation4D achieves the lowest depth AbsRel on both Waymo and nuScenes. The absolute AbsRel values are not reported in the provided text; the paper states only the ranking.

• Best detection and occupancy among compared methods. It reports the highest detection mAP on nuScenes and the highest occupancy mIoU on OpenOccupancy relative to the compared methods (again, the numeric values are not given in the excerpt).

• Joint training helps, but asymmetrically. Co-training detection together with geometry and occupancy improves top-class mAP by 1.2 points over detection-only training. In the other direction, co-training geometry with detection reduces camera-to-rig translation error by 13.4% relative to geometry-only training.

• Longer temporal context helps detection without retraining, i.e. the model can exploit more frames at inference time than it was trained with.

• Pose graph optimization beats sequential stitching. On PAI, PGO with a motion prior improves camera trajectory accuracy over sequential stitching (reported in Tables 9 and 4 of the paper).

• Predicted poses are nearly as good as ground-truth poses for rendering. Using predicted ego poses preserves overall Gaussian rendering quality close to what is obtained with ground-truth ego poses (Table 15).

• Predicted geometry and tracks are usable priors downstream. They serve as priors for lidarless NuRec, supporting reconstruction of driving scenes for simulation without ground-truth poses or tracks.

Methodology in Plain English

Shared perception backbone. Each camera image is encoded independently by a DINOv2 ViT-L/14 encoder with feature width 1024. A 1036×1036 image yields a 74×74 patch grid, which is bilinearly downsampled to 37×37 before aggregation, so the aggregator handles a quarter of the spatial tokens while the fine encoder features stay available to the detection and occupancy pyramid. Instead of one global attention operation over every image token, each of the 16 aggregation layers alternates within-image self-attention with attention either across cameras at a fixed time or across time for a fixed camera. Alongside the patch tokens, each image carries a rig token (for that camera's fixed mounting transform) and an ego token (for the vehicle's motion), plus four register tokens.

Geometry. Camera pose is factorized into a fixed rig transform per camera and a time-varying ego trajectory for the reference camera. Because each image keeps its own capture time, the ego trajectory is interpolated at that time (linear for translation, shortest-arc spherical for rotation) and composed with the rig transform. This matters for asynchronous cameras. DPT decoders output metric depth, an auxiliary pointmap, and 21-class per-pixel semantic logits at a 518×518 grid; the final pointmap is built by unprojecting predicted depth through the calibrated camera model rather than read directly from the auxiliary decoder. Supervision combines robust trimmed regression with a confidence-weighted term, pose losses in world-to-camera encoding, a relative-motion loss over nearby temporal pairs, and auxiliary semantic losses.

Detection. PETR-style queries anchored at learned 3D reference points predict class scores and cuboids (center, dimensions, yaw, planar velocity) over a region of interest of [-108,150] × [-108,108] × [-3,5] m, using 900 queries and six decoder layers of width 256. A feature pyramid shared with the occupancy branch fuses fine image features with the aggregator's contextual features. Training uses Hungarian matching, focal classification, query denoising, and an auxiliary YOLOX-style 2D detection head that is used only during training. The internal label set has 12 classes.

Occupancy. Rather than a dense voxel volume, intermediate features stay in bird's-eye view: a Deformable-DETR-style projector places four reference points along each BEV pillar and samples multi-scale image features. Output heads then reshape channels into height, producing binary and semantic logits on a 0.2 m voxel grid over [-20,100] × [-30,30] × [-3,5] m. This keeps memory low enough to run occupancy alongside the other tasks in one model.

Gaussian reconstruction. Following Instant NuRec, the scene is modeled with static Gaussians, actor-associated Gaussians, and a sky cubemap. Per source pixel the model predicts center, color, scale, quaternion, opacity, and a surface normal for auxiliary supervision. Crucially, at inference the branch consumes the model's own predicted camera poses and assembled actor tracks instead of ground-truth ones; it does not predict actor motion itself.

Long-sequence handling. A full recording is processed in windows; pose graph optimization with a motion prior refines a shared trajectory and the corrections are propagated to each camera at its own capture time, while detections are linked into identity-consistent tracks across windows.

Why This Matters

The paper argues that perception and simulation reconstruction should not be separate pipelines: both need geometry, objects, motion, and occupied space, so a shared backbone avoids reprocessing the same video several times. Making Gaussian reconstruction work from predicted poses and tracks is the practically important step, because ground-truth poses and box tracks are not available on real fleet data.

Real-world applications:

• Closed-loop simulation for autonomy. Re-rendering recorded drives from novel viewpoints to test planners against rare or dangerous scenarios.
• Lidarless scene reconstruction. The predicted geometry and trajectories act as priors for lidarless NuRec, so drives captured with cameras only can still be turned into renderable scenes.
• Fleet-scale data engine. The annotation pipeline turns raw multi-camera recordings into dense depth and occupancy labels, feeding training for vehicle programs.
• Online perception stacks. Depth, detection, and occupancy from one backbone reduce compute versus running separate networks per task.

Industry relevance: The work comes from NVIDIA and is explicitly framed around production concerns — memory-bounded occupancy representations, streaming-friendly windowed inference, multi-dataset training with missing annotations, and calibration estimated rather than assumed. The released benchmark and annotation pipeline aim to give the community a common measuring stick, which matters for comparing proprietary driving stacks.

Future Directions

• Closing the remaining gap between predicted and ground-truth poses for rendering. Rendering with predicted ego poses is close to, but not equal to, ground-truth quality (Table 15); further gains in pose accuracy or joint pose-reconstruction refinement are an obvious target.

• Extending temporal context and streaming. Longer context already improves detection without retraining, which raises questions about how far this scales and how to keep inference incremental over hour-long recordings rather than fixed windows.

• Better handling of genuinely asynchronous capture. The authors note that the rig factor is treated as fixed and that interpolating the ego trajectory does not correct a rig supervision target that may itself contain ego motion between exposures. This mismatch is left open.

• Predicting actor motion inside the reconstruction branch. The Gaussian branch currently consumes boxes and tracks but does not model actor motion itself, so dynamic-actor rendering depends on upstream tracking quality.

Target Audience

Researchers and engineers working on autonomous driving perception, multi-task learning, feed-forward 3D reconstruction, and neural simulation. It is most useful to readers already comfortable with transformer-based multi-view detectors and Gaussian Splatting who want to see how far a single shared backbone can be pushed across perception and rendering, and who need a benchmark with dense depth and occupancy labels on shared driving clips. Readers new to the area will find the related-work section a useful map, but the method sections are dense and assume background knowledge.

Authors’ abstract

We introduce Foundation4D, a unified feed-forward model for geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. A shared backbone uses factorized spatiotemporal attention to efficiently combine observations across cameras and time. Our video inference pipeline uses pose graph optimization to refine camera trajectories and assembles object tracks across input windows. We also develop a data annotation pipeline and release PAI-Foundation4D, a benchmark with dense depth and semantic occupancy labels for 1,253 driving clips. With benchmark-specific fine-tuning, Foundation4D achieves the lowest depth AbsRel on Waymo and nuScenes, the highest detection mAP on nuScenes, and the highest occupancy mIoU on OpenOccupancy among the compared methods. Our task-combination study shows benefits for both detection and geometry: co-training detection with geometry and occupancy improves top-class mAP by 1.2 points over detection-only training, while co-training geometry with detection reduces camera-to-rig translation error by $13.4\%$ relative to geometry-only training. We further use the predicted geometry and trajectories as priors for lidarless NuRec, supporting reconstruction of driving scenes for simulation.