MCSplat: Multi-View Photometric and Geometric Consistent Feed-Forward Gaussian Splatting for Driving Scenes

AI summary

Overview

• Research area: Computer vision — feed-forward 3D Gaussian Splatting for reconstructing dynamic urban driving scenes from sparse surround-view camera images.
• Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, differentiable rendering, geometry foundation models, bilateral grids, Fisher information, and camera projection geometry.
• Scope (1 sentence): The paper proposes MCSplat, a feed-forward Gaussian splatting framework that separates camera-specific appearance from geometry and adds uncertainty-weighted surface regularization, evaluated on Waymo (798 training / 202 validation scenes) and nuScenes.

What This Paper Is About

Feed-forward Gaussian splatting methods predict 3D Gaussians directly from multi-view driving images using geometry foundation model (GFM) depth priors, but they inherit two problems from per-scene 3DGS optimization. First, cameras differ in exposure, ISP, and illumination, so cross-view appearance differences get wrongly absorbed into Gaussian geometry (e.g., the same roof modeled as separate surfaces, producing floating Gaussians). Second, local geometric supervision via surface normals is unreliable near depth discontinuities, leaving Gaussians weakly constrained off the ego trajectory. MCSplat's goal is to make geometry learning photometrically consistent across cameras and to weight geometric self-supervision by how trustworthy each Gaussian's position estimate is.

Key Contributions

1. Geometry–appearance decoupling framework (MCSplat): A GFM-compatible pipeline that builds a canonical appearance space from depth-guided multi-camera correspondences (reference cameras Front, SideLeft, SideRight; FrontLeft and FrontRight anchored by two neighbors each) and learns Gaussians in that normalized domain, reducing cross-view geometry–appearance ambiguity.
2. Multi-scale Bilateral Grid Transformer (BGT): A lightweight transformer that predicts multi-scale 3D bilateral grids (each grid is 12 × L_m × H_m × W_m, storing flattened 3 × 4 affine color transforms) and applies them coarse-to-fine to map canonical renderings back to camera-specific raw appearance, enabling photometric supervision without contaminating Gaussian geometry.
3. Fisher-information-guided geometric self-supervision: A per-Gaussian scalar uncertainty estimated from the squared gradient norm of the photometric loss with respect to the Gaussian center, used to exponentially down-weight surface-orientation and flatness regularizers on well-constrained Gaussians and up-weight them on weakly constrained ones.
4. State-of-the-art results on the Waymo benchmark (including a 57-scene appearance-challenging subset) plus geometry-oriented evaluations showing improved cross-view depth consistency and pseudo-normal agreement, and generalization results on nuScenes.

Main Findings

• Front-camera reconstruction on Waymo: MCSplat reaches 31.05 dB PSNR, 0.89 SSIM, 0.12 LPIPS versus UniSplat's 28.93 dB / 0.86 / 0.18 — an improvement of +2.12 dB PSNR and −33.3% LPIPS. Novel-view synthesis also improves (27.64 dB / 0.84 / 0.16 versus 27.34 / 0.80 / 0.20). The authors note the front-camera input itself is unchanged, so canonicalizing neighboring views also benefits the reference view.
• Five-camera reconstruction on Waymo: 30.53 dB PSNR, 0.88 SSIM, 0.13 LPIPS versus UniSplat's 29.58 / 0.86 / 0.17; novel-view synthesis 26.35 dB / 0.76 / 0.22 versus 25.98 / 0.76 / 0.24.
• nuScenes generalization: 26.19 dB PSNR (+0.82 dB) and 0.783 SSIM (+0.018), versus UniSplat's 25.37 dB / 0.765, under UniSplat's protocol (middle frame as input; first, middle, last frames as targets).
• Cross-view depth consistency: Mean depth reprojection error in adjacent-camera overlap regions drops from 2.378 m to 0.910 m (−61.73%), and RMSE from 6.427 m to 3.920 m (−39.01%).
• Pseudo-normal consistency (vs. MoGe-2): Mean angular error falls from 115.05° to 92.82°; fraction of pixels below 30° rises from 1.18% to 17.38%; below 11.25° from 0.10% to 10.08%; below 5° from 0.02% to 4.24%.
• Canonical-space Stage-1 training matters: Training Stage 1 on canonical rather than raw images raises PSNR from 29.53 to 30.53 dB, SSIM from 0.85 to 0.88, and lowers LPIPS from 0.19 to 0.13.
• Component ablation: Canonical Appearance Construction (CAC) alone reduces mean depth reprojection error by 59.34% (2.378 → 0.967 m), RMSE from 6.427 to 4.051 m, and normal error from 115.05° to 97.25° (pixels below 30°: 1.18% → 4.08%). Adding the geometric term (Geo) lowers depth errors to 0.910 m mean / 3.920 m RMSE and lifts the below-30° fraction from 4.08% to 17.38%. BGT improves raw-target PSNR from 29.35 to 30.53 dB without changing depth or normal metrics.
• A geometry–photometry trade-off is reported: CAC + BGT without the Geo term attains a slightly higher raw-target PSNR of 30.76 dB than the full model's 30.53 dB, while the full model has better geometry.
• Uncertainty cue choice: Center-based Fisher information maps capture object boundaries (white truck boundary, jeep grille) more distinctly than color- or scale-based maps (Figure 3).
• Qualitative results: Fewer texture-induced geometric artifacts, including inside a yellow road sign comparison, and removal of camera-boundary artifacts in canonical renderings with effective subsequent appearance recovery.
• Not reported: The paper's main text references the 57-scene subset's detailed reconstruction numbers as being in the appendix (Table 6); those specific figures are not included in the provided content.

Methodology in Plain English

1. Make all cameras look alike first (canonical appearance construction). Choose some cameras as anchors (Front, SideLeft, SideRight on Waymo). For each pixel in an anchor view with a valid depth, project it into 3D and back into a neighboring camera, giving a correspondence. For sky pixels where depth is unreliable, intersect the camera ray with a distant ego-centered sky cylinder instead. Fit a per-channel affine color transform (a_c, b_c for R, G, B) between the two views using a robust penalty plus priors that pull the transform toward identity and shrink the offset. Cameras sitting between two anchors get a horizontally interpolated blending of two transforms so there is no hard switch. Sky and non-sky regions use separately estimated transforms.
2. Learn Gaussians in that shared space. The Gaussian prediction pipeline — frozen GFM backbone, sparse 3D convolution module, Gaussian head — is trained against the canonicalized images in Stage 1, using MSE and LPIPS losses plus dynamic-region and scale-alignment auxiliary losses.
3. Recover each camera's real look afterwards. A transformer decoder takes patch tokens from a frozen DINOv2 ViT-Small encoder on both canonical and raw images. At each scale, canonical tokens serve as queries and pooled residual tokens (raw minus canonical) serve as keys/values in cross-attention, which isolates the color change being predicted instead of redundant image content. A per-scale head outputs the bilateral grid, and the slices are composed in homogeneous coordinates to produce the final per-pixel color transform. Coarse grids handle global exposure/color shifts; fine grids handle local variations.
4. Weight geometric supervision by confidence. Run a Charbonnier photometric loss and measure how sensitive it is to nudging each Gaussian's center. Large sensitivity means the images pin the position down well; small sensitivity means the position is uncertain. Normalize these scores per mini-batch and map them through an exponential to get weights, computed online and detached.
5. Apply the weights to surface constraints. For each non-sky Gaussian, take the local surface normal from finite differences over neighboring Gaussian centers, compare it with the Gaussian's own thinnest axis, and penalize the absolute dot product mismatch. Also penalize the thinnest scale directly to encourage flat, surface-like Gaussians. Both losses are weighted averages using the uncertainty weights, so weakly constrained Gaussians get pushed harder.
6. Two-stage training. Stage 2 freezes most of the Gaussian head except the Gaussian MLPs and trains the bilateral grid transformer, adding an image loss, total-variation regularization on the grids, and the uncertainty-aware geometric loss. Input/rendering resolution is 350 × 518; training uses AdamW on 16 NVIDIA H20 GPUs with total batch size 32.

Why This Matters

Practically, the paper shows that the same PSNR can hide very different Gaussian geometry — a model can look good on training views yet produce floating or incoherent surfaces across cameras — so geometry-oriented metrics are needed alongside image metrics. Its decoupling idea is modular: the GFM backbone and Gaussian prediction branch stay unchanged, so the three additions (canonical appearance, bilateral grid transformer, Fisher weighting) could be plugged into other feed-forward driving Gaussian frameworks.

Real-world applications:

• Closed-loop autonomous driving simulation, where novel-view renderings from laterally shifted (cross-lane) camera trajectories must be geometrically sound for planning evaluation.
• Synthetic data generation and sensor/camera rig augmentation for perception training, where consistent geometry across overlapping cameras is essential.
• Fleet-scale scene reconstruction and replay, since feed-forward prediction avoids per-scene optimization, enabling cross-scene generalization.
• Robotics and mapping in surround-view setups, where the same cross-camera photometric inconsistency and sparse off-trajectory views arise.

Industry relevance: the work is co-authored with Zaofu Intelligent Technology Co., Ltd. and Tongji University and was supported by a set of Chinese national and Shanghai municipal grants, indicating direct interest from the autonomous-driving industry in scalable, reconstruction-based simulation.

Future Directions

• Stronger temporal appearance modeling: appearance is canonicalized per frame set; handling appearance change across time within a sequence is left as future work.
• Explicit ISP modeling: the authors list parameterizing ISP factors (exposure, vignetting, camera response) as a next step, extending beyond the current affine and bilateral-grid color transforms.
• Resolving the geometry–photometry trade-off: the ablation shows the full model gives up a small amount of raw-target PSNR (30.53 dB versus 30.76 dB without the geometric term) in exchange for better geometry; how to get both is an open question.
• Extending Fisher-based uncertainty beyond geometry: the paper uses photometric sensitivity only for the Gaussian center; whether the same signal can regularize rotation, scale, or opacity — and whether it generalizes to sparse overlap in non-surround-view rigs — remains unexplored.

Target Audience

Researchers and engineers working on 3D Gaussian Splatting, neural rendering for autonomous driving, and generalizable feed-forward reconstruction; practitioners building simulation or synthetic-data pipelines from surround-view camera rigs; and graduate students with a background in multi-view geometry and differentiable rendering who want a worked example of combining appearance canonicalization with uncertainty-weighted geometric regularization. Readers without that background will find the affine/bilateral-grid and Fisher-information machinery dense.

Authors’ abstract

Driving-scene reconstruction via feed-forward Gaussian splatting from sparse multi-view images remains challenging due to two factors. First, cross-camera photometric inconsistency, caused by variations in exposure, ISP, and illumination, can be incorrectly absorbed into Gaussian geometry. Second, unreliable local geometric supervision arises from normal-based constraints near depth discontinuities. To address these issues, we present \textbf{MCSplat}, a multi-view consistent feed-forward Gaussian splatting framework for driving scenes. MCSplat constructs a canonical appearance space from depth-guided multi-camera correspondences to decouple geometry learning from camera-specific appearance. A multi-scale Bilateral Grid Transformer restores raw camera appearance from canonical renderings, enabling photometric supervision without contaminating Gaussian geometry. To improve local geometric reliability, we estimate Fisher-information-based uncertainty from photometric sensitivity and use it to adaptively weight surface-oriented self-supervision. Extensive experiments demonstrate improved reconstruction quality and generalization across driving datasets. Geometry-oriented evaluations further reveal enhanced cross-view depth consistency and local surface-normal agreement, while qualitative results show fewer geometric artifacts and more coherent structures across overlapping views.