Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction

AI summary

Authors: Fengzhi Guo, Chih-Chuan Hsu, Sihao Ding (Mercedes-Benz Research & Development North America), Cheng Zhang (Texas A&M University) arXiv: 2510.12768v3 [cs.CV], 27 Feb 2026 | Category: Computer Vision | License: CC BY 4.0 Project page: https://tamu-visual-ai.github.io/usplat4d/

Overview

Research area: computer vision, specifically monocular dynamic 3D reconstruction and dynamic 3D Gaussian Splatting. Technical level: Advanced. Scope: The paper introduces USplat4D, an uncertainty-aware dynamic Gaussian Splatting framework that estimates per-Gaussian, per-frame uncertainty and uses it to build a spatio-temporal graph that guides motion optimization for 4D reconstruction from monocular video.

What This Paper Is About

Reconstructing a moving 3D scene from a single moving camera is under-constrained: parts of the scene are occluded at any moment and novel viewpoints are poorly observed, so motion estimates drift and renderings degrade. Existing dynamic Gaussian Splatting methods optimize every Gaussian uniformly, regardless of whether it is well or poorly observed. USplat4D addresses this by estimating how reliable each Gaussian is over time and letting the reliable ones anchor the motion of the unreliable ones.

Key Contributions

1. A method for estimating time-varying per-Gaussian uncertainty: a closed-form variance derived from the photometric loss under a local-minimum assumption, plus a per-pixel convergence indicator that assigns a large constant value to Gaussians whose covered pixels have not converged. This is extended from a scalar to an anisotropic, depth-aware 3D uncertainty matrix propagated through the camera-to-world rotation.
2. An uncertainty-encoded directed graph over Gaussians, where nodes are partitioned into a small key set and a large non-key set according to uncertainty, key nodes are chosen with a two-stage 3D-voxel-gridization and significant-period strategy, and edges are formed with an Uncertainty-Aware kNN (UA-kNN) using a Mahalanobis-style metric that favors reliable connections.
3. An uncertainty-aware optimization scheme with separate losses for key and non-key nodes, where non-key nodes are interpolated from nearby key nodes using Dual Quaternion Blending (DQB); the total loss is L_rgb + L_key + L_non-key. The framework is described as model-agnostic and compatible with any method that estimates per-Gaussian motion.
4. A synthetic Objaverse benchmark for extreme novel views, built because existing benchmarks lack ground-truth for extreme views, using 6 challenging articulated objects and cameras on a circular trajectory of 121 frames (3 degrees per step).

Main Findings

• DyCheck results: On 5 scenes at 1x resolution, USplat4D reaches mPSNR 16.85, mSSIM 0.650, mLPIPS 0.38, compared with SoM (16.72, 0.630, 0.45) and MoBlender (16.79, 0.650, 0.37). Lower-scoring listed baselines include MoDec-GS (15.01, 0.493, 0.44), SC-GS (14.13, 0.477, 0.49), 4DGS (13.42, 0.490, 0.56), and Deformable 3DGS (11.92, 0.490, 0.66).
• DyCheck at 2x resolution: On 7 scenes, USplat4D reaches 19.63 mPSNR, 0.716 mSSIM, 0.25 mLPIPS versus MoSca (19.32, 0.706, 0.26), Gaussian Marbles (16.72, -, 0.41), 4DGS (13.64, -, 0.43), and Dynamic Gaussians (7.29, -, 0.69).
• Objaverse results: Gains over SoM and MoSca are largest at wide viewpoint shifts. Against SoM on the (0°, 60°] range, USplat4D reaches 16.63 PSNR / 0.866 SSIM / 0.27 LPIPS; on (60°, 120°] 16.57 / 0.868 / 0.27; on (120°, 180°] 17.03 / 0.872 / 0.26. Against MoSca, USplat4D reaches 16.22 / 0.885 / 0.22, 15.98 / 0.884 / 0.23, and 16.31 / 0.886 / 0.21 across the same ranges.
• Extreme views are where the method separates itself: The paper states that DyCheck validation views are relatively easy because they are close to the input trajectory, and that the more significant improvements appear under extreme novel viewpoints, which are not in the quantitative table and are evaluated qualitatively (Figure 4 for DyCheck, Figure 5 for DAVIS, Figure 6 for Objaverse).
• Ablations with MoSca as the base model on the DyCheck validation set: Full model 19.63 / 0.716 / 0.25. Removing key node uncertainty drops to 18.86 / 0.688 / 0.28; removing UA-kNN gives 19.50 / 0.711 / 0.26; removing loss weighting gives 19.08 / 0.681 / 0.25; removing 3D gridization gives 19.50 / 0.712 / 0.25.
• Key/non-key ratio: The paper maintains a typical 1:49 key/non-key ratio by selecting the top 2 percent (around 1000-th) most confident Gaussians, and reports ablations over 0.5 percent to 4 percent in the appendix, with 2 percent described as lying on a stable plateau.
• Motion grouping emerges: After reordering key nodes with standard community detection such as spectral clustering, the key-graph weight matrix becomes approximately block-diagonal, and the resulting per-Gaussian motion labels closely match instance segmentation with dynamic tracking.
• Depth-aware uncertainty matters: The paper reports a "Camel" sequence where, without depth-aware uncertainty, the camel's body shrinks unnaturally, whereas the proposed formulation preserves its shape.
• Stated limitation: The framework remains influenced by the computational overhead and inherent errors of the underlying visual foundation models.

Methodology in Plain English

The pipeline has three stages.

First, the method measures how trustworthy each Gaussian is at each frame. Starting from the standard alpha-blending equation for rendered pixel color, the authors differentiate the photometric loss with respect to a Gaussian's color and solve at a local minimum, which produces a closed-form variance expressed as the inverse of the sum of squared blending weights over the pixels that Gaussian touches. That variance becomes the scalar uncertainty. Because the local-minimum assumption does not hold for unconverged pixels, a per-pixel indicator checks whether the color error is below a threshold; if any covered pixel fails, the Gaussian is assigned a large constant uncertainty instead. The scalar is then converted into a 3D anisotropic uncertainty matrix by rotating an axis-aligned diagonal matrix from camera space to world space with the camera-to-world rotation, using separate scaling factors along each axis, so that depth — the least reliable direction in monocular capture — is treated as more uncertain.

Second, that uncertainty is used to build a graph. Gaussians are split into a small set of reliable key nodes and a large set of non-key nodes. Candidate key nodes are found by dividing the scene into a 3D voxel grid per frame, discarding voxels that contain only high-uncertainty Gaussians, and picking one Gaussian per remaining voxel so that spatial coverage is uniform rather than redundant. A candidate is kept as a key node only if its uncertainty stays below a threshold for at least 5 frames, which the paper calls its significant period. Edges among key nodes are formed with UA-kNN at each node's most reliable frame, using a Mahalanobis-style distance that up-weights high-uncertainty directions, so edges connect nodes that are both close and trustworthy. Each non-key node is attached to the key node that stays closest to it over the whole sequence.

Third, optimization is split by node type. Key nodes are pulled toward their pre-trained positions in uncertainty-weighted directions, with motion locality terms (isometry, rigidity, rotation, velocity, acceleration) applied. Non-key nodes are pulled toward both their pre-trained positions and the trajectory interpolated from their key neighbors via Dual Quaternion Blending. Uncertainty appears three times: weighting key-node deviations, guiding non-key interpolation, and balancing the two losses.

Why This Matters

Impact on research: The paper argues that uncertainty should be a central modeling component of dynamic Gaussian Splatting rather than an auxiliary signal. It contrasts with MoSca, which introduces a soft motion score but no structured propagation, and with SE-GS, which estimates self-ensembling uncertainty for a static-scene perturbing strategy. The claim is that this is among the first attempts to model uncertainty and integrate it directly into graph-based motion modeling for dynamic reconstruction.

Real-world applications:

• Augmented reality, where a moving monocular camera must keep virtual content stable on a scene it only partly observes.
• Robotics, where partial and occluded observations are the normal case for perception and control.
• Human motion analysis, where bodies self-occlude and are viewed from restricted viewpoints.
• Content creation and re-viewing, where a viewer picks a camera angle or timestamp far from the original capture path.

Industry relevance: One author group is from Mercedes-Benz Research & Development North America, and the paper lists augmented reality, robotics, and human motion analysis among its motivating tasks, all of which map onto automotive and embodied systems that operate from a single moving camera.

Future Directions

• Reducing the computational overhead introduced by uncertainty estimation and the graph optimization, and reducing dependence on the errors of the underlying visual foundation models, both of which the paper names as current limitations.
• Extending the extreme-novel-view synthetic benchmark beyond the 6 articulated Objaverse objects, since the authors built it precisely because existing benchmarks lack ground-truth for extreme views.
• Expanding uncertainty-aware graph construction to dynamic Gaussian Splatting variants beyond the per-Gaussian motion parameterizations tested, given the framework is described as model-agnostic.
• Turning the emergent motion grouping into an explicit capability: the paper shows the key-graph weight matrix becomes approximately block-diagonal and the labels resemble instance segmentation with dynamic tracking, which suggests further work on segmentation and tracking. The paper also points to challenging cases (textureless regions, fast motion, deforming objects) discussed in Appendix E.

Target Audience

Researchers and graduate students working on 3D Gaussian Splatting, novel view synthesis, monocular dynamic reconstruction, and 4D scene representation. It also suits practitioners in AR, robotics, and human motion analysis who need stable reconstruction under occlusion, and readers interested in how uncertainty estimation can be embedded directly into a learned 3D representation rather than used only as a training heuristic.

Authors’ abstract

Reconstructing dynamic 3D scenes from monocular input is fundamentally under-constrained, with ambiguities arising from occlusion and extreme novel views. While dynamic Gaussian Splatting offers an efficient representation, vanilla models optimize all Gaussian primitives uniformly, ignoring whether they are well or poorly observed. This limitation leads to motion drifts under occlusion and degraded synthesis when extrapolating to unseen views. We argue that uncertainty matters: Gaussians with recurring observations across views and time act as reliable anchors to guide motion, whereas those with limited visibility are treated as less reliable. To this end, we introduce USplat4D, a novel Uncertainty-aware dynamic Gaussian Splatting framework that propagates reliable motion cues to enhance 4D reconstruction. Our approach estimates time-varying per-Gaussian uncertainty and leverages it to construct a spatio-temporal graph for uncertainty-aware optimization. Experiments on diverse real and synthetic datasets show that explicitly modeling uncertainty consistently improves dynamic Gaussian Splatting models, yielding more stable geometry under occlusion and high-quality synthesis at extreme viewpoints.