SMG: Semantic Motion Graph for Monocular Dynamic Gaussian Splatting

AI summary

Overview

Research area: Computer vision, specifically monocular dynamic 3D reconstruction and novel-view synthesis with 3D Gaussian Splatting; also touches 3D point tracking and 3D scene understanding.

Technical level: Advanced. The paper assumes familiarity with Gaussian Splatting, deformation fields, as-rigid-as-possible formulations, dual-quaternion blending, and optimization-based per-scene reconstruction.

Scope in one sentence: The paper proposes Semantic Motion Graph (SMG), a framework that organizes Gaussian motion into a graph of semantically coherent, locally connected nodes so that reliable motion can regularize unreliable regions, and releases a new ego-exo multiview dataset for evaluating dynamic Gaussian Splatting under hard real-world motion.

Affiliation: Cornell University. License: arXiv.org perpetual non-exclusive license (arXiv:2608.31023v1 [cs.CV], 31 Aug 2026). Project page: https://smg-gaussian.github.io/.

What This Paper Is About

Reconstructing a moving 3D scene from a single handheld video is severely under-constrained: the system must separate camera motion from object motion without dense viewpoints, and occluded regions receive almost no supervision. Existing dynamic Gaussian Splatting methods lean on smoothness or local-rigidity assumptions, which break down when a body part or object is hidden and motion must be inferred purely from prior knowledge. The authors' goal is to supply a stronger regularization signal by exploiting the fact that real-world motion follows semantic structure, then to demonstrate robustness on challenging real-world benchmarks plus a newly collected dataset.

Key Contributions

1. SMG pipeline: A monocular dynamic Gaussian Splatting pipeline that models Gaussian motion as the motion of a Semantic Motion Graph rather than as free per-Gaussian deformation. Graph nodes come from lifted 2D tracks, and edges require both spatial proximity (KNN on trajectory distance) and semantic similarity (cosine similarity of DINOv3 features above a threshold τ).

2. Uncertainty-aware motion propagation: Two mechanisms — confidence-aware as-rigid-as-possible (C-ARAP) and local rigid motion control (LRM) — that let reliable graph nodes guide the motion of unreliable or weakly observed nodes within the same semantic group, preserving graph topology and preventing locally inconsistent velocities.

3. A new multiview dataset: The SMG Dataset, collected in an ego-exo setup pairing a moving egocentric camera with a static exocentric camera, designed to stress dynamic Gaussian Splatting with strong camera motion and human–object interaction.

4. Benchmark results and an added capability: State-of-the-art claims for dynamic novel-view synthesis on real-world benchmarks, plus evidence that the same representation supports long-range 3D point tracking.

Main Findings

• Dycheck (iPhone) novel-view synthesis: SMG reports mPSNR 19.54, mSSIM 0.718, mLPIPS 0.250, evaluated on all 7 scenes (Apple, Block, Paper Windmill, Space Out, Spin, Teddy, Wheel) at half resolution with view 0 for training and views 1 and 2 for testing, with metrics computed on the covisible mask. Baselines include MoSca (19.32 / 0.706 / 0.264), the reproduced OriGS (19.43 / 0.695 / 0.281), and the OriGS paper-reported entry "OriGS*" (19.69 / 0.716 / 0.256). Older baselines score lower, for example 4DGS at 13.64 mPSNR and Dynamic Gaussians at 7.29 mPSNR. Against the OriGS* row, SMG's mSSIM (0.718 vs 0.716) and mLPIPS (0.250 vs 0.256) are better while its mPSNR (19.54) is lower; the paper states that SMG outperforms the baselines on all metrics, and notes OriGS did not release training hyperparameters or evaluation code for the iPhone dataset, so the reproduced OriGS result was obtained by modifying MoSca's hyperparameters.

• NVIDIA dataset: SMG reaches PSNR 26.87 and LPIPS 0.068, compared with MoSca (26.72 / 0.070) and DynPoint (26.53 / 0.068), among a long list of baselines (D-NeRF 21.49, HyperNeRF 17.60, 4DGS 21.45, and others). The authors attribute the modest margin to this dataset's narrow baseline, where most motion is fully observed and few regions are under-constrained.

• New SMG Dataset: Comprises 20 scenes total; 4 scenes (backpack, chess, laptop2, exo_ball) are used for quantitative evaluation with post-processed DA3 depth, and the remaining 16 for qualitative evaluation. Averaged over all four scenes, SMG scores PSNR 13.74, SSIM 0.559, LPIPS 0.396 versus MoSca (13.31 / 0.542 / 0.421) and OriGS (13.52 / 0.543 / 0.432). The gap is smallest on exo_ball (small camera baseline, largely front-facing egocentric camera) and larger where viewpoint change and occlusion are heavy. Camera poses are estimated with a PnP solver using calibrated 2D–3D correspondences, and metric depth is predicted by Depth Anything 3 (DA3).

• 3D tracking on Dycheck: On five scenes (Apple, Block, Spin, Teddy, Paper-windmill), SMG reports EPE 0.052 versus MoSca 0.055 and OriGS 0.057; δ3D at 5 cm is 74.1 for SMG versus 73.1 and 71.9; δ3D at 10 cm is 91.6 for SMG versus 89.6 and 89.7.

• Ablation (4 Dycheck scenes: Apple, Spin, Space-out, Wheel): Base vanilla dynamic Gaussian Splatting scores 12.92 PSNR / 0.451 SSIM / 0.598 LPIPS; SMG-only reaches 19.07 / 0.700 / 0.280; without C-ARAP 19.09 / 0.703 / 0.281; without LRM 19.96 / 0.731 / 0.248; full SMG 20.11 / 0.738 / 0.242. LRM alone gives limited improvement on top of SMG but is most effective combined with C-ARAP.

• Qualitative failure modes addressed: On the SMG Dataset, prior methods propagate arm motion onto a static mouse that is invisible in the training view (Laptop2, frames 123 to 281), and collapse object geometry when observations are limited (left hand in Laptop2; laptop and backpack in Backpack). On Dycheck, MoSca and OriGS break object geometry (human arm in Wheel and Spin) and produce floating Gaussians (human hand in Apple). On NVIDIA, motion leakage into the human face in Balloon2 is reduced.

• Implementation specifics: PyTorch on a single NVIDIA H100 GPU; KNN on 3D trajectory distance aggregated over the three co-visible frames with the smallest distances; top-16 nearest neighbors for topology; C-ARAP settings λm = λr = 1.0, κm = κr = 2.5, α = 0.6; semantic gating τ = 0.75 for Dycheck and NVIDIA, τ = 0.85 for SMG-Dataset; DINOv3 as the semantic feature extractor; dual-quaternion blending for motion blending, following MoSca. Training combines RGB, depth, mask, and track losses. Compute support came from the NVIDIA Academic Grant Program using 4× RTX PRO 6000 Blackwell GPUs.

Methodology in Plain English

The method starts from standard off-the-shelf predictions on a monocular video: 2D point tracks, their visibility, depth, and dynamic/static masks. Tracked 2D points are lifted into 3D to form the nodes of a graph.

Nodes are connected only if they are close in 3D trajectory space and share similar semantic features. This dual condition is the core idea: nearby points on different objects (a hand and the mouse it rests on, for example) should not exchange motion, while points on the same object should. Semantic descriptors per trajectory are built by projecting each track into its visible frames and averaging frame-wise features in a top-k cosine-consistent way.

Because both the off-the-shelf priors and the optimization itself are noisy, each node gets a per-frame confidence score built from visibility, how much its velocity deviates from its neighborhood average, and how much its edge lengths change between adjacent frames. These confidences make the graph edges asymmetric: low-confidence nodes are driven by high-confidence neighbors rather than the reverse. An as-rigid-as-possible loss (C-ARAP) then keeps the graph's local structure — relative directions and lengths — consistent over time.

A second loss (LRM) looks at each node's local neighborhood, estimates the single angular velocity that best explains the neighbors' relative motion, and penalizes any residual motion that a rigid twist cannot account for, using a Huber penalty for robustness.

Dynamic Gaussians are back-projected from pixels in dynamic regions with valid depth. Each Gaussian anchors to its nearest node at its reference timestamp and is driven by that node's K nearest semantic neighbors, blended with distance-aware weights via dual-quaternion blending. Gaussian parameters and the graph deformation are optimized jointly under the combined photometric and regularization objective.

Why This Matters

Impact on research. The paper reframes local rigidity — a purely geometric prior used by prior dynamic Gaussian work — as a consequence of semantic coherence, and turns that reframing into a concrete optimization mechanism that propagates motion across occlusions. It also contributes a harder evaluation setup (ego-exo capture) plus evidence that the same representation yields 3D tracks, linking dynamic reconstruction to 3D scene understanding.

Real-world applications:

• Augmented and virtual reality: re-rendering captured real scenes from new viewpoints, where occluded regions and handheld camera motion are the norm.
• Robotics: learning from egocentric human demonstration video, requiring reliable 3D motion of hands and manipulated objects.
• Motion capture and content creation: extracting coherent 3D motion and tracking from ordinary single-camera footage.
• 3D scene understanding and editing: semantic groups with consistent motion support object-level segmentation, editing, and querying of dynamic scenes.

Industry relevance. The pipeline is a test-time, per-scene optimization that consumes existing foundation-model outputs (DINOv3 features, depth, point tracks, flow-based masks), so it improves automatically as those upstream components improve. Its dependence on per-scene optimization, noted as a limitation, is also what makes it currently impractical for real-time or large-scale deployment.

Future Directions

• Reducing dependence on off-the-shelf priors: the authors state that inaccuracies in depth, tracks, and masks may affect performance, and that SMG naturally improves as these upstream predictions become more accurate.
• Scaling beyond per-scene optimization: the paper points to feed-forward 4D Gaussian Splatting as a promising direction for more efficient and flexible dynamic reconstruction.
• Extending the semantic motion graph to broader 3D tasks: the authors suggest the representation can benefit scene understanding such as 3D object segmentation and tracking, beyond novel-view synthesis.
• Evaluating on harder dynamics: the SMG Dataset is used quantitatively on only 4 of its 20 scenes (with post-processed DA3 depth), leaving the remaining 16 for qualitative use, and Dycheck dynamics are dominated by simple object rotations with relatively mild camera motion — both suggest room for more rigorous benchmarking under complex motion.

Target Audience

Researchers and graduate students working on dynamic 3D reconstruction, Gaussian Splatting, neural rendering, and monocular video-based novel-view synthesis; practitioners building AR/VR, robotics, or motion-capture systems that need 3D scene dynamics from a single camera; and researchers in 3D scene understanding interested in semantically structured motion representations and in the new ego-exo multiview dataset. Readers need a solid background in 3D representations and optimization to follow the method sections.

Authors’ abstract

We study dynamic Gaussian Splatting from monocular videos. While recent advancements in dynamic Gaussian splatting offer a promising foundation for modeling dynamic scenes, they often overfit to the training views and fail under occlusion or complex scene motion due to the lack of reliable regularization signals in under-constrained regions. We propose Semantic Motion Graph (SMG), a novel approach models the Gaussian motion as the low-rank semantic motion. Our key insight is that the real-world scene motion is often structured by semantic coherence: regions that are spatially close and semantically related tend to exhibit consistent dynamics. To leverage this prior, we construct SMG to model structured motion of the scene. The Gaussian motion is driven by the motion of SMG nodes. We further observe that the uncertainty of Gaussian motion arises from both unreliable off-the-shelf priors and weakly constrained regions during optimization. SMG addresses this by using reliable graph nodes to guide the motion of nearby unreliable nodes. To evaluate dynamic Gaussian splatting under challenging real-world scenarios, we introduce a new multiview dataset collected under an ego-exo setup. Extensive experiments demonstrate that SMG achieves state-of-the-art performance on monocular dynamic Gaussian splatting across challenging real-world benchmarks. Project page: https://smg-gaussian.github.io/.