Sparse4DGS: 4D Gaussian Splatting for Sparse-Frame Dynamic Scene Reconstruction

AI summary

Overview

• Research area: Computer vision / graphics — dynamic (4D) scene reconstruction and novel view synthesis from 2D images, built on 3D Gaussian Splatting.
• Technical level: Advanced. It assumes familiarity with Gaussian Splatting, deformation fields, NeRF-style volume rendering, and Stochastic Gradient Langevin Dynamics.
• Scope: The paper introduces Sparse4DGS, a texture-aware 4D Gaussian Splatting framework that reconstructs dynamic scenes from sparse-frame (low-frame-count) video inputs rather than dense video sequences.

What This Paper Is About

Existing dynamic Gaussian Splatting methods can produce photorealistic 4D reconstructions, but they assume dense-frame video as input; with only sparse frames available (e.g., due to device limitations such as low-FPS capture), these methods degrade badly, particularly in texture-rich regions where geometry collapses in both the canonical and deformed spaces. Sparse4DGS is proposed as the first method targeting sparse-frame dynamic scene reconstruction, aiming to keep high-fidelity rendering quality when only a limited set of frames is available for training.

Key Contributions

1. The authors demonstrate that high-quality dynamic scenes can be reconstructed from sparse-frame video sequences, and state this is the first work to focus on sparse-frame 4D scene reconstruction.
2. They frame the sparse-input optimization problem in terms of high-frequency texture information, and propose a Texture Intensity (TI) Gaussian Field that embeds per-Gaussian texture richness, supervised by a Pearson Correlation Coefficient (PCC) loss rather than L1.
3. They propose Texture-Aware Deformation Regularization (TADR), a texture-based depth alignment loss that aligns the texture intensity of rendered depth with the texture intensity of depth predicted by a mono-depth estimator (DPT).
4. They propose Texture-Aware Canonical Optimization (TACO), which reformulates canonical Gaussian gradient descent using Stochastic Gradient Langevin Dynamics with a texture-aware noise term, plus an opacity-based noise term.
5. They introduce the iPhone-4D dataset (four real-world scenes captured on an iPhone at 30 FPS) and report results across NeRF-Synthetic, NeRF-DS, Hyper-NeRF, and iPhone-4D.

Main Findings

• Failure mode identified: Under sparse-frame settings, existing dynamic reconstruction methods fail in both canonical and deformed spaces, with the degradation concentrated in texture-rich, high-frequency regions; the paper notes geometry tends to collapse there.
• Quantitative gains on NeRF-Synthetic (20 frames): Sparse4DGS reaches PSNR 25.31, SSIM 0.944, LPIPS 0.056, versus Deformable3DGS (22.65 / 0.927 / 0.073), 4DGaussians (22.47 / 0.931 / 0.071), and CoRGS (20.15 / 0.920 / 0.089).
• NeRF-DS (20 frames): Sparse4DGS reaches 22.34 / 0.801 / 0.233, versus 20.81 / 0.753 / 0.301 (Deformable3DGS), 19.70 / 0.697 / 0.350 (4DGaussians), and 19.86 / 0.746 / 0.319 (CoRGS).
• Hyper-NeRF (30 frames): Sparse4DGS reaches 23.91 / 0.711 / 0.294, versus 22.41 / 0.661 / 0.295 (Deformable3DGS), 20.64 / 0.637 / 0.414 (4DGaussians), and 20.50 / 0.638 / 0.364 (CoRGS).
• iPhone-4D at 30 FPS: Sparse4DGS reaches 29.81 / 0.929 / 0.177, versus 28.79 / 0.896 / 0.246 (4DGaussians), 27.01 / 0.909 / 0.210 (Deformable3DGS), and 21.58 / 0.851 / 0.266 (CoRGS).
• iPhone-4D at 5 FPS — the largest reported margin: Sparse4DGS reaches 27.51 / 0.910 / 0.205, versus 21.12 / 0.817 / 0.299 (Deformable3DGS), 16.81 / 0.737 / 0.374 (CoRGS), and 16.37 / 0.678 / 0.421 (4DGaussians).
• Overall accuracy claim: For most datasets, the method improves PSNR by more than 1 dB over baselines, and it consistently achieves the best performance on the three publicly available datasets.
• Ablation — module importance (NeRF-DS, 20 input views): Removing TADR causes a 0.45 PSNR drop (21.89) and removing TACO causes a 1.01 PSNR drop (21.33), against the full method's 22.34. The no-module baseline is 20.81 / 0.753 / 0.301.
• Ablation — loss weights: Both λ1 (for L_tex) and λ2 (for L_tadr) peak at 0.01 (PSNR 22.34 / SSIM 0.801 / LPIPS 0.233). For λ1 = 0.001, 0.1, and 1 the PSNR is 21.80, 21.79, and 20.17 respectively; for λ2 = 0.001, 0.1, and 1 the PSNR is 22.04, 22.12, and 21.86.
• Ablation — noise terms in TACO: Removing ε_o gives 21.81 / 0.792 / 0.246; removing ε_tex gives 21.57 / 0.783 / 0.260; both are below the full method's 22.34 / 0.801 / 0.233. Without ε_tex, Gaussians fail to concentrate in texture-rich areas; ε_o helps converge ambiguous, overly opaque Gaussians.
• Ablation — PCC versus L1: Replacing PCC in L_tex with L1 drops performance by 0.6 (21.71 / 0.789 / 0.245), and replacing PCC in L_tadr with L1 drops it by 0.3 (22.09 / 0.797 / 0.239). The stated reason is that L1 overlooks spatial inconsistencies in multi-view texture and depth maps.
• Ablation — texture-aware depth loss: Replacing the texture-aware depth loss with a conventional depth loss (PCC directly between rendered depth and DPT depth) gives 21.46 / 0.775 / 0.277, below the full method.
• Qualitative observations: CoRGS synthesizes reasonable novel views and frames from sparse inputs but fails to handle dynamic objects; Deformable3DGS and 4DGaussians degrade notably around object edges. On the "MacBook" case in iPhone-4D, Sparse4DGS reconstructs a clear Apple logo while baselines are blurry.
• Novel-view and novel-frame evaluation: Evaluation sets combine novel viewpoints (spatial interpolation) and novel temporal frames, using the same evaluation sets as prior work.

Methodology in Plain English

The method keeps the standard dynamic Gaussian Splatting pipeline — a canonical field of 3D Gaussians plus a deformation network (MLP) that offsets each Gaussian's position, scale, and rotation at a given time — and adds a texture signal to guide optimization when frames are scarce.

First, the authors compute a per-pixel gradient magnitude from each input image using horizontal and vertical Sobel operators, producing a 2D "texture intensity" map that measures local texture richness. They add a Texture Intensity attribute to each Gaussian and render it through the same differentiable rasterizer that renders color. Because the texture extractor is applied independently per image, spatial inconsistencies can appear, so the rendered texture map is matched to the ground-truth texture map using a Pearson Correlation Coefficient loss (L_tex = 1 − PCC) rather than L1.

Second, for the deformation network they extract texture intensity from the rendered depth map and from a depth map predicted by the DPT mono-depth estimator, then align the two using the same PCC formulation (L_tadr). This is a local, texture-based geometric constraint intended to capture local depth variation better than image-level depth regularization. The overall training loss is the RGB loss plus λ1·L_tex plus λ2·L_tadr, with both weights set to 0.01.

Third, for the canonical Gaussians they change how gradient updates are applied. Instead of plain stochastic gradient descent, they use a Stochastic Gradient Langevin Dynamics update that adds random noise each iteration: an opacity-based noise term (inherited from prior work) plus a new texture-aware noise term that shrinks as a Gaussian's TI value grows. Gaussians in low-texture areas keep receiving noise, which keeps perturbing their optimization until they settle in texture-rich regions, where TI approaches one and the texture noise approaches zero. The sigmoid-based transition uses k = 100 and t = 0.995, the same hyperparameters used for the opacity term since TI and opacity share the same value range.

Experiments compare against Deformable3DGS (CVPR24), 4DGaussians (CVPR24), and CoRGS (ECCV24), run without modifying their original settings, using PSNR, SSIM, and LPIPS. Training frames are uniformly sampled: 20, 30, and 40 frames for NeRF-Synthetic (8 synthetic scenes) and NeRF-DS (7 real-world scenes captured with two monocular cameras), 10, 20, and 30 frames for Hyper-NeRF (5 of its 17 real-world scenes selected), and 5 FPS versus 30 FPS inputs for the new iPhone-4D dataset (four real-world scenes captured on an iPhone at 30 FPS).

Why This Matters

• Research impact: It opens a distinct problem setting — sparse-frame dynamic reconstruction — that differs from static few-shot Gaussian Splatting, and it shows that texture/high-frequency supervision can substitute for the temporal information missing from sparse inputs. The ablation results quantify how much each proposed component matters.
• Real-world applications:
  • Augmented and virtual reality content creation, which the paper cites as the main motivation for photorealistic 4D scene modeling.
  • Consumer capture with equipment constraints or low-FPS cameras, where dense video is not available.
  • iPhone-based 4D capture pipelines, which the authors explicitly target with the iPhone-4D dataset at 5 FPS and 30 FPS.
  • Related domains the paper cites as benefiting from dynamic Gaussian Splatting: surgical scene reconstruction, dynamic scene stylization, and deblurring of monocular video.
• Industry relevance: Real-time rendering speed is a core appeal of Gaussian Splatting, and lowering the input frame requirement reduces capture cost and bandwidth for production pipelines (VR/AR, mobile capture, content generation) where dense multi-view video is impractical.

Future Directions

• Testing how far sparsity can be pushed beyond the evaluated settings — the paper samples 20/30/40 frames (NeRF-Synthetic, NeRF-DS), 10/20/30 frames (Hyper-NeRF), and 5 FPS versus 30 FPS (iPhone-4D), so behavior at still-lower frame counts is an open question.
• Reducing dependence on the mono-depth estimator (DPT), since TADR's depth supervision is anchored to its predicted depth maps and the paper itself notes image-level depth regularization has limitations in capturing local depth variation.
• Extending evaluation to more scenes: only 5 of Hyper-NeRF's 17 scenes were selected, iPhone-4D has four scenes, and the authors state that detailed quantitative and qualitative results are deferred to supplementary materials.
• Combining the approach with the feed-forward/generative Gaussian prediction methods and few-shot static techniques discussed in related work, which the paper does not experiment with.

Target Audience

Researchers and practitioners in computer vision and graphics working on novel view synthesis, 4D/dynamic scene reconstruction, and Gaussian Splatting, plus engineers building AR/VR or mobile capture pipelines who need high-quality reconstruction from limited frames. Readers should be comfortable with Gaussian Splatting, deformation fields, and stochastic optimization; the paper is not an introductory treatment.

Authors’ abstract

Dynamic Gaussian Splatting approaches have achieved remarkable performance for 4D scene reconstruction. However, these approaches rely on dense-frame video sequences for photorealistic reconstruction. In real-world scenarios, due to equipment constraints, sometimes only sparse frames are accessible. In this paper, we propose Sparse4DGS, the first method for sparse-frame dynamic scene reconstruction. We observe that dynamic reconstruction methods fail in both canonical and deformed spaces under sparse-frame settings, especially in areas with high texture richness. Sparse4DGS tackles this challenge by focusing on texture-rich areas. For the deformation network, we propose Texture-Aware Deformation Regularization, which introduces a texture-based depth alignment loss to regulate Gaussian deformation. For the canonical Gaussian field, we introduce Texture-Aware Canonical Optimization, which incorporates texture-based noise into the gradient descent process of canonical Gaussians. Extensive experiments show that when taking sparse frames as inputs, our method outperforms existing dynamic or few-shot techniques on NeRF-Synthetic, HyperNeRF, NeRF-DS, and our iPhone-4D datasets.