Dynamic Gaussian Splatting from Defocused and Motion-blurred Monocular Videos

AI summary

Overview

Research area: Computer vision — novel view synthesis, 4D (dynamic) scene reconstruction, 3D Gaussian Splatting, and image/video deblurring.

Technical level: Advanced. The paper builds directly on 3D Gaussian Splatting, NeRF-based deblurring, SE(3) motion bases, and depth/tracking priors, and assumes familiarity with differentiable rasterization and volumetric rendering.

Scope: The paper presents a single framework that reconstructs sharp, renderable dynamic 3D Gaussian scenes from monocular videos that suffer from either defocus blur or motion blur (and, in extension experiments, both at once).

Authors Xuankai Zhang, Junjin Xiao and Qing Zhang are affiliated with Sun Yat-sen University. The paper is arXiv:2510.10691v3 [cs.CV] (31 Oct 2025). Code is available at https://github.com/hhhddddddd/dydeblur and results at https://dydeblur.github.io/.

What This Paper Is About

Existing dynamic 3D reconstruction methods assume sharp input images, and existing deblurring-based reconstruction methods are built for only one blur type: defocus blur or motion blur. Because the two blurs form through fundamentally different physical processes, no prior method handles both while producing high-quality novel views.

This paper's goal is a unified pipeline that removes both blur types from monocular video and simultaneously learns a dynamic Gaussian scene representation, so that sharp novel views can be synthesized from any camera position and time.

Key Contributions

1. A unified framework for dynamic Gaussian Splatting from both defocused and motion-blurred monocular videos. The authors state this is, to their knowledge, the first attempt in this field, achieved by modeling both blur types as blur-kernel-based convolution.

2. A Blur Prediction Network (BP-Net) with a blur-aware sparsity constraint. BP-Net simultaneously predicts a per-pixel blur kernel and a per-pixel blur intensity, using camera information (learnable per-view embedding), scene information (rendered image, depth, motion mask), and pixel positional encoding.

3. A dynamic Gaussian densification (DGD) strategy. Dynamic Gaussians are initialized from only the visible 2D tracking points in the canonical frame; after a fixed number of iterations, dynamic regions are re-projected from all observation frames with depth maps and mapped back to the canonical frame via foreground remapping, to fill incomplete dynamic regions.

4. An unseen-view constrained optimization scheme. Appearance information for synthesized unseen views is used periodically to regularize scene optimization, with parallel-unseen views (interpolated between adjacent training views) and perpendicular-unseen views (perturbation of [0.5, 1] normalized units along a local perpendicular direction).

Main Findings

• Defocus blur results (D2RF dataset, 8 dynamic scenes): the method reaches PSNR 29.39, SSIM 0.859, LPIPS 0.078, versus D3DGS (22.54 / 0.715 / 0.215), SoM (28.32 / 0.784 / 0.164), D2RF (27.04 / 0.808 / 0.128), DyBluRF (26.24 / 0.788 / 0.159) and De4DGS (28.49 / 0.791 / 0.154).

• Motion blur results (DyBluRF dataset, 6 scenes): the method reaches PSNR 27.01, SSIM 0.876, LPIPS 0.056, versus D3DGS (21.54 / 0.675 / 0.287), SoM (26.21 / 0.823 / 0.109), D2RF (23.67 / 0.745 / 0.120), DyBluRF (24.53 / 0.864 / 0.079) and De4DGS (26.62 / 0.871 / 0.059).

• Beats deblur-then-reconstruct pipelines: feeding deblurred images from BSSTNet into D3DGS gives 24.42 / 0.723 / 0.179 (defocus) and 21.72 / 0.653 / 0.279 (motion); feeding them into SoM gives 28.56 / 0.786 / 0.164 and 26.33 / 0.825 / 0.105 — both below the proposed method. The authors attribute this to video deblurring methods not preserving 3D scene consistency.

• Speed and cost: training takes approximately 1 hour on an NVIDIA RTX 3090 at 512×288, with rendering at 65.143 FPS at the same resolution. Reported parameters are 192.2M, compared with 42.6M (D3DGS, 10 mins), 164.2M (SoM, 10 mins), 2.7M (D2RF, 48 hrs), 1.3M (DyBluRF, 48 hrs) and 754.6M (De4DGS, 20 hrs).

• Ablation — every component contributes: removing the sparsity constraint gives 29.03 / 0.842 / 0.086 (defocus) and 26.63 / 0.854 / 0.072 (motion); removing the BP-Net shortcut gives 29.12 / 0.845 / 0.086 and 26.74 / 0.852 / 0.078; removing dynamic Gaussian densification gives 29.19 / 0.843 / 0.085 and 26.53 / 0.847 / 0.109; removing unseen-view information gives 29.09 / 0.836 / 0.090 and 26.66 / 0.853 / 0.075. Full method: 29.39 / 0.859 / 0.078 and 27.01 / 0.876 / 0.056.

• BP-Net beats alternative blur models: substituting blur modeling from De3DGS yields 28.31 / 0.812 / 0.098 (defocus) and 26.05 / 0.823 / 0.118 (motion); from De4DGS yields 28.63 / 0.829 / 0.094 and 26.74 / 0.859 / 0.060; from D2RF yields 28.96 / 0.832 / 0.094 and 26.30 / 0.825 / 0.109.

• Kernel size effect: on the Gate and Dock scenes of D2RF and the Skating and Man scenes of DyBluRF, PSNR rises from K=5 (27.84 defocus / 29.53 motion) to K=9 (28.29 / 30.01) but plateaus at K=11 (28.30 / 30.02) and K=13 (28.31 / 30.04). K=9 was chosen to balance performance and computational cost.

• Joint blur and extra datasets: on the authors' D2RF-v2 (defocus, Shop and Car scenes) and DyBluRF-v2 (motion, Man and Seesaw scenes) datasets, the method achieves 30.26 / 0.885 / 0.062 and 28.55 / 0.859 / 0.064, above D3DGS, SoM, D2RF, DyBluRF and De4DGS. On the Deblur-NeRF dataset the method reaches 24.22 / 0.768 / 0.095 (defocus) and 27.14 / 0.835 / 0.096 (motion) versus De3DGS at 23.71 / 0.747 / 0.110 and 26.61 / 0.822 / 0.108.

• Deblurring comparison: on training views, the method reaches 34.85 / 0.977 / 0.027 (defocus) and 33.45 / 0.960 / 0.036 (motion), exceeding D2RF, De4DGS, BAGS and De3DGS, and described as comparable to the state-of-the-art video deblurring method BSSTNet (33.54 / 0.961 / 0.039 and 33.71 / 0.965 / 0.030).

• Stated failure modes: the method relies on 2D image priors, so depth estimation and segmentation errors can degrade results; it fails on dynamic scenes with large non-rigid motion blur (as do other state-of-the-art methods); and, like vanilla 3DGS, it must be optimized per scene separately.

Methodology in Plain English

The framework starts from Shape-of-Motion, which splits the scene into static Gaussians (background) and dynamic Gaussians (moving objects) and describes motion as a weighted combination of SE(3) motion bases. Dynamic Gaussians are seeded by reprojecting the depth of 2D tracking points (from Depth-Anything and TAPIR) into 3D in a canonical frame; scene motion up to 20 such bases is blended with a per-Gaussian motion coefficient.

The key idea for blur is that both defocus blur and motion blur can be approximated the same way: each blurred pixel is a weighted sum of a clear pixel and its neighbors, where the weights form a per-pixel blur kernel normalized to sum to one. The hard part is estimating those kernels. Rather than a plain CNN on the rendered sharp image — which the authors note can let the Gaussians and the network deform together into non-rigid distortions — the paper uses BP-Net, a four-layer CNN that takes a learnable camera-view embedding, a scene feature (computed by a three-layer CNN from the rendered image, rendered depth, and rendered motion mask, with skip connections after the first two layers), and a pixel positional encoding. It outputs both the kernel and a blur intensity per pixel. The final synthesized blurry pixel blends the sharp rendered pixel and the convolved blurry pixel using that intensity, so the reconstruction loss against the true blurry input pushes the sharp render toward sharpness while the network explains the remaining blur.

A blur-aware sparsity constraint keeps the kernels physically plausible: mildly blurred pixels should have kernels concentrated near the center, and severely blurred pixels should have dispersed kernels. The blur intensity, passed through a sigmoid and a stop-gradient, produces a target center weight (scale factor 5), and an L1 loss pulls the kernel's actual center weight toward it. This constraint is only switched on after 5,500 iterations once the intensity has stabilized.

Two further mechanisms support quality. Dynamic Gaussian densification runs once at iteration 2,500: since initialization uses only tracking points visible in the canonical frame, missing regions are filled by reprojecting dynamic pixels from all observation frames using depth, then mapping them back to the canonical frame using the affine transformation of the nearest existing dynamic Gaussian (foreground remapping; motion masks come from SAM). Unseen-view regularization, applied every 5 iterations starting at iteration 3,000, creates hypothetical views near the training trajectory and uses their colors and masks as extra supervision — with only the mask loss used, to avoid inaccurate unseen-view geometry distorting the scene.

Training optimizes Gaussians, motion bases, and BP-Net jointly with Adam: an L1 plus SSIM reconstruction loss (β = 0.2), depth and mask geometry losses (both weights 0.075), a smoothing loss on motion bases and dynamic Gaussian means, and the sparsity loss. Learning rates are 1.6×10⁻⁴ for motion bases and 5×10⁻⁴ for BP-Net, with the Gaussian learning rate following original 3DGS; each scene trains for 40,000 iterations.

Evaluation uses the D2RF defocus dataset and the DyBluRF motion dataset, training on left-view blurry sequences and testing on right-view sharp sequences, with downsampled images as in prior work, and reporting PSNR, SSIM and LPIPS.

Why This Matters

Impact on research. The paper challenges the assumption that defocus and motion blur need separate reconstruction pipelines, showing that a single kernel-convolution formulation plus a learned per-pixel kernel predictor can serve both. It also demonstrates that reconstruction-aware deblurring can outperform pipelining a state-of-the-art video deblurrer into a dynamic reconstruction method, because deblurring alone does not enforce 3D scene consistency. The ablation results give concrete evidence for three design choices (densification, unseen views, sparsity constraint) that other 4D reconstruction work could adopt.

Real-world applications (as framed by the paper and its problem setting):

• Augmented reality, where virtual content must appear sharp over camera footage that is blurred by motion or shallow depth of field.
• Virtual reality and immersive playback, where viewers move to arbitrary viewpoints.
• 3D content creation from ordinary handheld monocular video that was not captured under controlled exposure settings.
• Producing sharp training-view images as a side effect (the deblurring comparison suggests a use in salvage/restoration of blurry footage).

Industry relevance. Training takes approximately 1 hour per scene and renders at 65.143 FPS at 512×288 on a single RTX 3090, which is closer to interactive workloads than the 48-hour runtimes reported for D2RF and DyBluRF. The requirement to optimize each scene separately, and the reliance on external 2D priors (depth, tracking, segmentation), remain practical barriers to deployment.

Future Directions

• Reducing dependence on 2D priors. The stated limitation is that errors in depth estimation and segmentation propagate into the reconstruction; better or jointly optimized priors are an obvious next step.
• Handling large non-rigid motion blur. The authors show a failure case where their method and other state-of-the-art methods produce artifacts under large non-rigid motion; this is left open.
• Removing per-scene optimization. The method inherits the need to optimize each scene separately from vanilla 3DGS; generalizable or feed-forward variants are not addressed.
• Extending evaluation to naturally joint blur. The D2RF-v2 and DyBluRF-v2 datasets used for simultaneous motion and defocus blur were synthesized by the authors (DoF rendering from Bokehme applied to DyBluRF, and Davanet motion blur generation applied to D2RF), so validating on captured real-world joint blur remains untested.

Target Audience

Researchers and graduate students working on novel view synthesis, 4D/dynamic scene reconstruction, Gaussian Splatting, NeRF-based deblurring, or computational photography. Practitioners building AR/VR or 3D content-creation pipelines from casual monocular video will also find the runtime figures and ablation analysis relevant. Readers need prior exposure to 3DGS, differentiable rendering, and camera models; the paper's notation (SE(3) motion bases, rendering and camera intrinsics/extrinsics) is not introduced from scratch.

Authors’ abstract

This paper presents a unified framework that allows high-quality dynamic Gaussian Splatting from both defocused and motion-blurred monocular videos. Due to the significant difference between the formation processes of defocus blur and motion blur, existing methods are tailored for either one of them, lacking the ability to simultaneously deal with both of them. Although the two can be jointly modeled as blur kernel-based convolution, the inherent difficulty in estimating accurate blur kernels greatly limits the progress in this direction. In this work, we go a step further towards this direction. Particularly, we propose to estimate per-pixel reliable blur kernels using a blur prediction network that exploits blur-related scene and camera information and is subject to a blur-aware sparsity constraint. Besides, we introduce a dynamic Gaussian densification strategy to mitigate the lack of Gaussians for incomplete regions, and boost the performance of novel view synthesis by incorporating unseen view information to constrain scene optimization. Extensive experiments show that our method outperforms the state-of-the-art methods in generating photorealistic novel view synthesis from defocused and motion-blurred monocular videos. Our code is available at https://github.com/hhhddddddd/dydeblur.