Mono4DGS-HDR: High Dynamic Range 4D Gaussian Splatting from Alternating-exposure Monocular Videos

AI summary

Overview

Research area: Computer vision — 4D (dynamic 3D + time) high dynamic range (HDR) scene reconstruction and novel view synthesis from monocular video.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, camera pose estimation, tone mapping, and HDR imaging.

Scope: The paper presents Mono4DGS-HDR, a two-stage Gaussian Splatting system that reconstructs renderable 4D HDR scenes from unposed monocular LDR videos with alternating exposures, and introduces a new evaluation benchmark for this previously unstudied task.

What This Paper Is About

Existing HDR novel view synthesis methods largely target static scenes, and the one prior dynamic HDR method (HDR-HexPlane) requires known camera poses and a multi-camera setup. Meanwhile, unposed monocular 4D reconstruction methods assume videos with consistent brightness and break down when exposure alternates frame to frame. Mono4DGS-HDR addresses the practical case of a single handheld camera capturing a dynamic scene where frames alternate between short and long exposure times, with no camera poses provided. The goal is to recover the full 4D HDR scene, the camera parameters, and the ability to render both HDR and controllable-exposure LDR images from novel viewpoints.

Key Contributions

1. First system for the task: Mono4DGS-HDR is presented as the first system for reconstructing 4D HDR scenes from unposed monocular LDR videos captured with alternating exposures.

2. Two-stage optimization framework: The method learns video Gaussians in a 3D canonical orthographic camera coordinate space (removing the need for camera poses), transforms them into world space, and then jointly optimizes world Gaussians with camera poses.

3. Temporal luminance regularization (TLR): A flow-guided photometric loss aligns per-pixel HDR irradiance between consecutive frames to stabilize HDR appearance over time.

4. New evaluation benchmark: A benchmark of 25 dynamic scenes built from publicly available HDR video datasets, covering synthetic and real-world alternating-exposure clips.

Main Findings

• Best rendering quality on synthetic scenes: On Syn-Exp-3 test frames, Mono4DGS-HDR reaches 34.75 PSNR / 0.904 SSIM / 0.086 LPIPS on observed-exposure LDR, 34.54 / 0.915 / 0.081 on novel-exposure LDR, and 37.64 / 0.959 / 0.042 on HDR. The strongest baseline, MoSca-HDR, reaches 34.08 / 0.898 / 0.098, 33.92 / 0.910 / 0.092, and 36.89 / 0.952 / 0.053 respectively.

• Fastest rendering: At 864 × 480 resolution, the method renders at 161 FPS, versus 82 FPS for MoSca-HDR, 79 FPS for SplineGS-HDR, 51 FPS for GaussHDR, and 1 FPS for HDR-HexPlane. Training time is reported as 1.5h for the proposed method and MoSca-HDR/SplineGS-HDR, and 1h for GaussHDR and HDR-HexPlane.

• Stronger on real data: On Real-Exp-2 training frames, the method reports 31.82 PSNR / 0.928 SSIM / 0.052 LPIPS with HDR-TAE 0.046, compared with MoSca-HDR at 30.28 / 0.915 / 0.074 and TAE 0.054. On Real-Exp-3 test frames it reports 27.65 / 0.876 / 0.081 with TAE 0.067, versus MoSca-HDR at 27.23 / 0.872 / 0.084 with TAE 0.076.

• Baseline failure modes: GaussHDR and HDR-HexPlane perform poorly due to their inability to handle dynamic scenes and monocular videos respectively; GFlow fails to recover HDR scenes because it optimizes per-frame Gaussians independently and overfits the LDR observations; SplineGS behaves worst (17.59 PSNR LDR-OE on Syn-Exp-3) because it relies on photometric reprojection loss to recover camera motion, which is infeasible under alternating exposures. GFlow cannot support time interpolation of Gaussians, so it is only evaluated on training frames.

• Video Gaussian initialization matters most: Removing the video Gaussian stage and training directly in world space with track/depth initialization drops PSNR by more than 1 dB (Real-Exp-3 LDR-OE falls to 26.47; Syn-Exp-3 HDR falls to 36.07).

• Occlusion handling in transformation: Removing it causes a 0.3 dB PSNR drop (Real-Exp-3 LDR-OE 27.29) and blurry background reconstruction in regions that have been occluded.

• Temporal luminance regularization affects stability, not quality: Removing TLR leaves PSNR largely unchanged (Real-Exp-3 LDR-OE 27.63 vs. 27.65 for the full model) but degrades temporal stability substantially (HDR-TAE 0.082 vs. 0.067 on Real-Exp-3; 0.071 vs. 0.057 on Syn-Exp-3).

• Control point sampling interval: Sampling every N_s = 4 frames is chosen as the trade-off. Performance degrades at N_s = 8 (Syn-Exp-3 LDR-OE 34.16) and N_s = 16 (33.67), while N_s = 1 and N_s = 2 give slightly higher PSNR (34.86 and 34.82) but require more control points and thus more memory.

• Induced tracking: Projecting the learned dynamic Gaussian trajectories yields accurate 2D and 3D tracking of dynamic objects.

Methodology in Plain English

The input is a monocular video whose frames alternate between a short exposure time Δt_s and a long exposure time Δt_l, with no known camera poses. The system works in two stages.

Stage 1 — video Gaussians. Instead of working in world coordinates, the method optimizes a set of fully dynamic HDR Gaussians in an orthographic camera coordinate space, where the x and y coordinates are normalized pixel positions and z is depth. In this space, camera motion and object motion are both treated as Gaussian motion, so no camera parameters are needed and the HDR training video can be reconstructed robustly. Each Gaussian's position follows a cubic Hermite spline trajectory with control points, and its rotation is a cubic polynomial over time; scaling, opacity and color are time-invariant.

Scene priors. Before optimization, standard vision foundation models supply depth estimates, sparse long-term 2D pixel tracks, and per-frame epipolar error maps (computed via dense optical flow) that identify dynamic regions by thresholding. Optical flow is computed only between frames at the same exposure level. Bundle adjustment on static tracklets yields initial camera parameters.

Transformation from video to world space. Video Gaussians live in a pseudo-3D space, so they must be converted. First, each Gaussian is classified as dynamic or static by projecting its trajectory onto dynamic masks while accounting for occlusion (whether its depth exceeds the rendered depth). If the fraction of frames in which it falls in dynamic regions exceeds a threshold such as 0.1, it is dynamic. Positions and rotations are then transformed using the initial intrinsics and extrinsics; static Gaussians get averaged positions and quaternion-averaged rotations, while dynamic ones are re-fitted by resampling spline control points. Opacity and HDR color are inherited directly. Because video space and world space differ in scale, scaling is re-fitted by requiring the projected 2D covariance to stay the same before and after the transformation, solved by gradient descent, which the paper reports converges within 1000 iterations in 1 minute.

Stage 2 — joint refinement. World Gaussians and camera poses are refined together. A key enabler is the HDR photometric reprojection loss: because Stage 1 produces frames with consistent brightness, the classic photometric reprojection error becomes usable for pose optimization, which is not possible directly on alternating-exposure LDR frames.

Rendering and losses. At each iteration a random timestamp is sampled, and the system renders an HDR image, a depth map, and a flow/track map; tone-mapping MLPs with the exposure time convert HDR to LDR. Supervision combines LDR RGB loss, depth loss, track loss, unit exposure loss, motion regularizers (as-rigid-as-possible, velocity, acceleration), the temporal luminance regularization, and the HDR photometric reprojection loss. The temporal luminance regularization uses rendered flow maps with stop-gradients to warp the previous HDR frame to the current time and compares normalized irradiance, using a depth-order valid mask to exclude occluded pixels. It is only applied after Gaussian densification ends, so that flow rendering is reliable. Gaussian densification follows the standard 3DGS strategy.

Why This Matters

Impact on research. The paper opens a task that had not been studied: HDR 4D reconstruction from unposed monocular alternating-exposure video. It shows that 2D priors from vision foundation models remain extractable even under alternating exposure, that camera poses can become optimizable once a consistent-brightness HDR video is recovered first, and it supplies a benchmark of 25 scenes for future comparison. The two-stage video-to-world design also offers a general recipe for initializing world Gaussians when pose estimation from raw LDR frames is unreliable.

Real-world applications (as supported by the paper's framing):

• Handheld consumer video capture in mixed lighting, where a single phone camera must handle scenes with both bright and dark regions.
• Rendering novel views of a captured moment with user-adjustable exposure, since the method supports synthesizing LDR images at new exposure levels.
• Low-light or high-contrast dynamic scene capture where a single exposure cannot span the scene's brightness range.
• Deriving 2D and 3D motion tracks of dynamic objects as a byproduct of the Gaussian trajectories.

Industry relevance. The work is a collaboration between HKUST and vivo Mobile Communication Co., Ltd, and is supported in part by funding from the Hong Kong Research Grants Council and other industrial partners, pointing toward mobile and computational photography pipelines where alternating-exposure capture and post-capture relighting or view synthesis are relevant. Rendering at 161 FPS at 864 × 480 is relevant to interactive applications.

Future Directions

• Extending beyond two exposures: The paper formulates a 2-exposure case and states that other patterns such as 3-exposure can be similarly extended, but a general treatment across exposure counts and irregular patterns is left open.
• Dependence on foundation-model priors: The pipeline relies on DepthCrafter, SpatialTracker, and RAFT for depth, tracks, and flow; errors or noise in these priors propagate into the initialization and regularization.
• Broader scene coverage: The benchmark contains 25 scenes (9 synthetic with 3 exposure levels, 8 real with 3 exposure levels, 8 real with 2 exposure levels); expanding it, including to nighttime and low-light HDR cases, is a natural next step.
• Memory and representation trade-offs: The control-point sampling interval of 4 frames was selected as a trade-off between performance and storage, and memory cost remains a stated constraint on smaller intervals.

Target Audience

Researchers and advanced practitioners in computer vision and graphics working on Gaussian Splatting, dynamic scene reconstruction, novel view synthesis, HDR imaging, or computational photography. It will also interest engineers building mobile or handheld capture and rendering pipelines who need HDR dynamic scene reconstruction without camera pose input or multi-camera rigs.

Authors’ abstract

We introduce Mono4DGS-HDR, the first system for reconstructing renderable 4D high dynamic range (HDR) scenes from unposed monocular low dynamic range (LDR) videos captured with alternating exposures. To tackle such a challenging problem, we present a unified framework with two-stage optimization approach based on Gaussian Splatting. The first stage learns a video HDR Gaussian representation in orthographic camera coordinate space, eliminating the need for camera poses and enabling robust initial HDR video reconstruction. The second stage transforms video Gaussians into world space and jointly refines the world Gaussians with camera poses. Furthermore, we propose a temporal luminance regularization strategy to enhance the temporal consistency of the HDR appearance. Since our task has not been studied before, we construct a new evaluation benchmark using publicly available datasets for HDR video reconstruction. Extensive experiments demonstrate that Mono4DGS-HDR significantly outperforms alternative solutions adapted from state-of-the-art methods in both rendering quality and speed.