Seeing through Light and Darkness: Sensor-Physics Grounded Deblurring HDR NeRF from Single-Exposure Images and Events

AI summary

Overview

Research area: Computer vision, specifically neural radiance fields (NeRF) for novel view synthesis, high dynamic range (HDR) imaging, event-camera sensing, and image deblurring.

Technical level: Advanced. The paper assumes familiarity with NeRF volume rendering, camera response functions, event-camera generation models, and HDR tone mapping.

Scope (one sentence): The paper proposes See-NeRF, a NeRF framework that reconstructs sharp HDR 3D scene representations from a single blurry low dynamic range (LDR) image plus its corresponding event stream, by explicitly modeling the physical sensor pipelines of both RGB and event cameras.

What This Paper Is About

Ordinary cameras lose information in extreme lighting: highlights saturate at 255, underexposed regions clip at 0, and a long exposure turns moving scene content into motion blur. Event cameras, which asynchronously record brightness changes in the log domain, preserve the temporal and spatial radiance detail that the RGB image loses. The goal of this paper is to reconstruct a sharp, HDR 3D scene (renderable from novel viewpoints and novel exposures) from just one blurry LDR image and its events, by modeling how scene radiance is physically transformed into actual sensor measurements rather than assuming the two align directly.

Key Contributions

1. See-NeRF framework: A sensor-physics grounded framework that uses events to compensate for scene dynamics and learn a sharp HDR 3D representation from single-exposure blurry LDR images in extreme lighting scenes.

2. Two differentiable CRF models: A physically grounded RGB CRF model (applied at the pixel level after volume rendering, not on 3D points) and a latency-aware, photometrically calibrated event CRF model, both designed to bridge the discrepancy between physical radiance values and sensor measurements.

3. New datasets: Both synthetic and real datasets collected for training and testing the ERGB (event + RGB) deblurring HDR novel view synthesis task, since the prior work EvHDR-NeRF did not release its datasets.

4. State-of-the-art results: Reported best performance on both the ERGB-based HDR and deblurring novel view synthesis tasks, including on the public Real-World-Challenge dataset.

Main Findings

• HDR novel view synthesis on the synthetic dataset: See-NeRF reaches PSNR 24.13 / SSIM .8927 / LPIPS .1916, compared with EvHDR-NeRF at 21.73 / .8192 / .3446 and the sharp-image RGB baseline HDR-NeRF at 17.02 / .7384 / .3450. The multi-exposure reference HDR-NeRF ref is 26.91 / .8906 / .1705.

• HDR novel view synthesis on the real dataset: See-NeRF reaches 26.49 / .8953 / .1638, which the authors note is even better than the multi-exposure reference HDR-NeRF ref (26.26 / .8716 / .1576). EvHDR-NeRF is 19.00 / .6962 / .2612; HDR-NeRF is 10.87 / .5473 / .4229.

• Novel exposure synthesis: On synthetic data See-NeRF reaches 27.57 / .9270 / .1328 versus EvHDR-NeRF at 24.37 / .8712 / .2848; on real data See-NeRF reaches 29.11 / .9364 / .1060 versus EvHDR-NeRF at 23.76 / .8830 / .1650.

• Deblurring on the public Real-World-Challenge dataset: See-NeRF reaches 32.70 / .9564 / .1574, above the strongest ERGB-based deblurring method E3NeRF (31.40 / .9464 / .2000), the strongest RGB-based method DP-NeRF (28.85 / .9226 / .3015), and EvHDR-NeRF (27.19 / .8960 / .3731).

• Ablation — RGB CRF model matters: Removing events but keeping the authors' RGB CRF model (See-NeRF NoEv) drops synthetic HDR PSNR to 22.19 and real-data PSNR to 19.84, versus 24.13 and 26.49 for the full model; the base model on Real-World-Challenge is 28.79 PSNR.

• Ablation — 2D pixel-wise CRF beats 3D-point CRF: Replacing the 2D pixel-wise RGB CRF with the 3D-point CRF used in prior work (See-NeRF 3D) yields 20.67 on synthetic HDR, 24.95 on real HDR, and 31.58 on Real-World-Challenge, all below the full model, and leads to incorrect CRF curve estimation.

• Ablation — event CRF model adds a gain: Using a naive event CRF instead (See-NeRF NoEM) gives 22.23 / 25.87 / 32.47 across the three datasets, below the full See-NeRF. Substituting the eCRF from EvDeblur-NeRF (See-NeRF EvD) produces weak results, notably 16.43 PSNR on the real HDR data.

• Robustness to exposure time: On the synthetic "Catroom" scene, HDR PSNR is 24.66 at Δt₀ = 0.001 s, 25.08 at Δt₂ = 0.016 s, and 25.68 at Δt₄ = 0.256 s, so longer exposure inputs perform better — the authors attribute this to low-light regions being harder to learn.

• Robustness to event noise: Noise levels ×1, ×2 and ×4 give PSNR 25.08, 24.99 and 24.86 respectively. The authors state See-NeRF does not target event noise processing because multi-view NeRF training inherently weakens sparse random per-view noise.

• Timing and 3DGS substitution: See-NeRF trains in 2.5 hours versus 2.1 hours for HDR-NeRF, and renders at 0.37 fps. Replacing NeRF with 3DGS ("Ours + 3DGS") trains in 0.05 hours at 5.11 fps, faster than HDR-GS (0.03 hours, 4.29 fps rendering), and beats HDR-GS on the "computerhard" scene (novel exposure 20.39 versus 19.19; HDR 18.27 versus 9.25), but falls well short of full See-NeRF (24.15 novel exposure, 19.43 HDR).

Methodology in Plain English

The core idea is to stop asking a neural network to reproduce sensor output directly, and instead ask it to reproduce the light in the scene, then simulate the sensors in between.

1. Represent the scene in HDR, not LDR. The NeRF network F_θ outputs the actual radiance e and density σ of 3D points, rather than the processed LDR colors used in the original NeRF. Volume rendering then produces the raw HDR radiance that would hit a sensor pixel for a given camera pose.

2. Simulate the RGB camera after rendering. Because a real photo is an integral of radiance over the exposure window followed by a non-linear camera response function (CRF), the authors sample b + 1 time points, weight and sum the rendered radiance values into a raw HDR pixel value, then pass it through three MLPs (one per color channel) that fit the CRF in the log domain, scaled by exposure time. Critically, this CRF is applied after volume rendering (at the pixel level), which the authors argue matches the real imaging process and lets the CRF learn tone mapping while the NeRF learns geometry and radiance.

3. Simulate the event camera. The ideal event model counts threshold crossings in log radiance, but real event sensors deviate. The event CRF model has three parts: converting radiance into an RGGB Bayer pattern as the event sensor would see it; estimating a per-pixel temporal delay coefficient with an MLP and smoothing radiance through a second-order low-pass filter to reproduce latency (which is stronger in low light); and calibrating the photometric quantity by subtracting a pre-estimated offset derived from the actual input events.

4. Train jointly. Two L2 losses supervise everything at once: an LDR image loss (predicted LDR pixels versus the input blurry image) and an event loss (predicted event counts versus the input events). The combined loss is λ · event loss + LDR loss, with λ = 0.005 and b = 4. Camera poses during handheld exposure are estimated with an event-guided COLMAP strategy.

5. Evaluate. Because EvHDR-NeRF did not release data, the authors build a synthetic dataset of 8 HDR Blender scenes from HDR-NeRF (18 training and 17 test views each, 400 × 400 resolution, blur and events generated with Blender and the v2e simulator) and a real dataset of 5 scenes captured with a DAVIS 346 event camera (16 training and 12 test views each). They also test on the public Real-World-Challenge dataset. Metrics are PSNR, SSIM and LPIPS.

Why This Matters

Impact on research: The paper argues that the field's dominant approach — directly extending image-based HDR or deblurring NeRF frameworks to event data — leaves a sensor-physics mismatch unaddressed, and that explicitly modeling radiance-to-measurement transformations is what unlocks the complementary information in ERGB data. It provides both a framework and new datasets for the deblurring HDR novel view synthesis problem.

Real-world applications:

• Photorealistic 3D capture with consumer or handheld cameras in mixed bright/dark scenes, such as indoor spaces with windows and dim corners.
• Autonomous driving and robotics, where sensors must operate in tunnels, night scenes and glare, and motion blur degrades perception.
• Virtual and augmented reality content creation, where consistent appearance under arbitrary chosen exposures would let users re-expose a captured scene after the fact.
• Surveillance, inspection and cultural-heritage digitization, where capturing multiple exposures and a tripod may be impractical.

Industry relevance: The work targets any product pipeline that must turn cheap, non-ideal single-shot capture into high-quality 3D content. The authors' 3DGS experiment (trainable in 0.05 hours versus 2.5 hours for the NeRF version, with 5.11 fps rendering) points at where the speed-sensitive end of that market would go, while noting the accuracy trade-off.

Future Directions

1. Event noise handling. The authors explicitly state See-NeRF does not target event noise processing — the robustness test shows only gradual degradation — and call it a future optimization direction for ERGB-based novel view synthesis.

2. ERGB-based HDR 3D Gaussian Splatting. The "Ours + 3DGS" variant underperforms full See-NeRF on novel exposure, which the authors attribute to the MLP-based CRF model not perfectly matching 3DGS during optimization; they call ERGB-based HDR 3DGS a promising future work.

3. Scaling the datasets. The evaluation uses 8 synthetic scenes, 5 real scenes and the Real-World-Challenge benchmark; extending to larger and more diverse captures, and releasing the data (the authors state code and datasets are publicly available at the project GitHub page), would test generality.

4. Higher-fidelity CRF modeling. The 2D pixel-wise CRF consistently beats the 3D-point CRF used in prior work, and the event CRF's latency and photometric offset terms are estimated rather than measured — refining these models is a natural extension of the paper's central argument.

Target Audience

Researchers and graduate students working on neural radiance fields, 3D Gaussian Splatting, event-based vision, computational photography, or HDR and deblurring image reconstruction. It is also relevant to practitioners in computational imaging and 3D capture who need robustness to single-shot, low-quality input under extreme lighting, and who are willing to engage with the mathematical formulations of camera response functions and event generation models.

Authors’ abstract

Novel view synthesis from low dynamic range (LDR) blurry images, which are common in the wild, struggles to recover high dynamic range (HDR) and sharp 3D representations in extreme lighting conditions. Although existing methods employ event data to address this issue, they ignore the sensor-physics mismatches between the camera output and physical world radiance, resulting in suboptimal HDR and deblurring results. To cope with this problem, we propose a unified sensor-physics grounded NeRF framework for sharp HDR novel view synthesis from single-exposure blurry LDR images and corresponding events. We employ NeRF to directly represent the actual radiance of the 3D scene in the HDR domain and model raw HDR scene rays hitting the sensor pixels as in the physical world. A 2D pixel-wise RGB CRF model is introduced to align the NeRF rendered pixel values with the sensor-recorded LDR pixel values of the input images. A novel event CRF model is also designed to bridge the gap between physical scene dynamics and event sensor output. The two models are jointly optimized with the NeRF network, leveraging the spatial and temporal dynamic information in events to enhance the sharp HDR 3D representation learning. Experiments on the collected and public datasets demonstrate that our method achieves state-of-the-art HDR and deblurring novel view synthesis results with single-exposure blurry LDR images and corresponding events.