Learning Neural Exposure Fields for View Synthesis

AI summary

Overview

• Research area: Computer vision, specifically neural scene representations and view synthesis (NeRF-based novel view synthesis under varying exposure).
• Technical level: Intermediate. The paper assumes familiarity with neural radiance fields, volume rendering, and high dynamic range (HDR) imaging concepts such as tonemapping and exposure.
• Scope: The paper introduces Neural Exposure Fields (NExF), a method that predicts an optimal exposure value per 3D point and optimizes it jointly with a neural scene representation, enabling 3D-consistent, well-exposed view synthesis from RGB captures with strong exposure variation.

What This Paper Is About

Neural scene representations like NeRF and ZipNeRF produce high-quality results on curated datasets, but their outputs degrade on real-world captures where exposure changes from image to image, such as scenes mixing indoor rooms and windows with outdoor areas. Existing fixes either attach a per-image appearance embedding (which can leave parts of the scene over- or underexposed) or perform 2D tonemapping (which makes the same 3D point look inconsistent across views). The goal of this work is to reconstruct scenes with high-quality, well-exposed appearance that stays consistent in 3D, without needing HDR captures, professional tonemapping software, or a reference appearance created by an artist.

Key Contributions

1. A novel neural representation for exposure prediction: a neural field e_φ: ℝ³ → ℝ that predicts the optimal exposure value for each 3D point, rather than per image or per pixel as cameras do.
2. A joint optimization system: the exposure field and the scene representation are trained end-to-end, with a novel latent exposure conditioning mechanism that applies the log exposure transformation to the MLP bottleneck vector instead of to radiance.
3. Color-based training criteria: well-exposedness and saturation weights, plus a 3D smoothness regularizer on predicted exposure, that together allow the exposure field to be supervised from 2D observations without ground-truth 3D exposure.
4. A thorough evaluation: experiments on the HDRNeRF dataset, the Eyeful Tower v2 dataset, in-the-wild phone captures, and a large-scale ZipNeRF scene, reporting improvements of over 55% in MSE over the best-performing baselines.

Main Findings

• Performance on HDRNeRF exposure reconstruction: The method reaches PSNR 42.54 / SSIM 0.988 / LPIPS 0.014 on in-distribution exposures and 38.36 / 0.984 / 0.021 on out-of-distribution exposures. This is +3.5 PSNR over HDRNeRF (39.07 → 42.54) for ID and +0.8 PSNR (37.53 → 38.36) for OOD.
• Comparison to HDR-GS: Against HDR-GS, which uses full HDR captures during training, the method improves PSNR by +1.4 (ID) and +2.0 (OOD) and also improves SSIM, while achieving a slightly higher (worse) LPIPS value.
• Training speed: The method trains in 11 minutes, at least 3x faster than the fastest baseline, while HDRNeRF takes 542 minutes and NeRF-W takes 437 minutes.
• Performance on Eyeful Tower: The method leads all metrics with PSNR 26.48 / SSIM 0.876 / LPIPS 0.234, beating Ignore Exposure (16.72 / 0.682 / 0.444), Affine GLO (20.13 / 0.815 / 0.263), GLO (21.20 / 0.824 / 0.298), and HDRNeRF* (22.82 / 0.836 / 0.311). It improves over HDRNeRF* by 57% in MSE (+3.7 PSNR, +0.04 SSIM) and improves over Affine GLO by 12% in LPIPS (−0.03).
• Ablation on architecture and conditioning (HDRNeRF): Removing the modified view-dependent MLP drops ID PSNR to 33.85 from 42.54; removing latent exposure conditioning drops it to 39.88. Both components are needed for best results.
• Ablation on the exposure field (Eyeful Tower): Removing well-exposedness drops LPIPS to 0.247, removing saturation drops PSNR to 24.30, and removing regularization drops PSNR to 24.84. Removing affine GLO embeddings raises PSNR slightly to 27.44 but worsens SSIM and LPIPS to 0.854 and 0.306.
• Qualitative behavior: Ignoring exposure produces floating artifacts and scene degradation; GLO and affine GLO baselines still show over- and underexposure; the proposed method produces well-exposed colors across the whole scene, including fine details and distant regions.
• Improvement over ground truth: In the large-scale figures, the method's renderings improve over the ground-truth view that was captured with a single exposure value.

Methodology in Plain English

The system takes a set of RGB images with known exposure times and learns two networks at once. The first is a standard neural radiance field (a modified ZipNeRF) that predicts density and color for points in space; the second is a small MLP that predicts, for any 3D point, what exposure value would make that point look best.

During training, each point along a camera ray is fed to the radiance field together with the exposure of the input image. The paper conditions on exposure in the log domain, but unlike prior work it adds the log exposure to the network's internal bottleneck vector rather than to the radiance itself, which the authors find improves performance and stability. The predicted color is compared against the input image with a standard MSE loss.

The exposure field is trained separately from the color loss. Its predictions are volume-rendered onto the image plane using the same alpha-blending weights as the color renderer (with gradients detached so geometry is unaffected), and compared against the input image's exposure. Crucially, this comparison is weighted per pixel by two criteria: how well the pixel's color is exposed (colors near the middle of the range are good, colors near clipping are not) and how saturated the color is. This means poorly exposed pixels contribute little to the exposure loss, so the model only needs the assumption that most parts of the scene are well exposed in some view, not in every view. A regularizer encourages the exposure field to vary smoothly in 3D by penalizing differences between nearby points perturbed with 3D noise.

At test time, the radiance field is conditioned on the exposure field's predictions rather than on a per-image exposure. This yields one consistent, well-exposed appearance for the whole scene with no 2D tonemapping, no HDR input, and no artist-provided reference.

Why This Matters

Impact on research. The paper reframes exposure as a spatially varying, learned 3D quantity rather than a camera-level or image-level one, and shows this can be optimized end-to-end alongside a radiance field. This connects neural view synthesis to classic HDR and exposure fusion literature while avoiding their 2D inconsistencies, and it offers an alternative to appearance embeddings such as GLO, which the paper shows degrade under strong exposure change.

Real-world applications:

• Reconstructing indoor scenes with large windows or mixed indoor/outdoor areas, where current methods routinely fail.
• Building 3D assets and walkthroughs from ordinary phone photo collections taken under different lighting, without a multi-exposure capture rig.
• Improving the visual quality of large-scale scene captures for mapping and real-estate visualization.
• Removing the need for professionally authored reference appearances, which currently require expensive HDR software or manual artist work.

Industry relevance. The method uses a widely adopted radiance field backbone (ZipNeRF) and trains on modest hardware budgets (11 to 30 minutes reported), which makes it practical for production pipelines. Because it needs only RGB images with exposure metadata, it targets the far more common case of consumer captures, unlike HDR-based methods that require professional capture setups.

Future Directions

• Extreme lighting conditions: The authors state results may degrade for very low-light or extremely over-exposed captures, leaving room for methods that handle these regimes.
• Complex lighting effects: Semi-transparency and strong reflections are explicitly listed as cases where the method may break down.
• Backbone generalization: The paper notes 3DGS was not chosen due to degradation on challenging data and less stable joint optimization with the MLP exposure field, but mentions the model can be combined with distillation approaches, suggesting later integration with splatting representations.
• No 3D exposure ground truth: Since no target 3D exposure exists, supervision relies on hand-designed well-exposedness and saturation criteria and a smoothness regularizer; whether better or learned criteria exist is an open question.

Target Audience

Researchers and engineers working on neural radiance fields, novel view synthesis, and 3D reconstruction from real-world captures. It is also relevant to practitioners in computational photography and HDR imaging, and to graphics or VFX teams dealing with 3D-consistent appearance from photos with varying exposure. Readers should already understand volume rendering, NeRF-style architectures, and basic tonemapping concepts to follow the method and ablations.

Authors’ abstract

Recent advances in neural scene representations have led to unprecedented quality in 3D reconstruction and view synthesis. Despite achieving high-quality results for common benchmarks with curated data, outputs often degrade for data that contain per image variations such as strong exposure changes, present, e.g., in most scenes with indoor and outdoor areas or rooms with windows. In this paper, we introduce Neural Exposure Fields (NExF), a novel technique for robustly reconstructing 3D scenes with high quality and 3D-consistent appearance from challenging real-world captures. In the core, we propose to learn a neural field predicting an optimal exposure value per 3D point, enabling us to optimize exposure along with the neural scene representation. While capture devices such as cameras select optimal exposure per image/pixel, we generalize this concept and perform optimization in 3D instead. This enables accurate view synthesis in high dynamic range scenarios, bypassing the need of post-processing steps or multi-exposure captures. Our contributions include a novel neural representation for exposure prediction, a system for joint optimization of the scene representation and the exposure field via a novel neural conditioning mechanism, and demonstrated superior performance on challenging real-world data. We find that our approach trains faster than prior works and produces state-of-the-art results on several benchmarks improving by over 55% over best-performing baselines.