Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark

AI summary

Overview

• Research area: Computational photography and 3D computer vision — specifically multi-view low-light image restoration that fuses noisy RGB observations with Near-Infrared (NIR) images inside a neural radiance field (NeRF-style) volume-rendering pipeline.
• Technical level: Advanced. The paper assumes familiarity with neural radiance fields, volume rendering, positional encoding, MLP architectures, and Gumbel-Softmax / differentiable discrete sampling.
• Scope: The paper proposes a 3D-aware neural implicit fusion model that recovers clean RGB from extremely noisy multi-view RGB plus NIR captures without any clean RGB supervision, and validates it on a newly generated synthetic dataset and a newly captured real-world dataset.

What This Paper Is About

Existing RGB–NIR low-light enhancement methods are mostly trained on carefully curated "noisy RGB – NIR – clean RGB" triplets from a specific domain, which limits how well they generalize and how robust they are as noise levels change. This paper asks whether robust dark imaging can be achieved using only NIR and noisy RGB observations, with no clean RGB supervision at all. The authors answer yes, by moving the fusion from 2D image space into 3D space using a multi-view neural implicit model that jointly renders NIR and RGB.

Key Contributions

1. A new 3D-aware fusion model for RGB–NIR dark imaging that fuses noisy RGB observations with NIR cues in 3D space, unlike existing RGB–NIR models, and does so without requiring clean RGB supervision.
2. Two components tailored to the characteristics of RGB–NIR modalities: an NIR-modulated positional encoding (NIR-P.E.) scheme that suppresses noise-driven overfitting, and a Color Code MLP (C.C. MLP) that resolves the ill-posed NIR-to-RGB mapping via a learned color code distribution.
3. Both a synthetic dataset (generated with Mitsuba3) and a real-world captured dataset, together with evaluations in sRGB and RAW space, demonstrating superiority across multiple scenarios and noise levels.
4. Code release at https://github.com/MyNiuuu/3DarkFusion, with data and code stated as released to support future research.

Main Findings

• No clean RGB supervision is needed. The model is optimized only against NIR observations and noisy RGB observations via an L2 photometric loss, yet it recovers clean RGB and outperforms supervised baselines.
• Strong synthetic results across noise levels. On the synthetic benchmark, the method achieves SSIM 0.7572 / PSNR 19.53 / LPIPS 0.2361 at s=1/10; 0.7441 / 19.52 / 0.2354 at s=1/25; 0.7485 / 19.70 / 0.2365 at s=1/50; 0.7299 / 19.87 / 0.2485 at s=1/100; and 0.7028 / 19.68 / 0.2670 at s=1/200.
• Performance is stable as noise grows. Competing methods such as Restormer, ScaleMap, NVEU, SANet, NAID, RawNeRF, and LLNeRF degrade noticeably as the scale factor decreases (for example, SANet drops to SSIM 0.1592 and LPIPS 1.2321 at s=1/200, and LLNeRF to SSIM 0.4045 / PSNR 9.01 at s=1/200), while the proposed model stays comparatively flat.
• Beats two-stage "2D Fusion + NeRF" pipelines. Combining NVEU, SANet, or NAID with a NeRF trained on fused results yields worse numbers than the proposed model; for instance SANet + NeRF reaches SSIM 0.7539 / PSNR 17.54 at s=1/10 but collapses to SSIM 0.5231 / PSNR 14.42 / LPIPS 0.8674 at s=1/200.
• Real-world captures favor the method. On 4 real scenes, the model achieves PI 5.415, MUSIQ 56.51, MANIQA 0.2752, and HSE 3.450, compared with the next best baseline NVEU (PI 4.817, MUSIQ 36.01, MANIQA 0.2639, HSE 3.025) and much lower MUSIQ scores for SANet (35.03), ScaleMap (29.61), NAID (19.98), Restormer (16.08), and LLNeRF (13.23). The proposed method also leads on the HSE human study.
• Positional encoding causes checkerboard artifacts. Applying positional encoding to 3D coordinates and view directions under noisy RGB supervision produces visible checkerboard effects; applying it to the NIR estimate instead removes them.
• Frequency analysis explains why. The high-to-low frequency energy ratio ρ_f is 0.5261 for noisy RGB versus 0.0101 for clean RGB and 0.0034 for NIR, showing noisy RGB is dominated by random high-frequency noise while clean RGB and NIR have similar, much smoother spectra.
• The Color Code MLP resolves color ambiguity. A single NIR value can map to multiple RGB values; the C.C. MLP (with K = 16 categories) separates distinct color clusters that a model without it merges into blended estimates.
• Ablations confirm both components. Removing NIR-P.E. gives SSIM 0.7563 / PSNR 19.48 / LPIPS 0.2378 at s=1/10 and worsens to SSIM 0.6936 / LPIPS 0.3072 at s=1/200; removing the C.C. MLP gives SSIM 0.7428 / PSNR 19.34 / LPIPS 0.2563 at s=1/10. On real data, the full model scores PI 5.415, MUSIQ 56.51, MANIQA 0.2752, HSE 3.450 versus 5.531 / 49.17 / 0.2734 / 3.150 without NIR-P.E. and 5.586 / 55.73 / 0.2406 / 3.175 without the C.C. MLP.
• It generalizes to RAW. On 12-bit RAW Bayer data normalized by 4095, the method reaches PI 5.874, MUSIQ 55.70, MANIQA 0.2527, HSE 3.775, compared with RawNeRF (6.783 / 46.78 / 0.2251 / 3.150), NVEU (4.821 / 46.71 / 0.2482 / 2.850), and NAID (9.924 / 34.76 / 0.2240 / 2.625).

Methodology in Plain English

The authors keep the standard volume-rendering formulation from NeRF-style models, which enforces 3D consistency by aggregating information across multiple viewpoints, but they redesign the MLP architecture around the fact that NIR images are structurally similar to clean RGB.

First, they test a parallel design with two separate MLPs, one for NIR and one for RGB, where the NIR branch predicts density and gradients from the RGB branch are stopped. This recovers geometry better than a vanilla NeRF MLP but still misses fine 2D texture.

Second, they let the NIR output feed into the RGB MLP while still blocking gradients from flowing back into the NIR branch. This conditions the RGB prediction on the NIR structure.

Third, they change where positional encoding is applied. Because the frequency-domain energy analysis shows noisy RGB is dominated by random high-frequency content, they avoid encoding the 3D coordinates x and view directions d and instead apply sinusoidal encoding to the predicted NIR color, feeding x, d, and γ(c^N) into the RGB MLP to predict the RGB color. This acts as a frequency-aligned modulation and removes checkerboard artifacts.

Fourth, they address the fact that one NIR value can correspond to several RGB colors. A Color Code MLP predicts a non-uniform log-probability distribution over k categories; a Gumbel-Softmax-style one-hot selection (with i.i.d. Gumbel(0,1) samples g_1…g_k) produces a K-dimensional one-hot vector δ that is passed to the RGB MLP. K is set to 16 by default.

Training minimizes an L2 photometric loss between rendered NIR and RGB estimates and the observed NIR and noisy RGB, with the noisy RGB term raised to the power 1/φ to amplify exposure. The model trains for 200,000 iterations with the Adam optimizer at learning rate 5×10⁻⁴, β₁ = 0.9, β₂ = 0.999, taking about 3 hours on one NVIDIA RTX 4090 GPU with 9 GB peak memory.

The synthetic benchmark uses Mitsuba3, with 5 scenes, 40 randomly sampled camera views per scene, 850 nm NIR renderings at 960 × 1280, and 5 scale factors s ∈ {1/10, 1/25, 1/50, 1/100, 1/200} with physics-based shot and read noise; 5 views per scene are held out for testing and 35 for training. The real benchmark uses a JAI FS-3200T10GE-NNC camera over 4 scenes with 49 views at 768 × 1024 under 850 nm NIR LED illumination, with 5 test views and 44 training views per scene, camera poses calibrated with COLMAP.

Why This Matters

The work removes a practical bottleneck in low-light imaging research: the need to collect aligned clean RGB ground truth for every scene and noise level, which is expensive and often impossible in the dark. It replaces curated triplet supervision with multi-view consistency and NIR structure, and it shows that 3D-aware modeling can outperform both 2D fusion networks and multi-view RGB-only denoisers.

Real-world applications:

• Night-time photography and videography, where an NIR-assisted multi-view capture could recover usable color images without long exposures or bursts.
• Robotics and autonomous navigation in darkness, where paired RGB and NIR sensing already exists and 3D reconstruction is needed in parallel with denoising.
• Surveillance and security imaging, where 850 nm NIR illumination is already standard and color fidelity under near-zero visible light has traditionally been poor.
• Cultural-heritage, industrial, or field documentation, where capturing a scene with NIR illumination plus a set of views may be easier than bringing studio lighting.

Industry relevance: the method trains on a single NVIDIA RTX 4090 in about 3 hours with 9 GB peak memory, which is modest enough to be practical for product teams; the released code and datasets give a direct baseline for camera makers, mobile imaging pipelines, and AR/VR capture systems that ship NIR sensors alongside RGB sensors.

Future Directions

• Extending the model from static scenes to dynamic environments, which the authors explicitly identify as a limitation.
• Handling completely unaligned RGB and NIR capture (for example one RGB camera and a separate NIR camera used freely), where accurate camera poses for the RGB observation are hard to obtain.
• Broadening evaluation beyond the 5 synthetic scenes and 4 real scenes reported here to confirm generalization to more varied materials, illuminants, and NIR wavelengths.
• Investigating whether the NIR-P.E. and Color Code MLP ideas transfer to other guided-restoration settings where the guide modality and the target share a frequency structure.

Target Audience

Researchers and graduate students working on low-light enhancement, RGB–NIR fusion, neural radiance fields, and multi-view computational photography; engineers building night-time or NIR-assisted imaging pipelines who want a supervision-light alternative to paired-data training; and practitioners interested in differentiable discrete representations such as the Gumbel-Softmax-based Color Code MLP.

Authors’ abstract

Robust low-light imaging remains challenging for the community. Recent studies have explored fusing Near-Infrared (NIR) with noisy RGB to achieve improved enhancement, yet most methods depend on carefully curated training data pairs, with limited robustness under different scenarios. This paper offers a new perspective for RGB-NIR low-light imaging by incorporating 3D-aware neural modeling. Without using clean RGB supervision, a powerful model can be optimized to implicitly fuse extremely noisy RGB observations with NIR cues in 3D space, effectively recovering clean RGB images. The proposed model obviates the requirement for clean RGB data collection, generalizes across different noise levels. Extensive evaluations on synthetic and real data demonstrate its superiority. Codes available: https://github.com/MyNiuuu/3DarkFusion