Dark3R: Learning Structure from Motion in the Dark

AI summary

Overview

Research area: Computer vision — low-light 3D reconstruction, structure from motion (SfM), and neural rendering.

Technical level: Advanced. The paper builds directly on the MASt3R / MASt3R-SfM vision-transformer architectures, low-rank adaptation (LoRA), NeRF-style radiance fields, and raw sensor noise modeling, and assumes familiarity with bundle adjustment and epipolar geometry.

Scope: One sentence: the paper introduces a teacher–student distillation framework that adapts a pretrained 3D foundation model to raw images with signal-to-noise ratios below −4 dB, together with a new exposure-bracketed raw image dataset and a coarse-to-fine radiance field procedure for novel view synthesis in the dark.

What This Paper Is About

Passive 3D reconstruction — recovering camera poses and scene geometry from images alone — works reliably in good lighting but collapses in extreme low light, where sensor noise dominates the signal and existing feature detectors, matchers, and learned correspondence models produce spurious matches. The goal of this work is to make structure from motion work directly on noisy raw images in the very low-SNR regime, without any 3D supervision, and then to use the recovered poses for novel view synthesis of the same dark scene.

Key Contributions

1. Dark3R, a teacher–student distillation framework for low-light SfM. The method aligns the dense feature maps produced by a frozen MASt3R teacher on well-exposed raw image pairs with those produced by a student model on low-light raw image pairs, then uses the student's correspondences in the global reconstruction and bundle adjustment pipeline of MASt3R-SfM. It requires no 3D supervision — only noisy–clean raw image pairs, which can be captured directly or synthesized with a Poisson–Gaussian noise model applied to well-exposed raw images.

2. A new exposure-bracketed multi-view raw image dataset. Roughly 42,000 multi-view raw images with accurate 3D annotations, plus a further set of roughly 20,000 high-SNR multi-view raw images across nearly 100 different scenes, captured with a Sony Alpha I camera. Reference 3D annotations are derived from the high-SNR exposures.

3. A robust view synthesis procedure for the dark. A coarse-to-fine radiance field optimization (Dark3R-NeRF) that uses Dark3R's predicted poses and point maps, with depth supervision, stochastic preconditioning, and no black-level subtraction or clipping, to render novel views from noisy raw images.

4. Demonstrations of state-of-the-art pose estimation and novel view synthesis in the low-SNR regime, along with ablations of the distillation design, and applications including generalization to an iPhone 16 sensor without retraining and high dynamic range rendering from raw inputs.

Main Findings

• Pose estimation: Across the evaluated scenes, Dark3R achieves the best pose and depth metrics in Table 1 (average SNR of −3.87 dB, 120 input images per scene): ATE 0.050, RPE translation 0.020, RPE rotation 0.121, AbsRel 0.091, and δ<1.25 of 93.14. The strongest baseline, MASt3R-SfM, reaches ATE 0.088, RPE T 0.038, RPE R 0.201, AbsRel 0.196, and δ<1.25 of 79.39.

• Baselines degrade faster than Dark3R: VGGT scores ATE 0.252 and δ<1.25 of 63.28 on the same setting; COLMAP scores ATE 0.669 and δ<1.25 of 54.38; MASt3R alone scores ATE 0.787 and δ<1.25 of 39.66. Performance for all baselines drops sharply as SNR falls below 0 dB, while Dark3R degrades more slowly.

• Scaling to full image sets: With the full set (330 to 500 images) per scene, Dark3R (Full) reports ATE 0.139, RPE T 0.019, RPE R 0.121, AbsRel 0.093, δ<1.25 of 92.33, versus MASt3R-SfM (Full) at ATE 0.206, RPE T 0.039, RPE R 0.218, AbsRel 0.194, δ<1.25 of 74.95. The paper notes that average pose accuracy worsens slightly in this setting as the pipeline relies more on bundle adjustment than on the network's learned prior.

• Novel view synthesis: Dark3R-NeRF paired with Dark3R poses achieves PSNR 36.17, SSIM 0.866, LPIPS 0.257, outperforming RawNeRF with Dark3R poses (PSNR 34.24, SSIM 0.848, LPIPS 0.291), LE3D with Dark3R poses (PSNR 35.77, SSIM 0.878, LPIPS 0.339), and Dark3R-NeRF with MASt3R-SfM poses (PSNR 34.60, SSIM 0.835, LPIPS 0.308). An "oracle" configuration using poses from clean reference images reaches PSNR 37.16, SSIM 0.882, LPIPS 0.228 as an upper bound.

• Noise regime characterized: The framework targets image SNRs below −4 dB. Figure 2 shows SuperGlue and MASt3R performing reliably in well-lit conditions but worsening below −3 dB, whereas Dark3R retains correspondences in both regimes. Ablations of the radiance field show depth supervision, removal of black-level clipping, and stochastic preconditioning each contribute to the final PSNR of 36.17, versus 34.91 without depth supervision, 34.98 with black-level clipping, and 36.05 without stochastic preconditioning.

• Distillation design choices matter unevenly: Full fine-tuning instead of LoRA substantially worsens pose errors (ATE 0.476 versus 0.050). Operating on raw images rather than sRGB helps (ATE 0.087 for sRGB), as does supervising with clean raw rather than clean sRGB pairs (ATE 0.064). Training on only simulated noise (ATE 0.078) does reasonably well, and using only captured pairs (ATE 0.063) or both (proposed, ATE 0.050) improves further. Alternative fine-tuning schemes — encoder only (ATE 0.030), encoder and decoder (ATE 0.049), or removing the clean loss (ATE 0.051) — only modestly change performance.

• Generalization: Dark3R is evaluated without retraining on a dataset captured by an iPhone 16, and the paper reports in the supplement that it generalizes well to that sensor's different noise characteristics. The method also supports high dynamic range rendering directly from raw inputs by operating in the linear sensor irradiance domain.

Methodology in Plain English

The starting point is a pretrained 3D foundation model, MASt3R, which takes two images and predicts dense per-pixel feature maps, 3D point maps, and correspondence maps. That model works well on well-exposed images but not on very noisy ones. The researchers therefore set up two copies of the network with identical architecture: a frozen "teacher" that always sees clean, well-exposed raw image pairs, and a "student" that sees noisy versions of the same viewpoints. The student is initialized from the same weights and fine-tuned with low-rank adaptation so that its encoder features, decoder features, and correspondence features match the teacher's, which is a form of distillation. Because the student is also occasionally trained on clean pairs, it does not lose its ability on bright images.

One important detail is that the pipeline operates on raw sensor measurements that have been given only a trivial demosaicing (subsampling each Bayer color channel and averaging the two green channels), rather than on images pushed through a standard camera image signal processing pipeline, which applies non-invertible operations such as black-level subtraction and clipping.

Once trained, the student's predicted correspondences are fed into the same global reconstruction and bundle adjustment procedure as MASt3R-SfM, with the addition of a regularization term that keeps optimized camera intrinsics close to their calibrated values. For rendering, the predicted poses and point maps initialize a NeRF-based radiance field optimized with a coarse-to-fine scheme: Gaussian noise is added to ray sample positions and annealed away over the first 30,000 iterations, optimization continues to 90,000 iterations, and the predicted depth maps are used as an exponentially decaying supervision signal. The model is trained for up to 15 epochs on 8 RTX A6000 GPUs in about 15 hours, with a learning rate of 1×10⁻³ under AdamW and a clean-loss weight of 0.3.

Why This Matters

Impact on research. The paper shows that the strong priors inside large 3D foundation models can be transferred to an imaging regime far outside their training distribution without any 3D ground truth, using only paired noisy–clean images. It also establishes a benchmark dataset and metric suite for a problem — passive 3D reconstruction in extreme darkness — that previously sat below the operating threshold of every mainstream SfM pipeline.

Real-world applications:

• Night-time or extreme low-light capture on smartphones and consumer cameras, where images are often too noisy for conventional SfM even though the raw data exists.
• Robotics and autonomous navigation in dark environments, such as warehouses, tunnels, mines, or outdoor night operation, where passive vision previously required active illumination.
• Cultural heritage, scientific, and archival imaging where scenes must be documented without bright lighting that could damage or disturb the subject.
• High dynamic range and raw-domain photography workflows, since Dark3R operates in linear sensor irradiance and supports HDR rendering from raw inputs.

Industry relevance. The work is directly relevant to camera and sensor manufacturers (Sony Corporation of America is among the affiliations, and the acknowledgments note support from Sony Corporation), to computational photography teams building raw-domain pipelines, and to the developers of 3D foundation models who need their systems to generalize to non-ideal image distributions. The demonstrated transfer to an iPhone 16 sensor without retraining suggests the approach is not tied to a single camera.

Future Directions

• Feed-forward prediction at scale. The authors suggest extending the framework toward feed-forward prediction using large architectures such as VGGT, which would likely require adapting auxiliary encoders like DINO to be compatible with raw, low-SNR images. The paper notes that VGGT currently requires GPUs with more than 48 GB VRAM at scale, which limits direct comparison.

• Dynamic scenes in the dark. The authors propose building on recent work in dynamic 3D reconstruction to enable SfM on moving scenes captured in low light, which the current static-scene assumption does not cover.

• Generative priors for extreme darkness. Integrating generative models is suggested as a route to further improve robustness as SNR drops even lower than the demonstrated range.

• Reducing reliance on calibration. The current inference assumes known camera intrinsics; the paper evaluates performance with and without intrinsics in the supplement, leaving open how the method performs under fully uncalibrated capture.

Target Audience

Researchers and graduate students in computer vision and computational photography working on structure from motion, low-light imaging, or 3D foundation models; engineers building raw-domain camera pipelines or night-time perception systems for robotics and consumer devices; and practitioners who need multi-view 3D reconstruction or novel view synthesis from data that cannot be re-captured under better lighting. Readers need prior exposure to SfM pipelines, vision transformers, and radiance fields to follow the architecture and evaluation details fully.

Authors’ abstract

We introduce Dark3R, a framework for structure from motion in the dark that operates directly on raw images with signal-to-noise ratios (SNRs) below $-4$ dB -- a regime where conventional feature- and learning-based methods break down. Our key insight is to adapt large-scale 3D foundation models to extreme low-light conditions through a teacher--student distillation process, enabling robust feature matching and camera pose estimation in low light. Dark3R requires no 3D supervision; it is trained solely on noisy--clean raw image pairs, which can be either captured directly or synthesized using a simple Poisson--Gaussian noise model applied to well-exposed raw images. To train and evaluate our approach, we introduce a new, exposure-bracketed dataset that includes $\sim$42,000 multi-view raw images with ground-truth 3D annotations, and we demonstrate that Dark3R achieves state-of-the-art structure from motion in the low-SNR regime. Further, we demonstrate state-of-the-art novel view synthesis in the dark using Dark3R's predicted poses and a coarse-to-fine radiance field optimization procedure.