Adapting Stereo Vision From Objects To 3D Lunar Surface Reconstruction with the StereoLunar Dataset

AI summary

Overview

Research area: Computer vision / 3D reconstruction for space robotics — specifically stereo depth estimation and camera pose recovery on the lunar surface.

Technical level: Advanced. The paper assumes familiarity with multi-view stereo, Structure-from-Motion, deep 3D reconstruction networks (MASt3R, DUSt3R, VGGT), and physically based rendering concepts such as BRDF.

Scope: The authors build a photorealistic ray-traced lunar stereo dataset with dense ground truth and use it to fine-tune a general-purpose 3D reconstruction network so it works on the Moon's texture-poor, harshly lit terrain.

What This Paper Is About

Deep learning models that reconstruct 3D geometry from images (like MASt3R) are trained almost entirely on Earth-scale, texture-rich scenes such as cities, indoor rooms, and objects. Applied directly to lunar imagery, they produce flat, noisy, or geometrically meaningless output, because the Moon offers almost no texture, extreme lighting contrast, and descent trajectories with very little parallax. This paper asks whether a targeted, physically realistic synthetic dataset can close that domain gap — and shows that it can, with large gains in both surface shape and relative camera pose accuracy.

Key Contributions

1. LunarStereo — the first publicly released dataset of physically grounded, ray-traced stereo image pairs of the Moon, generated from the LOLA South Pole DEM (5 m/px) with the Hapke BRDF, realistic solar illumination, and three descent-inspired camera motion types.
2. Full geometric supervision — over 50,000 stereo pairs at 512×512 with dense per-pixel depth maps, camera intrinsics and 6-DOF extrinsics in a Moon-fixed frame, stereo baselines, and georeferenced trajectory metadata.
3. Domain adaptation of MASt3R — fine-tuning of the pretrained MASt3R model on ~31,000 curated lunar pairs, demonstrating that a terrestrial 3D foundation model can be adapted to a texture-starved extraterrestrial domain.
4. Terrain-aware evaluation protocol — beyond standard Chamfer/accuracy/completeness metrics, the authors introduce slope correlation, slope MAE, SSIM on depth maps, and central depth-profile correlation to measure whether reconstructions preserve the actual shape of ridges and craters, which matters for landing safety.

Main Findings

• Large reduction in reconstruction error: Chamfer distance drops from 236 m to 100 m for nadir views, from 322 m to 144 m for oblique, and from 279 m to 111 m for dynamic trajectories. Relative errors fall by roughly 50 percent across the board.
• Dramatic improvement in terrain shape fidelity: Slope correlation rises from 0.21 to 0.80 on nadir sequences and from 0.09 to 0.76 on the hardest dynamic case — meaning the network goes from barely relating to the terrain's shape to reproducing it well. The authors report an average reduction of over 70 percent in slope estimation error.
• Better structural consistency: SSIM on depth maps improves from 0.31 to 0.78 (nadir), 0.51 to 0.83 (oblique), and 0.24 to 0.76 (dynamic). Depth-profile correlation reaches 0.95–0.99, indicating relief rises and falls in the right places.
• Pose estimation is competitive, not uniformly better: On rotation accuracy (RRA) the fine-tuned model matches or beats MASt3R and clearly beats a classical ORB baseline. On translation accuracy (RTA) it is generally comparable but still shows limitations, especially in the nadir case — a realistic weakness given the small parallax in near-vertical descents.
• Strongest gains on the hardest configuration: The dynamic trajectory — with altitude variations up to ±30 percent, roll up to ±10°, and random baselines of 5–18 percent of altitude — shows the largest margin over the baseline, suggesting improved robustness to viewpoint and scale variation.
• Generalization to real imagery: Qualitative tests on real Chang'E-3 NavCAM descent footage show the fine-tuned model recovering crater structure and coherent slopes under both nadir and oblique configurations, while pretrained MASt3R aligns the views but produces noisy geometry with no discernible relief.
• Scale-invariance is deliberate: Because the Moon's fractal-like surface makes absolute metric scale ill-defined, training uses the scale-invariant regression loss from DUSt3R, and metric scale is recovered at inference via a RANSAC similarity transform.

Methodology in Plain English

The team could not rely on real lunar images with trustworthy geometry, so they built a simulator instead.

They started with a high-resolution elevation map of the lunar South Pole and rendered it with a physically based ray tracer that models how lunar soil actually reflects sunlight (the Hapke model, which captures effects unique to airless bodies). They placed virtual stereo cameras along three trajectory archetypes modeled on real descent profiles — straight-down nadir, tilted oblique, and a more erratic dynamic case — across ten altitude bands from 3.5 km to 30.5 km, and rendered each under three different Sun positions to produce varied shadow patterns. Every rendered pair comes with exact depth, camera pose, and baseline, because the ground truth is known by construction.

They then took MASt3R — a pretrained network that jointly predicts dense 3D point clouds and matching features — froze its image encoder, and fine-tuned the rest on about 31,000 carefully sampled pairs to cover the full range of altitudes, lighting, and baselines. They applied augmentations (color jitter, cropping, grayscale, bilateral filtering to strip texture) to push the network toward geometry rather than appearance, and trained for 25 epochs on two GPUs.

Evaluation used 3,000 freshly rendered pairs at altitudes interpolated between the training ones, plus standard pose metrics and their new slope- and profile-based terrain metrics. Finally, they ran the model on genuine Chang'E-3 descent video and inspected the results visually, since no ground truth exists for that footage.

Why This Matters

Impact on research: The paper shows that large 3D foundation models are not locked to the visual statistics of their training data — they can be re-aimed at severely out-of-distribution domains with a few tens of thousands of physically coherent synthetic pairs. It also supplies a reusable benchmark where none existed, and legitimizes terrain morphology metrics (slope, profile) as first-class evaluation criteria alongside pointwise distances, which is important in any domain where shape reliability matters more than average error.

Real-world applications:

• Hazard Detection and Avoidance during lunar landing, where slope estimates directly determine whether a landing site is safe.
• Descent and landing trajectory planning for autonomous or crewed missions, using onboard stereo to build local terrain models in real time.
• Topographic map refinement from archival or ongoing orbital imagery, improving on existing DEMs at landing-site scale.
• Adaptation to other planetary bodies — asteroids, Mars analogs, or any environment with weak texture and no GNSS.

Industry relevance: The work is a collaboration between academia (IRIT, Toulouse INP), industry (Airbus Defence and Space), and ESA, funded under an ESA contract. That framing signals direct interest from the European space sector in onboard learning-based perception, where lightweight models must run on constrained flight hardware. The finding that a general-purpose network can be adapted rather than replaced lowers the barrier for space agencies already invested in terrestrial 3D vision stacks.

Future Directions

• Add real texture and orthorectified imagery to the dataset, since the current rendering omits high-frequency surface detail — a known limitation the authors acknowledge could affect realism.
• Fine-tune and compare other architectures, such as DUSt3R and VGGT, to see whether the adaptation recipe is general or specific to MASt3R.
• Mix real and simulated data during training to improve robustness and cross-scale generalization, and to test the sim-to-real gap more rigorously than the current qualitative-only evaluation permits.
• Distill the model for onboard deployment, compressing it to run within the compute and power budget of a spacecraft.
• Extend to other planetary surfaces and validate whether the approach transfers to Mars, asteroids, or Earth environments that also lack texture.

Target Audience

Researchers and engineers working on deep learning for 3D reconstruction, planetary perception, or space robotics; graduate students in computer vision looking for a well-defined domain-adaptation case study; and practitioners in the space industry evaluating whether learning-based stereo can replace or augment classical SfM/MVS pipelines for landing and terrain mapping. Readers without a background in multi-view geometry or photogrammetry will find the methodology sections dense, though the motivation and results are accessible to anyone with a general machine learning foundation.

Authors’ abstract

Accurate 3D reconstruction of lunar surfaces is essential for space exploration. However, existing stereo vision reconstruction methods struggle in this context due to the Moon's lack of texture, difficult lighting variations, and atypical orbital trajectories. State-of-the-art deep learning models, trained on human-scale datasets, have rarely been tested on planetary imagery and cannot be transferred directly to lunar conditions. To address this issue, we introduce LunarStereo, the first open dataset of photorealistic stereo image pairs of the Moon, simulated using ray tracing based on high-resolution topography and reflectance models. It covers diverse altitudes, lighting conditions, and viewing angles around the lunar South Pole, offering physically grounded supervision for 3D reconstruction tasks. Based on this dataset, we adapt the MASt3R model to the lunar domain through fine-tuning on LunarStereo. We validate our approach through extensive qualitative and quantitative experiments on both synthetic and real lunar data, evaluating 3D surface reconstruction and relative pose estimation. Extensive experiments on synthetic and real lunar data validate the approach, demonstrating significant improvements over zero-shot baselines and paving the way for robust cross-scale generalization in extraterrestrial environments.