Enhancing Photogrammetric Digital Surface Models with Pretrained Diffusion Models and Multimodal Conditioning

AI summary

Overview

Research area: Remote sensing / geospatial computer vision, specifically the adaptation of large pretrained image diffusion models (Stable Diffusion 3) to elevation-map refinement from satellite stereo imagery.

Technical level: Advanced. The paper assumes familiarity with flow matching, rectified flows, latent diffusion, ControlNet conditioning, and transformer architectures.

Scope in one sentence: The paper adapts a text-stream-pruned Stable Diffusion 3 backbone with patch-wise normalization and dual ControlNet conditioning to refine noisy photogrammetric Digital Surface Models toward LiDAR-quality elevation across French cities.

What This Paper Is About

Stereo-photogrammetry from satellite images is a cheap way to build large-scale 3D elevation maps (Digital Surface Models, or DSMs), but the results contain noise, outliers, and voids. Aerial LiDAR gives far more accurate elevation, but at much higher cost and with limited availability.

This paper's goal is to take an already vertically calibrated, noisy stereo DSM plus Pléiades-HR optical imagery and use a pretrained diffusion model to correct local elevation errors, approaching LiDAR quality without acquiring LiDAR.

Key Contributions

1. Patch-wise normalization. A scale-and-offset scheme (with offset u and scale s = std/0.25) that stabilizes training when adapting a natural-image pretrained model to elevation data whose mean elevation and local variability vary widely between samples, and that restores predictions back into physical units at inference.

2. A pruned, image-only SD3 architecture. The text stream of the SD3 Multimodal Diffusion Transformer is removed — joint text-image cross-attention is converted to image self-attention, text-specific projection matrices, feed-forward networks, LayerNorms and modulation layers are dropped, and CLIP-L, CLIP-G, T5-XXL and the text pooled embedding are discarded — reducing the transformer from approximately 2B to 1B parameters while keeping pretrained image-stream weights.

3. Multimodal ControlNet conditioning. Two separate ControlNets, one for the photogrammetric DSM and one for Pléiades RGB imagery, whose residuals are summed element-wise and injected into the pruned backbone, evaluated as DSM-only versus DSM + RGB configurations under the same paired protocol.

4. A multi-city French benchmark and evaluation protocol. A dataset built from LiDAR-HD (20 French cities), Pléiades-HR imagery and CARS stereo DSMs (9 cities), with Bordeaux held out as a geographically isolated in-country test city, plus land-cover stratification via Theia Land Cover 2021.

Main Findings

• Multimodal conditioning reduces elevation error. Dense Urban RMSE fell from 6.00 to 3.45 m across the in-context (eight French test) cities, and from 4.16 to 2.77 m in the geographically held-out city of Bordeaux.

• Pretrained initialization beats training from scratch. Under the same pruned architecture, the pretrained configuration reached an FD DINOv2 of 169.4 versus 692.8 for the best scratch configuration. The scratch sweep covered learning rates {1, 2, 5} × 10⁻⁵ and {1, 2, 4} × 10⁻⁴, with 2 × 10⁻⁴ performing best. The paper states this is not a training-seed study, and FD DINOv2 is a relative feature-distribution distance, not an elevation error in meters.

• Pruning improves inference speed. Removing the text stream yields approximately 1.4× higher inference throughput at batch size 32 in a synthetic 50-step denoising and VAE-decoding benchmark, both with and without ControlNets.

• The pretrained VAE introduces modest distortion. A round-trip diagnostic over 9,594 LiDAR patches from the in-context and Bordeaux test sets (normalize, duplicate to three channels, encode/decode, denormalize) gave meter-space bias of −0.004 m, MAE 0.272 m, and RMSE 0.573 m; patch-averaged edge MAE was 0.709 m and gradient MAE 0.288 m/pixel, with edges defined from the top 10% of target gradient magnitudes per patch. The VAE was not fine-tuned, and the paper notes the VAE reconstruction is part of the evaluated pipeline.

• Patch-wise normalization was stable in practice. No epsilon, clipping, or zero-variance fallback was used, and no instability attributable to patch-wise scaling was observed in the reported runs. The minimum raw standard deviation in the scanned LiDAR support was 0.00473 m.

• Results after Section 3.3 are not available in the provided content. The supplied text stops mid-sentence at the start of Section 3.3 ("We compare"), so the full land-cover stratification results table and any subsequent analyses are not reported here.

Methodology in Plain English

The authors treat elevation refinement as a generative transport problem. Starting from Gaussian noise, a neural network learns a velocity field that moves samples toward realistic LiDAR-like elevation patches, using the rectified-flow / multisample flow matching formulation with a Joint Conditional Flow Matching objective.

The key trick is patch-wise normalization. Terrain patches have small internal elevation variation compared with the huge elevation range across a whole dataset (mountains versus coastlines), so training on raw meters is numerically awkward. For each patch, the authors compute an offset from the patch mean and a scale from the patch standard deviation divided by 0.25, normalize the target and the inputs, train in that normalized space, and then denormalize at the end by multiplying by the scale and adding the offset back. During LiDAR-only backbone training these statistics come from the LiDAR patch; during conditional training and inference they are read from the validated pixels of the stereo DSM (inference needs only the DSM). Voids are marked with a sentinel value of −1.

Architecturally, they take Stable Diffusion 3 Medium, throw away the text side to make it an image-only generator, and feed a single-channel normalized elevation patch duplicated across three channels so the pretrained VAE can process it. Two ControlNets — one taking the stereo DSM, one taking Pléiades RGB imagery — inject residuals into the transformer blocks. Scale and offset are embedded with sinusoidal encoders plus MLPs and added to the timestep embedding.

Data comes from LiDAR-HD point clouds (median-aggregated into 0.5 m rasters, with connected NoData components below an 800-pixel sieve threshold filled iteratively from a radius-3 neighborhood using the mean of values below the local 40th percentile for up to five iterations), Pléiades-HR orthoimagery, and already-processed CARS stereo DSMs. Everything is reprojected to Lambert93 at 0.5 m and cut into 512 × 512 patches with stride 512. Acquisition-wide vertical offsets were estimated by histogramming valid DSM–LiDAR differences over ±100 m in 4096 equal-width bins and subtracting the modal bin center. The authors state that the model assumes a vertically co-registered DSM (assumption A1) and surface-compatible DSM/LiDAR acquisitions (assumption A2); they retain only patches whose calibrated RMSE falls below the city-specific 90th percentile and do not verify A2 directly.

Training proceeds in stages on NVIDIA A100 GPUs: backbone adaptation (50k steps, batch 120, learning rate 1 × 10⁻⁵ from pretrained weights, or 2 × 10⁻⁴ from scratch), DSM-only ControlNet training (20k steps at batch 120 then 30k at batch 360, backbone frozen), and sequential multimodal training (Pléiades ControlNet first, then frozen while the DSM ControlNet is added). The paper states no controlled ablations were run on training order, joint training, or alternative schedules.

Why This Matters

Impact on research. The work shows that visual representations learned from large-scale natural-image pretraining transfer to the structurally different domain of metric elevation-map generation, and that several geospatial modalities can be combined in one generative model. The patch-wise normalization scheme is presented as a general, practical recipe for adapting diffusion and flow-matching models to physical-valued raster data whose offsets and dynamic ranges vary substantially between samples — potentially relevant to any raster domain with absolute units.

Real-world applications:

• Updating large-scale 3D mapping where LiDAR coverage is stale, expensive, or unavailable, using optical satellite imagery that is already being acquired routinely.
• Urban planning and building-height analysis, where the reported Dense Urban improvements (6.00 → 3.45 m in-context, 4.16 → 2.77 m in Bordeaux) matter most.
• Flood and terrain modeling that depends on surface elevation accuracy but cannot always afford airborne LiDAR.
• Secondary void recovery in elevation data, which the paper lists as an optional capability of the approach.

Industry relevance. The work is a collaboration between Thales and CNES and was published in Remote Sensing, indicating direct relevance to defense, space-agency, and geospatial-service providers who need cost-effective elevation products. The pruning result (2B to 1B parameters, roughly 1.4× throughput gain) is a practical deployment consideration, and the reliance on off-the-shelf pretrained weights means the approach does not require training a large generative model from scratch.

Future Directions

1. Generalization beyond the current setting. The paper explicitly leaves transfer to other countries, sensors, processing chains, and terrain types for future evaluation; only French urban areas were tested.

2. Quality control for paired data. The 90th-percentile RMSE curation was chosen empirically and not ablated or optimized. The authors call for more sophisticated pipelines to identify high-quality DSM–RGB–LiDAR triplets, motivated by cases where assumption A2 (surface compatibility) is plausibly violated, particularly seasonal vegetation change.

3. Architecture and training ablations. The element-wise residual sum for ControlNet fusion was not compared against alternative fusion mechanisms, and training order, joint training, and alternative schedules were not ablated — all listed as future work, along with a training-seed study.

4. Beyond the synthetic 50-step benchmark. The 1.4× throughput comparison comes from a synthetic benchmark; broader efficiency characterization and any additional evaluation beyond Section 3.3 would be needed, and the content provided does not report them.

Target Audience

This paper is best suited to researchers and engineers with prior exposure to diffusion or flow-matching models and remote sensing pipelines. Primary beneficiaries include geospatial researchers working on DSM/DEM enhancement and generative terrain modeling; machine learning practitioners interested in domain transfer from natural images to scientific raster data; and applied scientists at space agencies, defense organizations, and mapping services who need cost-effective alternatives to airborne LiDAR. Readers seeking an introductory treatment of diffusion models or of photogrammetry fundamentals will find the presentation dense.

Authors’ abstract

Large-scale Digital Surface Models (DSMs) can be produced cost-effectively from satellite images via stereo-photogrammetry. However, the resulting 3D maps are often contaminated by noise, outliers, and voids. On the other hand, aerial LiDAR provides high-accuracy elevation measurements at a substantially higher cost. In this work, we study diffusion models conditioned both on photogrammetric DSMs and Pléiades imagery to refine vertically co-registered DSMs. We introduce a modified Stable Diffusion 3 architecture with a pruned text stream and a patch-wise normalization strategy, enabling stable training on LiDAR data and transfer from natural images to elevation maps. Experiments in French cities demonstrate that multimodal conditioning improves elevation accuracy, reducing Dense Urban RMSE from 6.00 to 3.45 m in the in-context cities and from 4.16 to 2.77 m in the held-out city of Bordeaux.