Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagery

AI summary

Overview

Research area: Computer vision, specifically 3D scene reconstruction and generative modeling for urban environments, combining 3D Gaussian Splatting (3DGS) with text-to-image diffusion priors.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, NeRF-based novel-view synthesis, rational polynomial camera (RPC) models for satellite imagery, diffusion models, score/flow-based editing, and iterative dataset update (IDU) pipelines.

Scope (one sentence): The paper presents a two-stage framework that turns multi-view satellite imagery alone into real-time, freely navigable 3D city-block scenes by pairing satellite-specific 3DGS reconstruction with curriculum-driven diffusion refinement of degraded renders.

What This Paper Is About

Reconstructing photorealistic, explorable 3D cities normally requires large-scale 3D scans or massive internet image collections, both of which are expensive or noisy to obtain. Satellite imagery is abundant and high-resolution, but it has limited parallax and leaves building facades largely invisible, so naive 3D reconstruction produces floaters, blur, and incorrect geometry at ground level. The paper's goal is to synthesize immersive, geometrically consistent 3D urban scenes from satellite imagery alone, using an open-domain diffusion model as an external source of visual priors to "fill in" what the satellites never saw.

Key Contributions

1. Skyfall-GS framework. The authors claim the first method to synthesize immersive, real-time, free-flight navigable 3D urban scenes solely from multi-view satellite imagery, with no domain-specific 3D training data and no human intervention.
2. Satellite-specific 3DGS reconstruction. A reconstruction stage that pairs entropy-based opacity regularization with pseudo-camera depth supervision, plus illumination-adaptive appearance modeling (per-image and per-Gaussian embeddings with an MLP predicting affine color transforms). The authors state they are the first to show these designs tame the limited-parallax, multi-date satellite setting.
3. Curriculum-based iterative dataset update (IDU) for synthesis. A high-to-low elevation schedule coupled with multi-sample consensus that distills open-domain diffusion priors into photorealistic, cross-view-consistent facades in occluded regions.
4. Empirical validation across two datasets. Comparisons against satellite-reconstruction baselines (Sat-NeRF, EOGS, CoR-GS, Mip-Splatting) and city-generation baselines (CityDreamer, GaussianCity), plus ablations, user studies, scalability tests, and DSM-based geometric evaluation against DFC2019 LiDAR data.

Main Findings

• Large distribution-metric gains on DFC2019. Skyfall-GS reaches FID_CLIP 27.03 and CMMD 2.110, versus Sat-NeRF (86.52 / 4.788), EOGS (87.67 / 5.291), CoR-GS (84.95 / 5.692), and Mip-Splatting enhanced with the authors' appearance modeling (86.72 / 5.404).
• Best distribution metrics on GoogleEarth. The method reports FID_CLIP 10.29 and CMMD 1.959, compared with Mip-Splatting (16.09 / 2.086), CoR-GS (26.35 / 3.758), GaussianCity (28.76 / 2.915), and CityDreamer (36.66 / 4.200).
• Pixel-level metrics are mixed and reported as secondary. On GoogleEarth, Skyfall-GS reaches PSNR 14.42, SSIM 0.302, LPIPS 0.393; Mip-Splatting has slightly better LPIPS (0.379) and a slightly higher PSNR (14.13 vs 14.42 is lower for Mip-Splatting; Mip-Splatting's LPIPS is better). The authors note CoR-GS occasionally achieves higher SSIM on DFC2019 and attribute this to overly smooth reconstructions inflating SSIM. DFC2019 pixel metrics are reported per scene only in the supplementary material, which is not included in the provided content.
• Strong user-study preference. Two user studies with 44 participants each, 4 scenes per study, report win rates of approximately 90–94% on DFC2019 and approximately 79–82% on GoogleEarth across geometric accuracy, spatial alignment, and overall perceptual quality.
• Reconstruction-stage ablation (JAX_068). With appearance modeling only: FID_CLIP 41.90, CMMD 2.450, MAE 3.542 m, RMSE 5.218 m. Adding opacity regularization: 39.95 / 2.395 / 2.980 m / 4.527 m. Adding depth supervision as well: 38.01 / 2.307 / 2.250 m / 3.483 m. Running without any of the three components failed entirely.
• Synthesis-stage ablation. N_s = 1 gives FID_CLIP 34.11 and CMMD 3.189 at 3.44 h; the adopted N_s = 2 gives 28.35 / 2.875 at 6.37 h; N_s = 3 gives 28.64 / 2.769 at 7.19 h; N_s = 5 gives 29.17 / 2.677 at 9.80 h (lowest CMMD but 1.5x longer with marginal gains). Random elevation sampling instead of the curriculum gives 33.79 / 3.361; reversing the curriculum gives 53.03 / 4.170; a context-free prompt gives 30.78 / 2.981; replacing refinement with SDEdit gives 64.74 / 4.138.
• Runtime. On the JAX_214 AOI with a single NVIDIA RTX A6000 (48GB), the full pipeline takes approximately 6 h 45 min: 1 h 35 min for reconstruction and 5 h 10 min for synthesis. The paper compares this to Sat-NeRF/EO-NeRF (reported 10–20 h) and Mip-Splatting (1 h 38 min on the authors' hardware).
• Real-time rendering after fusion. Once learned appearance is fused into a standard 3DGS representation, the model renders at 60 FPS at 1920x1080 on a MacBook Pro M4 Pro.
• Multi-block scalability. Combining adjacent AOIs JAX_214 and JAX_260 yields a seamless scene of approximately 1 km x 512 m trained in about 9 h on one RTX A6000, with no stitching artifacts on shared structures and roughly 36–38% lower FID_CLIP than either single-AOI model on a boundary-spanning orbit.

Methodology in Plain English

The pipeline has two stages.

Reconstruction. Satellite images come with rational polynomial camera models, which are not directly usable by 3DGS, so the authors use SatelliteSfM to approximate perspective camera parameters and generate sparse initial points. Because the input images span multiple dates, global illumination, seasons, and transient objects vary; the authors follow WildGaussians and learn a per-image embedding plus a per-Gaussian embedding, fed through a small MLP that predicts an affine (scale, shift) color transform, with spherical harmonics limited to zero- and first-order terms. At inference everything is fused into a fixed static color, so the model stays compatible with standard real-time 3DGS renderers. To fight floaters from limited parallax, they add an entropy-based opacity regularizer that pushes Gaussians toward binary opacity (so weak ones get pruned), and they supervise depth from pseudo-cameras placed closer to the ground using a monocular depth estimator (MoGe) with a Pearson-correlation loss. The total reconstruction loss is a color term plus weighted opacity and depth terms (with lambda values of 0.2 for D-SSIM, 10 for opacity, and 0.5 for depth over 30,000 iterations).

Synthesis. Because renders degrade as the viewing angle drops toward the ground, the authors run an iterative dataset update over N_e = 5 episodes of 10,000 iterations each. In each episode they render views along orbital trajectories around N_p look-at points, with N_v = 6 cameras per point, using an elevation schedule that descends from high to low. Those renders are refined by prompt-to-prompt editing with FlowEdit and the FLUX.1 [dev] diffusion model, using source prompts describing degraded features and target prompts describing desired high-quality attributes. To avoid the cross-view inconsistency of independent 2D denoising, they generate N_s = 2 refined samples per view and let the photometric loss implicitly average over them, finding a consensus 3D representation. Each episode retrains the 3DGS only with color and depth losses on a training set sampled 75% from IDU-refined views and 25% from original satellite views, preserving fidelity to the satellite input.

Why This Matters

Impact on research. The paper attacks a specific gap: existing satellite 3D reconstruction methods (Sat-NeRF, EOGS, SkySplat, and others) do not recover occluded facades, and layout-conditioned city generators (InfiniCity, CityDreamer, GaussianCity) rely on semantic or BEV map assumptions and overfit to small domain-specific datasets. By showing that open-domain diffusion priors can be constrained by satellite-observed facades rather than freely hallucinating them, the work suggests a route to scalable 3D world creation that avoids costly 3D annotation and physical data collection.

Real-world applications (as listed in the paper):

• Gaming and filmmaking, where large, 3D-grounded environments support rendering and storytelling.
• Navigation planning and drone-view navigation, using immersive exploration of real city blocks.
• Robotics and embodied physics simulation, which need navigable, semantically plausible 3D environments.
• Virtual entertainment and simulation more broadly, as an alternative to hand-built 3D assets.

Industry relevance. The method consumes commercially available satellite imagery (the paper cites Maxar WorldView-3 capturing roughly 680,000 km² daily at up to 31 cm/pixel) and outputs standard 3DGS assets that render in real time on consumer hardware (60 FPS at 1920x1080 on a MacBook Pro M4 Pro). That combination matters for geospatial, mapping, simulation, and virtual production pipelines, since it turns an existing imagery archive into explorable 3D content with a one-time, per-scene offline training cost.

Future Directions

• City-wide scaling. The conclusion names city-scale expansion as a promising direction; the largest demonstrated result here is the approximately 1 km x 512 m combined two-tile scene, so how far the approach stretches before partitioning or level-of-detail machinery becomes necessary is not reported.
• Dynamic scene modeling. Also named in the conclusion, but no dynamic or time-varying results appear in the provided content.
• Pure nadir input. The authors state that synthesizing facades requires off-nadir views (WorldView spans roughly ±40°), and that purely nadir satellite input remains an open challenge.
• Street-level fidelity. The authors explicitly place strict pedestrian-level synthesis (below 10°, facade-close) beyond scope, citing transient objects across multi-date captures and Google Earth Studio reference distortion at low elevations; closing that gap would require both better reconstruction and better evaluation.
• Camera trajectory design. The fixed heuristic trajectory "works well in general but may leave blind spots in complex geometries," occasionally causing minor artifacts at extreme street-level views, so learned or geometry-aware scheduling is a natural next step.

Target Audience

Researchers and practitioners in 3D computer vision who work on novel-view synthesis, 3D Gaussian Splatting, NeRFs, and generative 3D content creation, particularly those interested in remote sensing, aerial and satellite reconstruction, and cross-view synthesis. It is also relevant to applied teams in geospatial analytics, simulation, and virtual production who want to convert satellite imagery archives into navigable 3D environments. Readers without background in 3DGS, RPC camera models, and diffusion-based editing will find the method sections demanding, though the problem framing and results tables are approachable.

Authors’ abstract

Synthesizing large-scale, explorable, and geometrically accurate 3D urban scenes is a challenging yet valuable task for immersive and embodied applications. The challenge lies in the lack of large-scale and high-quality real-world 3D scans for training generalizable generative models. In this paper, we take an alternative route to create large-scale 3D scenes by leveraging readily available satellite imagery for realistic coarse geometry and open-domain diffusion models for high-quality close-up appearance synthesis. We propose Skyfall-GS, a novel hybrid framework that synthesizes immersive city-block scale 3D urban scenes by combining satellite reconstruction with diffusion refinement, eliminating the need for costly 3D annotations, and also featuring real-time, immersive 3D exploration. We tailor a curriculum-driven iterative refinement strategy to progressively enhance geometric completeness and photorealistic texture. Extensive experiments demonstrate that Skyfall-GS provides improved cross-view consistent geometry and more realistic textures compared to state-of-the-art approaches. Project page: https://skyfall-gs.jayinnn.dev/