Georeferenced Cross-View 3D Reconstruction from Ground and Satellite Images

AI summary

Overview

Research area: Computer vision — cross-view geo-localization combined with multi-view 3D reconstruction.

Technical level: Advanced (assumes familiarity with camera pose estimation, point maps, transformer architectures, and feed-forward 3D reconstruction models such as VGGT).

Scope: The paper introduces X-VGGT, a feed-forward transformer that takes a set of ground-level images plus one georeferenced satellite image and outputs camera poses, dense 3D point maps, and a scene-level transform that places the whole reconstruction in the satellite's geographic coordinate frame.

What This Paper Is About

Existing systems can either estimate roughly where a single ground photo was taken (cross-view geo-localization) or reconstruct 3D geometry from multiple ground photos in an arbitrary local coordinate frame — but not both at once. This paper asks whether a single model can take several ground images and a georeferenced satellite image and return a metric 3D reconstruction in which every ground pixel and every camera center has a real geographic location. The proposed X-VGGT model does exactly this, predicting the local 3D scene and the transform that aligns it to the satellite plane jointly rather than in separate stages.

Key Contributions

1. A joint cross-view 3D reconstruction model. X-VGGT is a feed-forward network that processes ground images and a georeferenced satellite image together, predicting ground camera extrinsics and intrinsics, dense metric 3D point maps, and a satellite-to-ground alignment transform that geo-localizes every ground image pixel.
2. A Satellite Alignment Head (SAH) with dedicated georeferencing losses. The head predicts a scene-level transform parameterized by a gravity vector, a yaw vector, a logarithmic scale term, and a 2D translation. Three new losses supervise the transform parameters directly, the projected dense point maps, and the projected camera centers.
3. Flexible ground-view inputs. The model handles ordered video sequences, unordered image sets, and perspective views cut from panoramas, and grounds all of them in the satellite coordinate frame.
4. New training and evaluation protocols. The authors build a satellite-augmented training set (Sat-MegaDepth, derived from Aerial-MegaDepth) and define an evaluation protocol for camera geo-localization, pixel-wise geo-registration, and local 3D reconstruction across four cross-view datasets.

Main Findings

• Best camera geo-localization among individual baselines. Averaged across KITTI, Mapillary Metropolis, WRIVA, and ACC-NVS1, X-VGGT reaches 22.38 m camera-center error and 30.93° orientation error. Per-dataset results are 24.62 m / 22.50° (KITTI), 14.35 m / 17.59° (Metropolis), 26.05 m / 40.64° (WRIVA), and 24.49 m / 42.99° (ACC-NVS1).
• Large gains over a two-stage combination. A stronger baseline that aligns VGGT's reconstruction to locations predicted independently by Loc² scores 31.46 m / 81.16° on KITTI and 37.06 m / 89.19° on Metropolis. X-VGGT reduces localization error by 6.84 m on KITTI and 22.71 m on Metropolis, and orientation error from 81.16° to 22.50° and from 89.19° to 17.59° respectively.
• Localization-only methods get position roughly right but orientation badly wrong. FG² reports 39.81 m / 89.00° on KITTI and Loc² 32.00 m / 69.99°; reconstruction-only baselines such as VGGT and Aerial-MegaDepth get much larger position errors (VGGT: 130.05 m on KITTI, 408.13 m on Metropolis) despite sometimes lower orientation error (Aerial-MegaDepth: 24.68° on KITTI).
• Best dense pixel geo-registration. On KITTI, X-VGGT achieves 25.52 m pixel error versus 43.96 m for Loc² (an improvement of 18.44 m over the strongest baseline). On Metropolis it achieves 17.84 m versus 50.54 m for VGGT + Loc² (an improvement of 32.70 m).
• Two-stage alignment can hurt dense accuracy. VGGT + Loc² improves over standalone Loc² on Metropolis (50.54 m vs 88.96 m) but worsens dense registration on KITTI (178.45 m vs 43.96 m), showing that fitting camera centers to predicted GPS locations does not guarantee accurate dense geo-registration.
• Local 3D reconstruction quality is preserved. On joint RRA/RTA AUC up to 30°, X-VGGT scores 83.90 (KITTI), 51.34 (Metropolis), 21.17 (WRIVA), and 72.91 (ACC-NVS1), close to or slightly better than VGGT (83.29, 55.31, 16.90, 67.96) and far above MASt3R (6.64, 5.05, 0.03, 6.25).
• Each georeferencing loss helps. On KITTI, supervising only the transform parameters gives 30.35 m camera error, 32.87° orientation error, and 33.17 m pixel error; adding dense projection supervision gives 27.95 / 34.49 / 30.91; adding camera-center supervision gives the best result at 27.24 / 30.73 / 29.45.
• More ground views help. On Mapillary Metropolis, both localization and orientation errors decrease as the number of input ground frames increases.

Methodology in Plain English

The model takes N ground images plus one orthorectified satellite image (with known meters-per-pixel) as a single input batch. All images pass through a shared visual encoder initialized from DINOv2, producing patch tokens with embedding dimension 1024. Extra learnable "camera" and "register" tokens are attached to each view, with separate token sets for the first ground view, the remaining ground views, and the satellite view.

The transformer alternates between two attention operations, following the VGGT design: frame attention updates tokens within a single image, and global attention lets all views exchange information. This is what allows the satellite image to act as context for the ground reconstruction.

The satellite image is treated differently from a camera. It has no perspective pose, focal length, or depth map to predict. Instead, its token is read by the Satellite Alignment Head, which outputs a single scene-level transform. That transform works in four parts: a gravity direction that defines the ground plane, a yaw rotation within that plane, a scale factor (stored as a logarithm) that converts reconstruction units to real meters, and a 2D translation in normalized satellite coordinates. In effect it is an orthographic projection from the local reconstruction frame down onto the satellite map. The head runs four refinement steps, each time feeding back the previous estimate and reading only the satellite token.

Ground geometry is handled by two heads copied from the VGGT design: a camera head that predicts a 9-dimensional pose encoding per view (3D translation, a 4D quaternion, and 2D field-of-view parameters), and a dense prediction transformer (DPT) head that outputs point maps and confidence maps.

Training data consists of calibrated ground images, depth-derived point maps, camera poses in an East-North-Up frame, and a satellite image. Because the model predicts in a normalized local frame, all georeferencing targets are recomputed in that same frame before supervision. Gravity comes from the reference camera's rotation; yaw, scale, and translation are then solved for by minimizing the distance between projected camera centers and their true satellite coordinates. The total loss combines camera, point-map, transform-parameter, camera-center, and dense projection terms, with the camera loss weighted at 5.0 and all other terms at unit weight. Training uses KITTI plus Sat-MegaDepth, a version of Aerial-MegaDepth augmented with orthorectified satellite imagery; the mix of a driving dataset and scattered MegaDepth-style scenes teaches the model to handle both ordered sequences and unordered sets.

Why This Matters

Impact on research. The paper reframes geo-localization and 3D reconstruction as one joint problem rather than two pipelines glued together. Its ablation and baseline comparisons show that combining a 2D localizer with a 3D reconstructor in sequence is not equivalent to solving the joint task — the two-stage approach can even degrade dense accuracy. The released training and evaluation protocol gives the field a way to measure georeferenced reconstruction rather than only single-camera GPS error.

Real-world applications:

• Localization in GPS-denied or GPS-degraded environments, where a robot or vehicle must place itself on a map from imagery alone.
• Disaster response, where post-event ground photos could be localized and dropped directly onto a satellite map of the affected area.
• Infrastructure inspection, where observed defects could be assigned real-world coordinates on a map rather than a vague description.
• Mapping from crowd-sourced ground imagery, since the method needs only ground photos plus an existing georeferenced satellite image.

Industry relevance. Autonomous navigation, mapping and GIS providers, insurance and asset-management surveying, and any operation that already owns high-resolution satellite imagery but lacks ground truth for ground-level capture. Because the method requires no auxiliary sensors or drone views, only ordinary cameras plus an existing satellite tile, deployment barriers are relatively low — provided accurate orthorectified imagery is available.

Future Directions

• Robustness to poor satellite imagery. The authors note performance may degrade when the satellite image is outdated, low resolution, or occluded, or when it is visually inconsistent with the ground views — handling these cases is an open problem.
• Generalizing beyond the trained crop scale. The model was trained on satellite crops roughly 150–250 m across. Applying it to much smaller or larger regions would likely require fine-tuning or a new crop protocol.
• Dropping the orthorectified-satellite assumption. Removing the dependence on a known spatial resolution and an idealized nadir map would broaden applicability to oblique aerial imagery and imperfect map products.
• Extending the scene-level transform. The current ground-to-satellite map is a single rigid-plus-scale alignment with a gravity constraint; whether more expressive alignment models help in terrain with significant elevation change is untested here.

Target Audience

Researchers and graduate students in computer vision working on geo-localization, structure-from-motion, feed-forward 3D reconstruction, or vision-based localization. It is also relevant to practitioners in robotics, autonomous driving, and GIS who need to place ground-level imagery into geographic coordinates, and to readers already familiar with DUSt3R/VGGT-style point-map models who want to see how far that paradigm extends to extreme viewpoint changes.

Authors’ abstract

Georeferenced 3D reconstruction from ground images requires recovering not only the local scene geometry, but also localizing every camera center and 3D point in a geographic coordinate frame. Existing cross-view geo-localization methods are able to estimate the GPS coordinate of a single ground camera, but do not reconstruct dense 3D geometry, while recent feed-forward multi-view 3D reconstruction models predict point maps and camera poses only in a local coordinate frame. We introduce X-VGGT, a feed-forward cross-view geometry transformer that unifies these tasks by jointly processing a set of ground-level images and a georeferenced satellite image to predict ground-view camera poses, dense 3D point maps, and a scene-level transform parameterized by gravity, scale, yaw and translation. This transform projects the local reconstruction into the satellite plane, resulting in the geo-registration of all ground images with the georeferenced satellite image. We further propose an evaluation protocol for this joint task and show that, across multiple cross-view datasets, X-VGGT produces accurate georeferenced reconstructions and outperforms prior cross-view geo-localization and multi-view 3D reconstruction models.