AutoCompass: Accurate Visual Localization on Public Maps by Learning from Weak Labels

AI summary

Overview

Research area: Computer vision — 3-DoF (2D position plus heading) visual localization by matching a ground-level image against a free, publicly available 2D map such as OpenStreetMap.

Technical level: Intermediate. The task and losses are described clearly, but the paper assumes familiarity with bird's-eye-view (BEV) feature representations, cross-correlation, negative log-likelihood training, and pose marginalization.

Scope in one sentence: AutoCompass is a weak-supervision scheme that trains neural map matchers from noisy GPS positions and relative poses instead of precise geo-referenced 3-DoF labels, and it outperforms strongly supervised baselines across driving and egocentric benchmarks.

What This Paper Is About

Neural map matchers estimate where a camera is and which way it is facing by matching a learned top-down (bird's-eye-view) representation of an image to a 2D map. Training them normally requires hundreds of thousands of images labelled with accurate global position and heading, and automated geo-referencing pipelines are noisy — the paper's Figure 1 shows an MGL training sequence whose pseudo ground-truth poses are offset by several meters, placing images shot on a sidewalk inside a building. The goal of AutoCompass is to stop models from learning and reproducing these label errors by treating the labels as weak rather than exact, while still achieving better localization accuracy than methods trained on the supposedly accurate labels.

Key Contributions

1. An absolute pose loss with built-in error tolerance. It assumes the true image position lies anywhere within a tolerance region around the geo-referenced position label, and it requires no geo-referenced heading label at all.
2. Training from unoptimized GPS position labels alone. Using this loss, a neural map matcher can be trained from raw 2-DoF GPS coordinates, with no optimized 3-DoF pose supervision.
3. Relative pose losses for practical scenarios. Several losses supervise relative rotation, relative translation, and relative distance between training images, covering cases such as non-metric poses or only approximately geo-referenced poses. These improve on existing single-image supervised approaches.
4. A demonstrated accuracy gain. Across driving (KITTI) and egocentric (LaMAria, Oxford Day-and-Night) benchmarks, AutoCompass trained on raw GPS alone outperforms baselines trained with strong absolute pose supervision, and with relative pose supervision it reports the best results among the OSM-based methods compared.

Main Findings

• Heading labels are unnecessary. Maximizing the likelihood of the ground-truth position after marginalizing over all headings is enough for accurate headings to emerge automatically. The authors attribute this to the geometric inductive bias of cross-correlating learned map features with learned BEV features. They note this is related to an observation in C-BEV, but C-BEV used 360-degree panoramic images and positive and negative aerial tiles, whereas AutoCompass uses only single perspective images.
• A tolerance region around GPS improves positional accuracy. The chunk loss maximizes the probability mass of a ±r neighbourhood around the GPS label. The paper reports that r is not a sensitive parameter, with similar performance for r in [5, 20] m, making the loss robust to GPS errors of the magnitude commonly observed with receivers in egocentric devices.
• Relative poses improve the learned absolute pose distribution. Relative poses from SLAM or SfM provide only local constraints but yield a more accurate training signal. Figure 3 shows the relative losses are invariant to translation and rotation offsets shared by the datapoints involved, whereas strong supervision penalizes those same errors.
• On KITTI (driving), weak supervision beats strong supervision. With a ResNet-101 backbone, retrained OrienterNet reaches recall@1/3/5 m of 37.7/67.3/77.4 lateral, 17.6/42.6/53.7 longitudinal, and 15.7/42.1/57.1 orientation. AutoCompass with raw GPS reaches 43.1/78.8/85.6, 26.8/56.1/65.1, and 21.1/53.0/69.2. With relative poses it reaches 56.6/83.2/87.8, 33.2/59.5/66.0, and 28.9/64.4/75.9.
• A DINOv2 image encoder adds further gains. On KITTI with DINOv2, AutoCompass with relative poses reports 70.8/91.1/94.2 lateral, 34.1/70.8/77.6 longitudinal, and 37.9/78.5/88.5 orientation recall — the best among the OSM methods listed, including ones trained on more data.
• Egocentric results follow the same pattern. On LaMAria, single-frame XY recall@1/3/5 m is 1.9/10.9/18.2 for retrained OrienterNet (ResNet-101) versus 8.1/25.2/33.0 for AutoCompass with relative poses. With DINOv2, AutoCompass with relative poses reports 20.9/49.2/56.3 XY and 19.5/47.0/59.5 orientation.
• Sequential fusion benefits from the better single-view estimates. On LaMAria, fusing under 10 frames lets AutoCompass surpass the high recall@5 m of GPS, which benefits less because its measurements are biased; a retrained OrienterNet needs over 20 frames to reach that point.
• Learned features become sharper. The qualitative comparison in Figure 4 shows strongly supervised OrienterNet learning smooth map features that cover coarse structures such as whole buildings, while AutoCompass learns sharper features focused on image-visible keypoints such as building corners.
• Different weak label types all train usable models. Figure 5 (Right) reports that performance correlates with label informativeness across OrienterNet, GPS labels, GPS labels with non-metric relative poses, metric relative poses, and approximately geo-referenced relative poses. The paper's additional analysis of these supervision strategies continues past the truncated text of Section 4.4.

Methodology in Plain English

The authors keep the existing neural map matcher design — the one used by OrienterNet — and change only how it is trained, so the effect of the new supervision can be isolated. The network outputs a probability distribution over a grid of candidate poses: every cell is a possible 2D position on a map tile, crossed with a set of discrete headings.

Instead of demanding that all probability mass sit on one exact labelled pose, the new losses ask the model to place mass anywhere inside a region the authors are willing to trust:

• If only GPS is available, the model must put mass somewhere within a small square neighbourhood around the GPS coordinate, over any heading. Nothing tells it which heading is correct, so the heading must come from the geometry of matching image features to map features.
• If relative poses are available — from SLAM or SfM, which handle noise and outliers with robust optimization — the model's own predicted distributions for two images are combined, and the loss is applied to the relative rotation and the relative shift between them. Because the loss only involves differences between the two images' labels, a constant offset in the geo-referencing of both images cancels out.
• A third loss supervises only the distance between two images, which frees training from needing the relative translation expressed in the same coordinate system as the map tiles; this is useful when poses are non-metric or only approximately geo-referenced.

Training uses the Mapillary Geo-Localization (MGL) dataset: 760k images from 12 cities across Europe and the US, minus Amsterdam (72k images), which was no longer available at the time of writing, leaving 11 cities. Image pairs are sampled from the same SfM cluster and pairs more than 100 m apart are rejected. The experiments use a U-Net architecture with ResNet-101 and VGG-19 backbones, plus a DINOv2 image encoder variant, trained with Adam, batch size 12, typically 500k steps on two 40 GB NVIDIA A100 GPUs (about 3 days).

Why This Matters

The paper argues that the field's assumption of clean geo-referenced ground truth is the bottleneck, not the model architecture: at the scale needed for training, annotation must be automated and will therefore contain errors, and models faithfully learn to reproduce those errors. AutoCompass shows that less confident supervision produces more accurate models, which lowers the cost of building training data and removes the need for heading annotation entirely.

Real-world applications:

• Vehicle and pedestrian navigation where satellite positioning is unreliable due to multipath in urban canyons or deliberate jamming, using maps that are already free and public.
• Augmented reality and wearables, where the paper specifically notes that GPS errors on egocentric devices reach magnitudes its ±r tolerance is designed to absorb.
• Robotics and autonomous driving needing a position and heading estimate that can be fused across a sequence of frames to beat biased GPS.
• Large-scale map-based localization services, since the approach relies on 2D maps (the paper cites roughly 200 KB for a 128x128 m OpenStreetMap area) rather than dense, memory-intensive image-based 3D maps.

Industry relevance: The work was done with Niantic Spatial, with authors from Niantic Spatial, Universidad de Zaragoza and the University of Oxford, and it targets the practical problem of training production localization models from the messy, automatically geo-referenced imagery that large-scale mapping providers actually have.

Future Directions

• Scaling weak supervision to geo-referencing itself. The paper states that geo-referencing at scale remains a challenging problem; whether AutoCompass-style losses can be extended to improve the geo-referencing pipelines that produce the training labels is left open.
• Stronger or larger image encoders. DINOv2 improved results consistently in every comparison shown, inviting exploration of other foundation-model backbones.
• More supervision strategies for practical settings. Section 4.4 analyzes GPS labels, non-metric relative poses, metric relative poses and approximately geo-referenced relative poses; the paper describes its proposal of various relative pose losses "tailored to practical scenarios," suggesting further variants remain to be explored.
• Handling datasets with heterogeneous label quality. The paper notes Xia et al. adapt training to varying levels of informativeness in ground-truth labels; combining that idea with AutoCompass's tolerance-based losses is a natural extension.

Target Audience

Researchers and engineers working on visual localization, geo-localization, and cross-view matching; practitioners who train localization models on crowd-sourced or automatically geo-referenced imagery and need robustness to label noise; and readers interested in weak supervision as an alternative to expensive precise annotation. Familiarity with the OrienterNet formulation of neural map matching will make the method sections considerably easier to follow.

Authors’ abstract

Neural map matchers estimate an image's 3-DoF pose relative to a 2D map. These models are trained on large-scale datasets of geo-referenced images, whose position and heading labels often contain noise that affects the trained models. To address this, we present AutoCompass, a supervision approach for training neural map matchers from inaccurate absolute pose labels. First, we show that heading labels are unnecessary: trained from raw GPS labels, models learn to predict accurate headings, automatically. Second, defining a tolerance region around raw GPS improves positional accuracy. Third, if available, our supervision uses relative poses between training images, obtained via SLAM or SfM, which provide a more accurate training signal. Across driving and egocentric benchmarks, AutoCompass consistently outperforms counterparts trained with the usual strong reliance on absolute pose labels.