HierLoc: Hyperbolic Entity Embeddings for Hierarchical Visual Geolocation

AI summary

Overview

Research area: Computer vision — specifically global visual geolocalization (predicting the location where an image was taken), combined with hyperbolic representation learning and multimodal contrastive learning.

Technical level: Intermediate. The core idea (match an image to a place, not to another image) is easy to grasp, but the method rests on the Lorentz model of hyperbolic space, exponential/logarithmic maps, Riemannian optimization, and a geometry-weighted InfoNCE objective, so the methodology sections require comfort with differential geometry and contrastive learning.

Scope: The paper proposes HierLoc, a system that embeds a four-level hierarchy of geographic entities (country, region, subregion, city) in hyperbolic space and aligns images to those entities with a distance-weighted contrastive loss, reporting state-of-the-art results on the OSV5M benchmark and competitive results on IM2GPS, IM2GPS3K, and YFCC4K.

What This Paper Is About

Visual geolocalization is hard because the search space is the entire planet, visually similar scenes (beaches, skylines, street signage) recur across continents, and geography is naturally hierarchical. Existing approaches pick one of three strategies: retrieval against millions of stored image embeddings, grid-based classification that discards geographic continuity, or generative diffusion over space that struggles at fine scales.

The paper's goal is to sidestep image-to-image retrieval entirely by learning a compact set of geographic entity embeddings in hyperbolic space, so that an image is matched directly to a country, region, subregion, and city, and the prediction is refined by beam search down the hierarchy.

Key Contributions

1. Image-to-entity reformulation of geolocation. The method replaces retrieval against millions of images with alignment against 240k entity embeddings (233 countries, 4,946 regions, 29,214 subregions, 209,894 cities), cutting search complexity while improving accuracy.
2. Hyperbolic geographic hierarchy. It is, to the authors' knowledge, the first application of hyperbolic embeddings to represent hierarchical geographic entities for geolocation explicitly, exploiting hyperbolic volume growth to keep fine-grained locations separable where Euclidean space crowds them.
3. Geo-Weighted Hyperbolic InfoNCE (GWH-InfoNCE). A contrastive objective that reweights negative entities by their great-circle (haversine) distance from the image location, emphasizing geographically proximal negatives and improving fine-scale discrimination.
4. State-of-the-art results on OSV5M. Gains at every level (country +8.8%, region +20.1%, subregion +43.2%, city +16.8%) and a 19.5% reduction in mean geodesic error relative to the strongest baselines.

Main Findings

• OSV5M state of the art: HierLoc with a DINOV3 backbone reaches a GeoScore of 3963, a mean geodesic error of 861 km, and classification accuracy of 82.9% (country), 55.0% (region), 40.7% (subregion), and 23.3% (city). For comparison, SC Retrieval scores 3597 GeoScore at 1386 km with 73.4/45.8/28.4/19.9 accuracy, and the generative RFM S₂ model scores 3767 at 1069 km with 76.2/44.2/–/5.4.
• Backbone fairness: HierLoc with a ViT-L/14 backbone, matching the baselines' encoder, still reaches 3850 GeoScore, 1067 km, and 80.1/52.9/39.0/22.2 accuracy, surpassing all baselines — so the gains are attributed to the framework rather than the encoder.
• IM2GPS: Median error 21.4 km with 10.5% @ 1 km, 51.9% @ 25 km, 67.5% @ 200 km, 83.1% @ 750 km, and 92.4% @ 2500 km, versus PIGEON's 70.5 km median and 14.8/40.9/63.3/82.3/91.1 recall.
• IM2GPS3K: Median error 73.4 km, roughly half of PIGEON's 147.3 km, with a 7.1-point improvement at 25 km recall (43.8% vs 36.7%); overall recall is 11.3/43.8/58.4/74.1/85.1.
• YFCC4K: Median error 341.9 km with recall 8.4/30.2/43.3/61.7/75.8, improving over PIGEON's 383.0 km median on city and region level recall.
• Efficiency versus generative models: Trained on 4.7M images, HierLoc matches the mean geodesic error (2058 km) of the RFM₁₀M S₂ model trained on 48M images, with a GeoScore of 3189 versus 3210; it trails slightly on continent-scale recall (75.9 vs 77.7 at 2500 km).
• Hyperbolic beats Euclidean and spherical: Ablation on OSV5M gives Hyperbolic 3963 GeoScore / 861 km, Euclidean 3865 / 968 km, and Spherical 3364 / 1258 km; the spherical variant collapses at fine levels (15.9% subregion, 4.3% city), attributed to higher distance distortions.
• Loss and attention matter: Replacing GWH-InfoNCE with standard InfoNCE degrades performance (3840 GeoScore, 949 km); removing cross-modal attention causes the largest drop (2904 GeoScore, 1366 km); removing the squared hyperbolic distance also hurts (3752 GeoScore, 1043 km). A zero-shot DINOV3 baseline reaches only 2962 GeoScore and 1999 km.
• Modalities: Removing text and location features lowers performance only slightly (3890 GeoScore, 1029 km, 81.8/52.1/37.9/21.1), indicating image signal is the dominant modality, though the combination of all three is best.
• Beam search helps: Flat per-level search gives 79.6/50.8/39.4/22.1 accuracy, hierarchical beam search with beam=1 gives 79.4/48.9/36.4/21.9, and beam=10 gives 82.9/55.0/40.7/23.3.
• Encoder consistency: On OSV5M, DINOv3 (3963/861/82.9/55.0/40.7/23.3), StreetCLIP (3862/1051/80.3/53.1/39.2/22.5), and ViT-L/14 (3850/1067/80.1/52.9/39.0/22.2) all outperform non-HierLoc baselines. On MediaEval'16, DINOv2 (3106/2211) trails DINOv3 (3189/2058) but still outperforms the RFM S₂ model.
• Data compression: Roughly 9.6 million image records across both datasets are distilled into about 240k entities.
• Reported limitation: 1 km localization is bounded by the discretization limit of 240k fixed entities, which lacks dense street-level granularity.

Methodology in Plain English

The world is treated as a tree of named places. From the training metadata, the authors build four levels — country, region, subregion, and city — and assign each training image a tuple of four entities, using the official quadtree-aligned labels for OSV5M and deterministic reverse geocoding with Nominatim for MediaEval'16. Each entity gets three descriptive signals: the average image embedding of all training images assigned to it, a CLIP text embedding of its name, and its latitude/longitude passed through the SphereM+ location encoder.

Both images and entities are then placed on a hyperbolic manifold — specifically the Lorentz (hyperboloid) model with constant curvature −1/K. Because neural network layers are not well defined on a curved manifold, all linear operations happen in the flat tangent space at a fixed origin, using exponential and logarithmic maps to move between the tangent space and the manifold. Entity embeddings start from a random anchor on the manifold and are nudged by a learnable update vector derived from their fused multimodal features. Image embeddings, produced by a frozen backbone such as DINOV3, are projected into the same tangent space and lifted onto the manifold.

A cross-modal attention block with 8 heads per level then lets each image query the entity embeddings at all four levels. The four level-wise contexts are concatenated, fused by a small MLP, and added back to the image representation. Only the image stream is updated this way; entity embeddings stay fixed, which the authors describe as an asymmetry that prevents entity overfitting while still providing hierarchical context.

Training uses GWH-InfoNCE: for each hierarchy level, the correct entity is the positive, all other entities at that level are negatives, and distances are squared hyperbolic geodesic distances. Each negative's contribution is upweighted by how close it is geographically, using the haversine distance and a Laplace decay kernel (1 + λ·exp(−g/σ)), with τ, λ, and σ learnable. Per-level losses are combined with level weights β.

At inference, a beam search with width k=10 walks the hierarchy, ranking candidates by hyperbolic geodesic distance and retaining the top-k at each step. At the city level, the predicted entity's coordinates become the location estimate, from which mean geodesic error and GeoScore are computed. Nearest-neighbor lookups use FAISS FlatIP with a time-coordinate flip so Lorentz inner products can be ranked without explicit distance computation. Two separate models are trained, one on OSV5M and one on MediaEval'16, using AdamW for Euclidean parameters and RiemannianAdam for manifold parameters, batch size 16, learning rate 2×10⁻⁴, on 6×NVIDIA L40S GPUs for 5 epochs, about 60 hours per run.

Why This Matters

Impact on research. The paper argues that hyperbolic embedding spaces offer a principled advantage for multimodal representation learning whenever data have inherent hierarchical structure, with geolocation as a suitable testbed. It also reframes a long-standing retrieval problem as image-to-entity alignment, and shows a compact entity index can beat a far larger image index — a claim supported by comparisons against retrieval, classification, and generative baselines on OSV5M as well as cross-dataset benchmarks.

Real-world applications (as listed in the paper):

• Biodiversity monitoring
• Cultural heritage preservation
• News verification
• Augmented reality

The paper also motivates the work by noting that many real-world images lack geotags in their metadata, making automated localization increasingly important, and it identifies interpretable, structured error analysis and potential client-side deployment as benefits of the compact entity representation.

Industry relevance. The work originates from Huawei Riemann Lab, Hilbert Research Centre, with co-authors at the Technical University of Munich's Chair of Cartography. The practical pitch is memory, indexing, and runtime: retrieval methods scale linearly with O(N) comparisons over millions of images, whereas HierLoc operates on 240k entities with beam search, which the authors describe as scaling sub-linearly and enabling faster inference and possible on-client deployment. HierLoc also reaches the mean geodesic error of the 1M-iteration generative RFM S₂ and matches the 10M-iteration variant trained on 48M images while itself training on 4.7M images.

Future Directions

• Breaking the discretization ceiling. The paper states that 1 km localization is bounded by the choice of 240k fixed entities and the lack of dense street-level granularity, framing this as a fundamental trade-off. Finer or adaptive entity sets, or hybrid entity-plus-regression heads, are the natural next step.
• Broadening the empirical base. The authors train only two separate models (OSV5M and MediaEval'16) and note that MediaEval'16 has no public official split. Extending to more datasets and standardizing splits would strengthen the comparisons.
• Deeper hyperparameter and geometry analysis. The paper references appendix ablations on mean image embeddings, hyperparameter sensitivity, Lorentz curvature choice, and the weight-decay function for geo-weights (Laplace versus Gaussian and inverse kernels), leaving these as open levers for tuning and theoretical study.
• Transfer to other hierarchical multimodal problems. The authors claim the findings generalize to any multimodal setting with inherent hierarchy; testing hyperbolic entity embeddings on other tree-structured tasks is directly implied.

Target Audience

Researchers and practitioners in computer vision and geospatial machine learning, especially those working on image geolocalization, place recognition, or multimodal retrieval; engineers who need a memory-efficient alternative to million-scale image retrieval indexes; and representation-learning researchers interested in hyperbolic embeddings for hierarchical or cross-modal data. Readers without a background in Riemannian geometry will find the conceptual framing accessible but will need

Authors’ abstract

Visual geolocalization, the task of predicting where an image was taken, remains challenging due to global scale, visual ambiguity, and the inherently hierarchical structure of geography. Existing paradigms rely on either large-scale retrieval, which requires storing a large number of image embeddings, grid-based classifiers that ignore geographic continuity, or generative models that diffuse over space but struggle with fine detail. We introduce an entity-centric formulation of geolocation that replaces image-to-image retrieval with a compact hierarchy of geographic entities embedded in Hyperbolic space. Images are aligned directly to country, region, subregion, and city entities through Geo-Weighted Hyperbolic contrastive learning by directly incorporating haversine distance into the contrastive objective. This hierarchical design enables interpretable predictions and efficient inference with 240k entity embeddings instead of over 5 million image embeddings on the OSV5M benchmark, on which our method establishes a new state-of-the-art performance. Compared to the current methods in the literature, it reduces mean geodesic error by 19.5\%, while improving the fine-grained subregion accuracy by 43%. These results demonstrate that geometry-aware hierarchical embeddings provide a scalable and conceptually new alternative for global image geolocation.