CoST: Semantic-Aware Urban Understanding via Spatial-Temporal Alignment

AI summary

Overview

• Research area: Geospatial representation learning from satellite imagery — specifically self-supervised learning for urban analysis (socio-economic indicator prediction, land-use classification, and change detection).
• Technical level: Advanced. The work assumes familiarity with contrastive learning (InfoNCE), vision transformers and ResNet backbones, remote sensing benchmarks, and linear-probe evaluation protocols.
• Scope: A single paper proposing CoST, a contrastive spatial-temporal pre-training framework that aligns geographic neighborhood structure with multi-year urban change semantics, evaluated across eight city-indicator settings plus classification and change detection benchmarks.

What This Paper Is About

Satellite imagery is widely used to learn representations of urban regions, but existing methods either depend on region-specific auxiliary data (points of interest, mobility traces) that vary in availability and quality across cities, or learn purely visual features that do not reflect high-level urban meaning. The paper argues that this produces representations that neither transfer well to unseen cities nor remain interpretable in terms of real urban change. CoST addresses both problems by pre-training an image encoder so that nearby geographic regions share structure and so that distances between time steps of the same location reflect the type and magnitude of the semantic transitions that actually occurred there.

Key Contributions

1. A unified spatial-temporal pre-training framework (CoST) for satellite imagery that jointly models spatial neighborhood structure and temporal semantic transitions, targeting cross-region generalization and semantic interpretability at the same time.
2. Three complementary components: spatial neighborhood modeling (a soft target distribution that gives neighborhood-level positives auxiliary weight, following Tobler's First Law of Geography), temporal semantic guidance (encoding multi-year transitions by both change type and change extent rather than as a binary change/no-change signal), and spatial-temporal alignment (a corrective term that penalizes spatial similarity between neighbors whose semantic change directions are misaligned).
3. A curated spatial-temporal aligned satellite dataset spanning 11 years (2010–2020) across five pre-training cities — Beijing, Guangzhou, Shanghai, Shenzhen, and New York — plus a held-out Chicago set excluded from pre-training for cross-city generalization testing.
4. Extensive evaluation across cross-city and cross-task settings with two encoder variants, CoST-R (ResNet50) and CoST-V (ViT-B/16), reporting an average relative gain of 8.7% over the strongest competing methods across eight city-indicator settings.

Main Findings

• Urban indicator prediction: CoST-R reaches an R² of 0.664 for GDP and 0.698 for population in New York City, which the authors describe as outperforming prior methods by over 5%. The paper states similar improvements were observed in Shanghai and Shenzhen, reported in Appendix A (not included in the available content).
• Cross-region generalization: On Chicago, which is excluded from pre-training, CoST achieves GDP and population R² of 0.646 and 0.691 respectively, surpassing region-dependent baselines.
• Land-use classification: CoST-R and CoST-V reach 97.13% and 98.73% top-1 accuracy on UC Merced Land-Use. On the multi-label BigEarthNet benchmark, CoST-V reaches 83.55 mAP and CoST-R 82.78 mAP. ScaleMAE reports 98.41% top-1 on UCM (higher than both CoST variants) and 82.57 mAP on BigEarthNet (lower than both CoST variants).
• Change detection: CoST-R achieves an F1 score of 0.6425 on LEVIR-CD and 0.5329 on OSCD, surpassing specialized bi-temporal methods. The comparison point MoCoV3 reports 0.6220 on LEVIR-CD and 0.5315 on OSCD. CoST-V change detection results are marked with a dash in Table 3 (not reported).
• Semantic interpretability: Embedding arithmetic (Figure 5a) shows that subtracting one attribute, such as built-up areas, and adding another, such as farmland, retrieves images in which the target semantic components change while surrounding context is preserved. The authors interpret this as evidence that the latent space is semantically disentangled rather than memorizing pixel-level patterns.
• Ablation: Removing spatial neighborhood modeling (w/o Spatial) produces a pronounced drop, with R² decreasing by 15–30% across cities. The results for the w/o Temporal and w/o ST variants, and the details of the RQ4 analysis on temporal dynamics and domain invariance, are not reported in the available paper content (it is truncated at this point).
• Complexity of the semantic accumulation step: Aggregating yearly change vectors yields linear O(T) complexity rather than the O(T²) of direct pairwise time-point comparison.

Methodology in Plain English

CoST pre-trains an image encoder from scratch using contrastive learning, then freezes it and trains lightweight task-specific heads on top.

1. Modeling space. Standard contrastive learning treats each satellite tile as an independent instance. CoST instead uses a batch target distribution in which the anchor's augmented view gets target score 1, each of its k nearest geographic neighbors gets a small score epsilon (in the range 0 to 1), and everything else gets 0. The model's predicted similarity distribution is matched to this target, so geographically proximate regions end up embedded near each other while the anchor still dominates the signal.

2. Modeling time. Before pre-training, the authors build a semantic change signal in three steps. They generate yearly land-use semantic maps with Grounded-SAM, measure the proportion of changed pixels per land-use category between consecutive years to get yearly change vectors, sum these vectors to obtain a cumulative change vector over an interval, and then project that vector into an embedding via a frozen text encoder over class embeddings. An interval with little change gets a soft label near 1 (pulling the two timestamps together); a large change gets a label toward 0 (pushing them apart). This is trained with a binary cross-entropy objective. Because the authors note that BCE alone converges slowly and unstably when the change embeddings are noisy, they add an InfoNCE term with augmented positives and hard temporal negatives — the same location at different time steps where the semantic change magnitude is non-zero.

3. Tying space to time. Independently optimizing the spatial and temporal objectives can create false positives: two nearby tiles may look alike but diverge semantically over years. For each anchor, CoST finds its most cosine-similar neighbor, identifies the timestamp of maximal semantic divergence, forms normalized semantic change vectors for both tiles, and down-weights the spatial similarity when those change directions disagree.

4. Pre-training and evaluation. The total objective is a weighted sum of the spatial loss, the two temporal losses, and the spatial-temporal alignment loss, with weights tuned by grid search. Both variants use a feature dimension of 128 and are pre-trained for 3,000 epochs on two NVIDIA RTX A6000 GPUs. Downstream heads are linear-probe style: an MLP regression head for static indicator prediction, an LSTM plus MLP combination for dynamic prediction using historical representation sequences, an MLP with softmax for land-use classification, and a lightweight U-Net decoder over concatenated bi-temporal representations for change detection.

Why This Matters

Impact on research. The paper reframes what a good geospatial representation must satisfy: not just augmentation invariance or texture discrimination, but transferability across cities with different visual styles and development patterns, and enough semantic structure to reflect real urban transitions. It also shows that multi-temporal imagery can serve as a supervision source without dense human annotation, using foundation-model-derived semantic maps plus a frozen text encoder to synthesize soft change labels.

Real-world applications:

• Population density and GDP estimation for regions where ground surveys are sparse or outdated, using WorldPop and publicly available global GDP data as targets.
• Land-use and land-cover mapping at scale, evaluated on UC Merced Land-Use and BigEarthNet categories.
• Bi-temporal change detection for construction monitoring and land conversion, evaluated on OSCD and LEVIR-CD.
• Cross-city or cross-country deployment where POI and mobility data are unavailable — the paper's held-out Chicago test is designed to approximate this setting.

Industry relevance. The reliance on freely available satellite imagery from Google Earth, rather than proprietary auxiliary datasets that must be re-collected per region, matters for scalability. The paper explicitly notes that current multimodal approaches require data recollection and model retraining for each new region, and positions CoST as avoiding that cost.

Future Directions

• Filling in the full ablation. Only the w/o Spatial result is reported in the available content; how much temporal semantic guidance and spatial-temporal alignment each contribute individually is an open question here.
• The RQ4 analysis. The paper poses the question of how temporal dynamics regularize semantic consistency and improve domain invariance across cities, but the findings are not reported in the available content.
• Extending beyond the studied geography. Pre-training covers five cities (Beijing, Shanghai, Guangzhou, Shenzhen, New York) with one held-out city (Chicago); generalization to other continents, rural regions, or climates is untested here.
• Dependence on foundation model outputs. The semantic change signal is derived from Grounded-SAM semantic maps and a frozen text encoder, and the paper itself notes that noise in these foundation-model-derived embeddings motivates the added InfoNCE term — improving the robustness or replacing this signal is a natural next step.

Target Audience

Researchers and practitioners working on remote sensing representation learning, geospatial foundation models, and self-supervised learning for Earth observation. It is also relevant to urban analytics and computational social science groups doing socio-economic indicator prediction, and to applied teams building cross-city or cross-country mapping pipelines who need representations that transfer without region-specific auxiliary data. Readers without background in contrastive learning or remote sensing benchmarks will need to consult the cited baselines (SimCLR, MoCoV3, SatMAE, ScaleMAE, SeCo, CACo, among others) to fully follow the experiments.

Authors’ abstract

Geospatial representation learning from satellite imagery is a fundamental problem for large-scale urban analysis and real-world applications. Despite recent advances, current methods struggle with cross-region generalization and semantic interpretability due to their reliance on region-specific auxiliary data and the neglect of semantic alignment within multi-temporal urban imagery. Therefore, we present CoST, a novel \underline{Co}ntrastive-based \underline{S}patial-\underline{T}emporal framework that aligns spatial context with multi-temporal semantics to extract universal geographic regularities shared across regions. Specifically, CoST explicitly models spatial correlations to capture transferable geographic structures and exploits multi-year urban change semantics to align learned representations with high-level geo-semantics. Extensive experiments demonstrate that CoST consistently achieves superior performance across various downstream tasks and in unseen scenario, yielding an average relative gain of 8.7\% over the strongest competing methods across eight city-indicator settings. The code is available in \href{https://github.com/Arandinglv/CoST}{this repo}.