Improving Region Representation Learning from Urban Imagery with Noisy Long-Caption Supervision

AI summary

Overview

Research area: Urban computing and cross-modal representation learning — specifically, learning general-purpose vector representations of city regions from street-view and satellite imagery, using text captions generated by Large Language Models (LLMs) as auxiliary supervision.

Technical level: Advanced. The paper assumes familiarity with contrastive vision-language pre-training (CLIP), self-distillation with momentum teachers, positional embedding interpolation, and standard urban-computing downstream regression tasks.

Scope: The paper proposes UrbanLN, a cross-modal pre-training framework that (1) makes CLIP's text encoder digest captions far longer than its native 77-token limit, and (2) suppresses noise in LLM-generated captions at both the data level and the model level, then evaluates the learned region embeddings on socioeconomic prediction tasks across four cities.

Paper metadata: arXiv:2511.07062v2 [cs.AI], 30 Nov 2025. Authors: Yimei Zhang, Guojiang Shen, Kaili Ning, Tongwei Ren, Xuebo Qiu, Mengmeng Wang, Xiangjie Kong. Affiliations listed: Zhejiang University of Technology (College of Computer Science and Technology), Zhejiang Key Laboratory of Visual Information Intelligent Processing, and State Key Laboratory for Novel Software Technology, Nanjing University.

What This Paper Is About

Urban regions can be described by how they look — the buildings, streets, and infrastructure visible in satellite and street-view imagery. Recent work feeds these images to multimodal LLMs, gets back long textual descriptions, and uses them as extra knowledge to train better region embeddings. The problem is that those captions are typically longer than what CLIP's text encoder can accept, and they routinely contain hallucinations, omissions, or bland generic phrasing, so the extra knowledge does not get integrated well. UrbanLN attacks both problems: it stretches CLIP's positional encoding so long captions survive intact, and it adds a two-level noise-suppression scheme so the model can still learn from imperfect captions.

Key Contributions

1. Information-Preserved Stretching Interpolation (IPSI). A positional-embedding interpolation strategy that relaxes CLIP's input length constraint with minimal extra computation, letting the model extract fine-grained and holistic features from long urban captions while preserving the well-trained lower positional embeddings.

2. A data-level noise mitigation pipeline. Multi-MLLM captioning, a divide-and-conquer refinement step (SAM segmentation plus local short captions plus phrase-level filtering), and a consensus-based evaluation that selects the caption with the highest inter-model agreement — all without human annotation.

3. A model-level noise mitigation mechanism. Momentum-based self-distillation (MSD), where an EMA-updated copy of the student acts as a teacher and supplies stable pseudo-targets, so the student learns representations robust to noisy or semantically ambiguous captions.

4. Extensive empirical validation. Experiments across four real-world cities (Beijing, Shanghai, Shenzhen, New York) and multiple downstream tasks, with reported improvements over state-of-the-art baselines plus ablation, hyperparameter, transferability, and latent-space analyses.

Main Findings

• Beijing results (Table 1). UrbanLN+SV (street-view imagery) reaches R² of 0.705 (Pop), 0.440 (GDP), 0.514 (Night), 0.591 (Com), and 0.677 (CO₂). Relative to the best-performing baseline, the reported improvements are 13.9%, 18.3%, 13.2%, 6.5%, and 39.0% on R² across those five tasks. UrbanLN+SI (satellite imagery) reaches 0.686 (Pop), 0.437 (GDP), 0.519 (Night), 0.597 (Com), and 0.688 (CO₂), with improvements of 10.8%, 17.5%, 14.3%, 7.6%, and 41.3%. Averaged over the strongest baselines, the paper reports gains of 18.23%, 7.84%, and 8.32% in R², RMSE, and MAE respectively.

• Shanghai results (Table 2, R² only). UrbanLN+SV scores 0.669 (Pop), 0.480 (GDP), 0.552 (Night), and 0.697 (CO₂), with improvements of 16.1%, 1.5%, 9.1%, and 9.4%. UrbanLN+SI scores 0.661, 0.476, 0.555, and 0.718, with improvements of 14.8%, 0.6%, 9.7%, and 12.7%.

• New York results (Table 3, R² only). UrbanLN+SV achieves 0.676 (Pop), 0.723 (Crime), and 0.650 (POI), with improvements of 26.6%, 54.8%, and 11.5%; the paper states an average improvement of 30.97% across NY prediction tasks for UrbanLN+SV. UrbanLN+SI scores 0.725 (Pop, +35.8%), but 0.323 (Crime, −30.8%) and 0.521 (POI, −10.6%). The authors attribute the satellite-imagery shortfall in crime and POI prediction to the inherent limitations of satellite imagery and the lower spatial granularity of the NY dataset.

• ResNet-18 is the weakest baseline. Its table entries include a negative R² of −0.079 on New York population, illustrating how poorly generic ImageNet features transfer to this task.

• Ablation: IPSI matters most (Figure 3). Removing IPSI causes an average performance reduction of 26.45%, the largest of any variant. Removing the divide-and-conquer refinement module yields a 10.45% average drop. Replacing consensus-based selection with random choice also degrades performance, supporting inter-model agreement as a caption-quality proxy. Removing momentum-based self-distillation produces a notable drop, described qualitatively rather than with a stated percentage.

• Hyperparameter sensitivity is low (Table 4). Across μ ∈ {0.3, 0.4, 0.5, 0.6}, population-prediction R² varies only within a narrow band (for example, street-view BJ: 0.687, 0.697, 0.705, 0.692). μ = 0.5 is adopted as the default.

• Efficiency. UrbanLN is reported to have a comparable total parameter count to UrbanCLIP and UrbanVLP while substantially reducing trainable parameters, and it is described as striking a better accuracy/efficiency balance than UrbanCLIP, which attains the highest FPS but at the expense of accuracy. Exact parameter counts and FPS numbers are not reported in the provided text.

• Cross-city transferability. Pre-training in a source city and applying the encoder to other cities consistently yields high prediction accuracy on both street-view and satellite imagery (Figure 5), which the authors read as evidence that the model captures fundamental commonalities across urban environments.

• Latent-space structure (Shenzhen case study). PCA projection followed by K-means yields three clusters with clear spatial distinctions. Cluster 1 (green) shows a sharp, narrow KDE peak corresponding to dense central areas with high population and images containing high-rise buildings, busy streets, and commercial infrastructure. Cluster 0 (blue) overlaps with the other two and likely represents transitional zones. Cluster 2 (orange) corresponds to suburban or rural regions with lower population.

Methodology in Plain English

Step 1 — Generate captions with several LLMs, then clean them up. Five multimodal LLMs (LLaMA-Adapter V2, ShareGPT4V-7B, Qwen2.5-VL-7B, DeepSeek-VL2-tiny, InternVL3-8B) each independently caption every urban image, producing multiple long descriptions per image. Diversity across models acts as a form of text augmentation and dilutes any single model's quirks. Next, the Segment Anything Model (SAM) cuts the image into salient visual elements, and a maximal rectangle algorithm reduces overlap between those crops. Each MLLM then writes a short local caption for each element, filling in details the long caption omitted. To suppress hallucinations, the Factual parser extracts visual-element phrases from both the long and short captions, and OWLv2 scores each phrase; anything scoring below 0.01 is discarded. The same MLLM then rewrites a more complete, natural description using the surviving details.

Step 2 — Pick the best caption by consensus. With no ground-truth captions available, the authors treat agreement among independent models as a reliability signal. They use the CAPTURE metric, which extracts objects, attributes, and relations via the Factual parser and matches them through exact, synonym, and soft (embedding-based) matching, then combines per-type F1 scores as CAPTURE = (α·F1_obj + β·F1_attr + γ·F1_rel) / (α + β + γ), with α, β, γ set to 5, 5, 2. Each caption's consensus score is its average CAPTURE score against all other captions for the same image; the highest-scoring caption wins.

Step 3 — Teach CLIP to read long captions. CLIP's text encoder is capped at 77 tokens, while the average caption here exceeds 100 words. Rather than interpolating all positional embeddings — which prior work suggests disrupts the well-trained lower ones — the method keeps the first 20 positions untouched and interpolates only the remaining 57, using a weighted blend of the two neighboring original positions with ratio λ = 4. This extends the maximum input length to 248 tokens.

Step 4 — Learn despite residual noise. Even after cleaning, captions can be wrong, and visually similar neighborhoods can make a "negative" caption actually accurate for the wrong image. So a momentum teacher — an exponential moving average of the student, updated with momentum 0.995 — provides pseudo-targets. The student minimizes a contrastive loss L_C and a KL-divergence distillation loss L_D against those pseudo-targets, combined as L = (1 − μ)·L_C + μ·L_D with μ = 0.5. Two dynamic queues of length 4096 store recent teacher-encoded image-text representations.

Step 5 — Predict downstream indicators. Once pre-trained, the image encoder is frozen and used to produce a compact feature vector per region (average-pooled across a region's multiple street-view images). A lightweight multi-layer perceptron head is then fine-tuned per task. Training uses the AdamW optimizer with learning rate 1e-7.

Data. Street-view and satellite images for Beijing, Shanghai, and Shenzhen come from Baidu Maps; New York data come from a prior released dataset (Yong and Zhou 2024). Street-view images are sampled at fixed 500-meter intervals along the OpenStreetMap road network, at 1024×512 resolution; satellite images are 256×256 RGB. New York replaces the originally used Guangzhou to inject greater geographical and cultural variance. Region counts: BJ 4,584; SH 5,168; SZ 3,383; NY 517. Street-view image counts: 29,814 (BJ), 37,469 (SH), 33,897 (SZ), 44,617 (NY). All indicator values are log-transformed via Y = ln(Y_original + 1), and each downstream dataset is split 6:2:2 into train/validation/test.

Why This Matters

Impact on research. The paper reframes a practical obstacle — LLM-generated text is noisy and verbose — as a first-class modeling problem rather than something to fix with manual annotation. UrbanCLIP relies on manual calibration, which the authors call time-consuming and non-scalable; UrbanVLP relies on fixed category schemas and rigid prompt templates, which the authors argue loses high-level semantics and produces homogeneous descriptions. UrbanLN offers a reproducible alternative: automatic multi-model consensus plus a self-distillation objective that tolerates whatever noise survives. The IPSI trick also gives a cheap, general recipe for adapting CLIP-style encoders to long inputs without retraining positional embeddings.

Real-world applications.

• Urban planning and policy: Predicting population, GDP, and nighttime light intensity from imagery alone can help officials target investment in areas where survey data is stale, expensive, or missing.
• Environmental monitoring: Carbon-emission estimation (CO₂ modelled here from the ODIAC database) from visual appearance supports city-level climate accounting where sensor coverage is sparse.
• Commercial site selection: Predicting restaurant comment volume (Com) or POI counts gives businesses and municipal planners a proxy for foot traffic and commercial vitality without proprietary mobility feeds.
• Public safety: Crime-incidence prediction — where UrbanLN+SV reported the largest single gain on the NY dataset at 54.8% R² improvement over the best baseline — could inform patrol allocation, though the authors do not discuss deployment or fairness implications.

Industry relevance. The efficiency analysis is aimed squarely at practitioners: a comparable total parameter count to strong baselines but substantially fewer trainable parameters, plus a favorable accuracy/FPS trade-off relative to UrbanCLIP, which is described as the fastest but less accurate. That combination is the profile that matters for real-time or large-scale urban analytics deployments. Because MuseCL requires mobility-flow data unavailable in BJ, SH, and SZ, UrbanLN's reliance on publicly obtainable imagery plus LLM captions is a meaningful practical advantage for cities that lack proprietary mobility datasets.

Future Directions

1. Close the satellite-imagery gap on fine-grained social tasks. On New York, UrbanLN+SI underperforms on Crime (−30.8%) and POI (−10.6%) relative to the baseline. The authors attribute this to satellite imagery's limitations and the dataset's coarser spatial granularity, but they do not propose a fix — a natural next step is a hybrid or modality-adaptive pipeline that switches to street-view cues when fine social structure matters.

2. Reduce dependence on the multi-MLLM caption pipeline. Five MLLMs plus SAM plus OWLv2 plus the Factual parser is a heavy, multi-stage pre-processing stack. The paper reports model-side efficiency but does not report the cost of caption generation. Distilling that pipeline into a single captioner while retaining consensus-quality captions would broaden applicability.

3. Quantify and characterize the remaining noise. The ablation shows MSD helps, but the paper reports the w/o MSD effect only qualitatively, and no direct measurement of residual caption noise after refinement and consensus selection is given. A noise-rate analysis would clarify how much headroom remains.

4. Extend transferability testing systematically. The transferability study shows a source-city model working in other cities, but the paper does not report a full city-pair matrix. Establishing how performance degrades across very different urban contexts — and whether the latent clusters generalize — would test the claim that the model captures "fundamental commonalities" rather than city-specific cues.

Target Audience

This paper is most useful to researchers and graduate students in urban computing, remote sensing, and multimodal machine learning who are already comfortable with contrastive vision-language pre-training. It will particularly appeal to those working on self-supervised region representation learning and on robust learning from noisy LLM-generated supervision, since both the positional-interpolation trick and the dual-level denoising scheme are transferable to other domains where captions are long and imperfect. Practitioners building city-scale analytics platforms will find the efficiency comparison and the reliance on publicly available imagery relevant to feasibility decisions. Readers without a background in CLIP, distillation, or geospatial data will find the methodology dense, though the problem framing and the results tables are accessible on their own.

Authors’ abstract

Region representation learning plays a pivotal role in urban computing by extracting meaningful features from unlabeled urban data. Analogous to how perceived facial age reflects an individual's health, the visual appearance of a city serves as its "portrait", encapsulating latent socio-economic and environmental characteristics. Recent studies have explored leveraging Large Language Models (LLMs) to incorporate textual knowledge into imagery-based urban region representation learning. However, two major challenges remain: i) difficulty in aligning fine-grained visual features with long captions, and ii) suboptimal knowledge incorporation due to noise in LLM-generated captions. To address these issues, we propose a novel pre-training framework called UrbanLN that improves Urban region representation learning through Long-text awareness and Noise suppression. Specifically, we introduce an information-preserved stretching interpolation strategy that aligns long captions with fine-grained visual semantics in complex urban scenes. To effectively mine knowledge from LLM-generated captions and filter out noise, we propose a dual-level optimization strategy. At the data level, a multi-model collaboration pipeline automatically generates diverse and reliable captions without human intervention. At the model level, we employ a momentum-based self-distillation mechanism to generate stable pseudo-targets, facilitating robust cross-modal learning under noisy conditions. Extensive experiments across four real-world cities and various downstream tasks demonstrate the superior performance of our UrbanLN.