TerraMesh-Masks: Open-Vocabulary Segmentation for Earth Observation

AI summary

Overview

Research area: Computer vision / Earth observation — open-vocabulary segmentation (OVS) with multimodal satellite data.

Technical level: Advanced. The paper assumes familiarity with segmentation metrics (IoU, F1), vision–language models such as CLIP, and multimodal pretraining architectures.

Scope: The paper introduces TerraMesh-Masks, a 16-million-mask open-vocabulary segmentation dataset built on satellite imagery, plus an expert-reviewed 523-sample benchmark, and demonstrates its value by adapting the TerraMind model to open-vocabulary binary segmentation.

What This Paper Is About

Open-vocabulary segmentation lets a user describe a concept in natural language (for example "golf course" or "coral reef") and receive a pixel mask, instead of being restricted to a fixed set of trained classes. Existing OVS models are built for natural photos and transfer poorly to satellite imagery, where concepts appear at coarser spatial resolution (10 m here), often need multispectral sensing, and vary strongly by geography. The authors address the lack of both a large-scale training resource and a rigorous benchmark for OVS in Earth observation by building a planetary-scale mask dataset from OpenStreetMap and Overture Maps and by showing that EO-specific multimodal pretraining on it substantially outperforms general-purpose models.

Key Contributions

1. TerraMesh-Masks dataset: Roughly 16 million multimodal image–mask samples with language annotations, derived from the 9-million-sample TerraMesh collection, covering 174 countries across 22 continental regions at 10 m resolution, with modalities including Sentinel-1 SAR, Sentinel-2 optical, RGB, and DEM.
2. TerraMesh-Masks-Eval benchmark: An expert-reviewed evaluation set of 523 samples covering 207 unique class labels (over 200 semantic concepts), grouped into 14 semantic clusters, produced from 4,211 individual reviews by four expert annotators.
3. TerraMind-OVS model: An open-vocabulary extension of TerraMind that treats binary masks as an additional modality, using a frozen CLIP text encoder (OpenAI ViT-B-32) and a lightweight projection head with minimal architectural changes, so that gains can be attributed to the dataset rather than to model design.
4. Empirical demonstration of dataset impact: TerraMind-OVS reaches 35.18% IoU versus 16.29% for DINOv3.txt and 11.33% for SAM-3, with the authors reporting relative mIoU improvements of 479% over SAM-3 and 53% over DINOv3.txt.

Main Findings

• Large margin over generalist models: On TerraMesh-Masks-Eval, TerraMind-OVS-L (S2 input) reaches 35.18% IoU_pos, 63.54% mIoU, and 39.80% F1, versus 16.29% IoU_pos / 41.40% mIoU for DINOv3.txt and 11.33% IoU_pos / 10.97% mIoU for SAM-3.
• EO-specific baselines are stronger but still behind: SegEarth-OV achieves the best IoU_pos among baselines at 21.59%, and TerraMind-OVS-L improves on it by 13.59 percentage points. GeoPixel has the second-best mIoU at 49.25%, which is 14.29 percentage points below TerraMind.
• Model size scales performance: TerraMind-OVS-S reaches 27.37% IoU_pos, TM-OVS-B reaches 32.18%, and TM-OVS-L reaches 34.96% (RGB input, S2+RGB training), showing consistent gains from small (247M parameters) to large (934M parameters).
• Multispectral input helps, extra modalities at inference do not: Sentinel-2 multispectral input gives the top results overall, with particular benefits on Forest & Wetland and Barren terrain groups; Sentinel-1 and DEM inputs significantly reduce performance, and adding multimodal inputs at inference did not improve predictions as the authors expected.
• Class-group strengths and weaknesses: TerraMind-OVS performs particularly well on land-cover-related categories, while building-related categories remain challenging because of many small masks — an area where GeoPixel is strongest. Baselines tend toward overprediction, producing high average recall.
• Much faster inference: TerraMind-OVS requires 52–73 ms per sample on a single A100, versus 310 ms for SAM-3, 319 ms for DINO Soars, 432 ms for DINOv3.txt, 488 ms for SegEarth-OV, 524 ms for RemoteSAM, and 6,298 ms for GeoPixel.
• Competitive on multi-label benchmarks: In multiclass experiments, the models outperform generalist models by up to 7.78 percentage points mIoU on reBEN-7k and perform competitively on out-of-distribution grayscale SAR imagery from CaFFe. Scaling behavior is less pronounced on these multi-label tasks than on TerraMesh-Masks-Eval.
• Dataset construction statistics: Starting from roughly 93 million candidate masks, heuristic filtering discards 43.5 million (46.4%), LLM processing covers 20.17 million (21.5% of the raw pool), and 16.07 million are accepted — a 79.7% acceptance rate at a quality threshold of 0.5.
• Vocabulary breadth: More than 44,000 unique class labels and over 537,000 unique synonym strings, with frequent labels such as shrubland, forest, grassland, barren land, and river, and long-tail concepts such as coral reef, golf course, and airport taxiway.
• Reviewed noise levels are substantial: Across 4,211 reviews, 44% of verdicts were valid, 29% invalid, and 28% unsure; the authors caution that the invalid rate is not directly the training-set error rate, since some masks are correct but unverifiable from the image alone.

Methodology in Plain English

The authors start from TerraMesh, an existing collection of about nine million co-registered satellite patches at 10 m resolution from Sentinel-1, Sentinel-2, DEM, and other layers. They spatially intersect these patches with two map sources — Overture Maps (a land-cover-oriented base layer with standardized tags) and OpenStreetMap (rich but heterogeneous volunteer-contributed tags) — to obtain candidate polygons that can serve as segmentation masks.

Because the raw pool is about 93 million candidates, they first apply cheap heuristic filters: dropping masks smaller than a default of 1,000 m², merging adjacent polygons with the same label, and attaching country and continent metadata. They then sample up to three candidate masks per image using a weighted score combining a frequency-based quality prior (global and local), an NDVI agreement score that measures whether the polygon's interior looks spectrally different from its surroundings, and a term that downweights overrepresented tags.

Each sampled candidate goes to a multimodal LLM, Mistral-Large-3, which sees an RGB patch with the polygon overlaid plus metadata such as area, country, continent, and source tags. The prompt casts the model as a conservative annotator that treats the map tags as the primary signal, penalizes ambiguity, and rejects administrative concepts not visible in imagery. The model returns a justification, a quality score between 0 and 1, a natural-language class label, and 2–4 synonyms; masks scoring above 0.5 are kept. Annotation cost was around 4,000 NVIDIA H200 GPU hours for 20 million sampled masks.

For the benchmark, four experts performed over 4,200 reviews on 2,300 candidate masks, with pairwise agreement between 62% and 83% when ignoring "unsure" verdicts. Only masks with at least two valid reviews and no invalid review were kept, duplicates of the same image were removed by preferring the rarer class, and each class was capped at 10 samples, yielding 523 masks with 207 class labels. Overlapping samples were removed from the training corpus to prevent leakage.

To show the dataset's value, the authors continuously pretrain TerraMind at small, base, and large sizes, treating binary masks as an extra modality: masks are tokenized into 196 discrete tokens on a 14×14 grid, and the text query is encoded with CLIP and mapped into five learned query tokens that condition the decoder. Training used 224×224 crops, class-dependent repeat-factor resampling for imbalance, and 15 billion tokens (3% of TerraMind's original training); the large model required about 250 NVIDIA A100 80GB GPU hours.

Why This Matters

This work shifts OVS for Earth observation from small, high-resolution aerial benchmarks toward planetary-scale, multimodal, semantically diverse supervision. The finding that a domain-specific dataset yields large gains over off-the-shelf generalist models — and over EO-specific baselines such as SegEarth-OV — gives the field a concrete reference point and a public benchmark for comparing future methods. The dataset is released under a permissive license with ODbL-derived annotations, which lowers the barrier for reproducible research.

Real-world applications include:

• Land-cover and ecosystem monitoring at national or global scale using flexible natural-language queries instead of fixed taxonomies.
• Infrastructure and settlement mapping — transportation, utilities, industry, and civic/institutional categories — for planning and humanitarian response.
• Rapid custom mapping where a user can query concepts such as "quarry," "power substation," or "mangrove forest" without collecting and labeling new training data.
• Cost-efficient large-area inference, given inference times of 52–73 ms per sample versus seconds for competing models such as GeoPixel.

For industry, the practical relevance is that a single model can serve many mapping tasks across 174 countries, using freely available Sentinel data, without per-task fine-tuning — and at a fraction of the compute cost of generalist vision–language segmentation systems.

Future Directions

• Handling dynamic phenomena: The vocabulary currently covers only persistent map features; segmenting clouds, floods, fire, and seasonal change would require annotations time-stamped to image acquisition, which the authors identify as future work.
• Reducing annotation noise: Wrong labels, imprecise boundaries, and concepts unresolvable at 10 m remain in the training data, and the manual review shows a substantial invalid rate; better filtering or human-in-the-loop verification is an open problem.
• Fixing geographic bias: Coverage mirrors OpenStreetMap density, with North America (126 samples) and Europe (101) dominant and Africa contributing 23 samples; only 13 of the 523 evaluation samples lie above 60° latitude, and high latitudes and underrepresented regions need better representation.
• Better multimodal architectures: Since adding Sentinel-1 and DEM at inference reduced performance rather than helping, the authors explicitly call for more sophisticated multimodal architectural designs for multimodal OVS.

Target Audience

Remote sensing and computer vision researchers working on segmentation, vision–language models, and geospatial foundation models; practitioners building land-cover or infrastructure mapping pipelines from Sentinel data; and benchmark or dataset curators interested in how LLM-assisted, map-grounded annotation can be scaled to millions of samples. Readers should be comfortable with standard segmentation metrics and multimodal model architectures, though the dataset description and evaluation protocol are accessible to newcomers.

Authors’ abstract

Open-vocabulary segmentation (OVS) enables models to segment a variety of concepts specified in natural language, removing the need for fixed label sets and fine-tuning. While recent OVS approaches have shown strong performance on natural images, they transfer poorly to Earth observation (EO) data due to differing semantics, scales, sensing modality, and geographic context. To date, large-scale EO-specific datasets and benchmarks for OVS remain scarce. We, therefore, introduce TerraMesh-Masks, a large-scale open-vocabulary segmentation dataset for EO, comprising 16 million binary masks aligned with nine million multimodal TerraMesh samples and annotated using OpenStreetMap and Overture-derived semantics. In addition, we provide an expert-reviewed evaluation set of 523 samples covering over 200 classes to enable robust evaluation. To demonstrate the impact of TerraMesh-Masks, we build an open-vocabulary version of TerraMind. Our results demonstrate significantly improved performance over general-purpose state-of-the-art models reaching 35.18% IoU versus 16.29% for DINOv3.txt and 11.33% for SAM-3. We release the dataset publicly under a permissive license at https://huggingface.co/datasets/ibm-esa-geospatial/TerraMesh-Masks.