UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenes

AI summary

Overview

Research area: Computer vision and remote sensing — specifically instruction-driven (language-guided) image segmentation of geospatial imagery using multimodal vision-language models.

Technical level: Advanced. The paper assumes familiarity with segmentation architectures (Mask2Former-style decoders), transformer-based vision encoders, large language models, and multimodal training pipelines.

Scope: This paper introduces a million-scale instruction-driven segmentation dataset for remote sensing (GeoSeg-1M), a curated evaluation benchmark (GeoSeg-Bench), and a unified baseline model (UniGeoSeg) that handles referring, interactive, and reasoning segmentation within a single framework.

What This Paper Is About

Existing methods for segmenting remote sensing imagery from natural language instructions are built for isolated, single-purpose tasks, so they cannot share knowledge across task types or generalize to complex, open-world geospatial scenes. They are also trained on small, narrowly scoped datasets that lack the linguistic and visual diversity needed to support genuine instruction understanding. This paper's goal is to unify these fragmented tasks — referring, interactive, and reasoning segmentation — under one dataset, one benchmark, and one model that generalizes to unseen scenes and instructions.

Key Contributions

1. GeoSeg-1M, the first million-scale instruction-driven segmentation dataset for remote sensing, containing 590,413 images, 117 semantic categories, and 1,148,504 image–mask–instruction triplets spanning referring, interactive, and reasoning segmentation. It was built via an automated pipeline that filters noisy masks and generates instructions with GPT-4o, cross-validated by open-source vision-language models (InternVL3-78B, QwenVL2-72B).

2. GeoSeg-Bench, a manually cross-validated benchmark of 6,892 samples (2,870 interactive, 2,311 referring, 1,711 reasoning) with patch-level deduplication from GeoSeg-1M, designed to stress contextual understanding and spatial reasoning in complex geospatial scenes.

3. UniGeoSeg, a unified vision–language segmentation framework combining three components: Task-Adaptive Text Enhancement (TATE), Latent Knowledge Memory (LKM), and Progressive Task Scheduling (PTS).

4. State-of-the-art empirical results across GeoSeg-Bench, EarthReason, and RRSIS-D, plus strong zero-shot generalization on interactive segmentation (SIOR) and visual grounding (DIOR-RSVG).

Main Findings

• Existing models fail badly without training: On GeoSeg-Bench, untuned general-purpose and remote-sensing models score near-zero on interactive segmentation (LISA: 2.52 cIoU; PixelLM: 0.08 cIoU) and remain weak on reasoning, revealing that current systems lack the contextual and spatial reasoning required for geospatial instructions.

• Fine-tuning helps but is not sufficient: Fine-tuning competing models (PSALM, GeoPixel, Earthmind, LISAT, SegEarth-R1) on GeoSeg-1M produces large gains, yet all still trail UniGeoSeg. On reasoning, the best competitor reaches 53.31 cIoU versus UniGeoSeg's 58.35.

• UniGeoSeg leads on GeoSeg-Bench: 74.44/75.56 (cIoU/gIoU) on interactive, 72.93/74.58 on referring, and 58.35/53.12 on reasoning — best in every category.

• Gains on public benchmarks: On EarthReason test, UniGeoSeg scores 74.61 cIoU and 73.08 gIoU, an absolute improvement of 6.65 and 2.16 percentage points over the previous best. On RRSIS-D it reaches 69.25 gIoU, the strongest result among compared methods (though GeoPixel retains a higher cIoU at 81.77).

• Strong zero-shot transfer: Without any target-domain training, UniGeoSeg achieves 87.42 cIoU / 88.77 gIoU average on SIOR interactive segmentation, outperforming SAM2 (80.34/84.36) and HQ-SAM (79.12/83.27) on average and by a wide margin on point prompts (84.61 cIoU vs. SAM2's 71.47). On zero-shot visual grounding (DIOR-RSVG), it reaches 70.82 cIoU and 59.67 gIoU versus 31.28/34.49 for the prior best method.

• Every component contributes: Removing TATE and LKM together drops performance; adding both raises interactive gIoU by 2.33, referring by 1.75, and reasoning by 4.18. Task-specific branches in TATE outperform a shared branch. LKM works best with 4 memory slots and a fusion weight of 0.2. PTS improves referring (+0.07) and reasoning (+0.31) gIoU without harming interactive (+0.01).

• Reasoning remains the hardest task: Even the best model reaches only ~58 cIoU on reasoning, showing that attribute- and context-driven geospatial reasoning is still far from solved.

Methodology in Plain English

The authors first assembled raw material: they gathered over twenty public remote sensing datasets with pixel or region annotations, converted everything into a single annotation format, and turned bounding boxes into masks where needed. Because raw masks are often fragmented and imprecise, they ran a filtering step that split each mask into connected pieces, discarded unreliable regions, and used a vision-language model (InternVL3) to judge which masks were worth keeping.

Instructions were then generated for three task types. For reasoning, GPT-4o wrote questions that require either attribute knowledge (what an object does, what it is made of) or contextual reasoning (how objects relate spatially to one another). For referring, prompts were designed to encourage spatial and relational descriptions rather than direct naming. For interactive, instructions were derived mechanically from mask geometry — bounding boxes and sampled points. All generated text was cross-scored by two open-source language-vision models, and only high-quality samples survived.

For the model, they used a standard three-part architecture: a Swin-B visual encoder (frozen during training) for image features, a Phi-1.5 language model for interpreting instructions, and a Mask2Former decoder for producing masks. On top of this, TATE preprocesses text differently depending on the task — injecting coordinates for interactive prompts, using a learnable attention query to highlight key phrases for referring, and applying multi-query attention plus global aggregation for reasoning. LKM maintains a small set of learned memory slots that all tasks share, retrieving relevant prior knowledge and blending it with the language model's output. PTS adjusts the training mixture over time, starting with more easy interactive samples and gradually shifting weight toward harder reasoning samples, similar to curriculum learning. Training used eight A800 GPUs, batch size 16, three epochs, and 512×512 inputs.

Why This Matters

This work directly addresses the fragmentation problem in geospatial segmentation research: rather than building another single-task model, it provides shared data, a shared benchmark, and a shared baseline, which lowers the barrier for future comparison and lets researchers build on a common foundation. The reasoning results, in particular, show that models can be pushed beyond simple "find the object named X" toward genuinely interpreting instructions that require world knowledge and spatial context.

Real-world applications:

• Urban planning and infrastructure monitoring: Automatically delineating buildings, roads, and land-use zones from satellite imagery based on spoken or written descriptions, without hand-crafted task-specific models.
• Environmental monitoring and land-cover mapping: Tracking forests, water bodies, agricultural land, and vegetation changes across large regions with flexible instruction-driven queries.
• Disaster assessment: Rapidly segmenting damaged infrastructure, flood extents, or debris fields from post-event imagery using natural language guidance, potentially by non-specialist responders.
• Interactive geospatial analysis tools: Enabling GIS analysts to click points or draw boxes and receive instant segmentations that generalize beyond the specific objects a tool was trained on.

Industry relevance: satellite imagery providers, GIS software vendors, agricultural technology companies, and government Earth-observation agencies all rely on segmentation pipelines that currently require per-task retraining. A unified, zero-shot-capable model with an open dataset behind it could substantially reduce the engineering cost of deploying new segmentation capabilities, and the benchmark gives commercial and academic teams a common way to measure progress.

Future Directions

• Closing the reasoning gap: Reasoning segmentation remains the weakest task by a wide margin (around 58 cIoU for the best model). Extending the reasoning data beyond attribute and contextual questions — toward temporal reasoning, causal inference, and multi-step deduction — is a natural next step.
• Temporal and multi-image segmentation: GeoSeg-1M is built from static images. Extending instructions to time series or image pairs would enable change detection and monitoring tasks driven by language.
• Reducing dependence on proprietary models for data generation: The instruction pipeline relies heavily on GPT-4o with open-source models only for verification. Building an open, reproducible generation pipeline would make dataset expansion cheaper and more broadly accessible.
• Multilingual and more diverse instructions: The current instructions are English-only and average roughly twelve words. Expanding to other languages and to longer, more conversational instructions would test whether unified models truly generalize across linguistic diversity.

Target Audience

Remote sensing and computer vision researchers working on segmentation, vision-language models, or multimodal learning will find the dataset, benchmark, and architectural components most directly useful. Practitioners building geospatial analysis systems — GIS engineers, Earth-observation analysts, and applied machine learning engineers — will benefit from the practical zero-shot results and the open-source release. Graduate students entering the field of instruction-driven segmentation will find GeoSeg-1M and GeoSeg-Bench a valuable starting point for experimentation, and researchers studying curriculum learning or cross-task knowledge transfer will find the TATE, LKM, and PTS designs instructive as concrete case studies.

Authors’ abstract

Instruction-driven segmentation in remote sensing generates masks from guidance, offering great potential for accessible and generalizable applications. However, existing methods suffer from fragmented task formulations and limited instruction data, hindering effective understanding and generalization. To address these issues, we introduce GeoSeg-1M, the first million-scale dataset for remote sensing instruction-driven segmentation, constructed via an automatic mask filtering and instruction generation pipeline that synthesizes referring, interactive, and reasoning segmentation instructions from multiple public datasets. GeoSeg-1M contains 590K images, 117 categories, and 1.1M image-mask-instruction triplets. Building upon this foundation, we further curate GeoSeg-Bench, a challenging benchmark designed to evaluate contextual understanding and reasoning capabilities across diverse instruction-driven tasks and complex geospatial scenes. Furthermore, we present UniGeoSeg, a unified framework that serves as a strong baseline, incorporating task-aware text enhancement, latent knowledge memory, and a progressive training strategy to facilitate multi-task learning. Extensive experiments demonstrate the state-of-the-art performance of UniGeoSeg across GeoSeg-Bench and diverse public benchmarks, while exhibiting strong zero-shot generalization. Datasets and source code were released at https://github.com/MiliLab/UniGeoSeg.