OSMGen: Highly Controllable Satellite Image Synthesis using OpenStreetMap Data

AI summary

Overview

Research area: Computer vision / generative AI applied to remote sensing and geospatial data, specifically conditional satellite image synthesis and training-free image editing.

Technical level: Advanced. The paper assumes familiarity with diffusion models, latent diffusion, ControlNet, cross-attention conditioning, and DDIM inversion. The concepts are explained clearly, but the methodology builds on a dense stack of prior techniques.

Scope: A one-sentence summary: OSMGen is a ControlNet-based diffusion framework that generates 256x256 satellite image tiles directly from raw OpenStreetMap JSON — including vector geometries, semantic tags, location, and date — and that can produce co-registered before/after image pairs from user edits to the map data.

What This Paper Is About

Urban monitoring and geospatial AI suffer from a shortage of curated datasets covering specific urban features and how those features change over time. Existing satellite image generators condition on raster map tiles or bounding boxes, which flatten away the semantic detail and exact geometry that OpenStreetMap actually stores. OSMGen's goal is to generate realistic, spatially accurate satellite imagery conditioned on the full structured OSM JSON, and to make edits to that JSON produce controlled, consistent visual changes while everything outside the edit stays the same.

Key Contributions

1. Generation conditioned on raw OSM JSON. Rather than rendered raster tiles or bounding boxes, the framework consumes vector geometries, semantic key–value tags, geographic coordinates, and capture dates, enabling fine-grained control over which feature subtypes appear in a generated scene.

2. Co-registered before/after image pairs. Using DDIM inversion and re-denoising under an edited condition, OSMGen produces image pairs where a user edit to the OSM input (adding, removing, or changing an element) translates into a targeted visual change while the rest of the scene is preserved.

3. Paired (JSON, image) data generation. The pipeline emits complete JSON–image pairs for both static and changed states, which the authors position as the data needed to train models that could close the loop — detecting changes in new satellite imagery and proposing structured OSM updates automatically.

4. A publicly released end-to-end pipeline. Source code is available at https://github.com/amir-zsh/OSMGen.

Main Findings

• Mask conditioning drives structural fidelity. The model accurately reproduces road networks and building footprints from the general mask, which groups features into broad categories such as roads, water bodies, vegetation, buildings, and other primary surface types.

• The specific mask captures rare classes. The specific mask, which assigns each fine-grained point-of-interest subtype (e.g., lakes, rivers, storage tanks, solar farms) its own mask color, allows the model to render rare POI classes such as stadiums and storage tanks with correct shapes and context.

• Edits are local and consistent. DDIM inversion-based editing supports operations that add a stadium, add a building, remove some buildings, remove some storage tanks, change a lake to grass, and change a crop field to solar farm. The pipeline produces consistent pairs without introducing artifacts outside the intended changes.

• Temporal conditioning changes appearance, not geometry. Holding the semantic masks and text description fixed for a single location and varying only the date input produces distinct winter and summer images, altering vegetation density, color palette, and lighting while leaving geometry unchanged.

• Classifier-free guidance can be reduced. Strong spatial conditioning from the masks lets the authors lower the CFG scale, which mitigates the known instability of DDIM inversion at high CFG scales.

• Evaluation is qualitative. Results are presented as qualitative figures on held-out FMoW locations; the paper reports no quantitative metrics (no FID, IoU, or similar scores).

Methodology in Plain English

The team started by sampling roughly 20,000 points from the Functional Map of the World benchmark, covering urban centers, suburbs, and rural areas. For each point they fixed a zoom level in advance — typically z=18 for fine structural detail or z=15 for wider context — and computed 256x256-pixel tile bounds around the center latitude and longitude using standard Web Mercator tile formulas. For each tile they retrieved the matching satellite image and the raw OSM JSON.

From that JSON they built a multimodal set of conditions. Two segmentation masks come from the vector geometries: a general mask for broad categories like roads, water, vegetation, and buildings, and a specific mask that gives every fine-grained POI subtype its own color. Location is encoded with SatCLIP, the capture date with Date2Vec, and a short text summary of the tile's most salient categories is encoded with a frozen CLIP text encoder.

The generator is a frozen Stable Diffusion U-Net augmented with a trainable ControlNet branch. The two masks are stacked into a multi-channel tensor, fused through a small convolutional/MLP layer, and fed to ControlNet, whose control features are added into the U-Net blocks to enforce geometry. The spatial and temporal embeddings each pass through a linear projection and are added to the diffusion timestep embedding, while the text embedding enters through cross-attention. Only the ControlNet component, the mask-fusion layer, and the spatial/temporal projections are trained, using the standard diffusion loss where the network predicts the ground-truth noise from the noisy latent given the fused mask and the three embedding types.

For generating "after" images, the authors use DDIM inversion. They give three reasons for this choice: cross-attention editing methods cannot handle their non-textual conditions; DDIM inversion is architecture-agnostic and simple to implement; and the strong mask conditioning permits a lower guidance scale, sidestepping DDIM inversion's instability at high guidance scales. An inversion depth parameter sets edit strength — smaller values preserve more of the original image and give weaker edits, larger values give stronger edits.

Training ran for 500 epochs with a batch size of 2048. Evaluation used approximately 2,000 held-out locations from the FMoW-derived dataset, and all synthesis ran on a single NVIDIA A100 GPU.

Why This Matters

Impact on research. The work reframes map data as a rich conditioning signal rather than a flat image, and it supplies exactly the kind of paired before/after data that change detection and segmentation models need but rarely have. By releasing paired (JSON, image) data for both static and changed states, it opens a path toward a closed-loop system where satellite imagery drives structured map updates rather than the other way around.

Real-world applications:

• Urban planning: planners can preview the visual impact of proposed interventions — a new park, a new building, new infrastructure — simply by editing the JSON map data before anything is built.
• Dataset augmentation: generating vast pixel-perfect labeled tiles to address scarcity and class imbalance for tasks like building footprint segmentation and land-use classification.
• Infrastructure monitoring: supplying training data for detecting storage tanks, solar farms, and similar assets that are underrepresented in existing datasets.
• Environmental management: simulating seasonal and land-cover changes — for instance lake-to-grassland or crop-field-to-solar-farm transitions — for environmental analysis.

Industry relevance. Companies and agencies that maintain geospatial databases, run change detection, or build mapping products face a constant manual cost of keeping maps current. A system that generates co-registered change pairs, and potentially proposes OSM JSON updates from new imagery, targets that cost directly. The availability of code, and the fact that it builds on widely used components like Stable Diffusion and ControlNet, lowers the barrier to adoption.

Future Directions

1. Quantitative evaluation. The paper reports only qualitative results. Benchmarks such as FID or segmentation-based fidelity scores on the held-out FMoW set, and systematic measurement of consistency outside edited regions, would establish how good the synthesis actually is.

2. Closing the loop. The authors explicitly frame automated detection of changes in satellite imagery leading to structured OSM JSON updates as the next step enabled by their paired data. Building and validating that reverse direction is the largest stated open question.

3. Extending controllability. The JSON schema is described as flexible enough for users to select specific feature types, adjust per-class palettes, and introduce new categories of entities — whether that extensibility works without retraining is not demonstrated.

4. Geographic and temporal generalization. The training data derives from FMoW sampling; how the model behaves on regions, feature types, or seasons outside that distribution is not reported.

Target Audience

This paper is most useful to researchers and engineers working on generative models for remote sensing, controllable image synthesis, and diffusion-based image editing. It also speaks to geospatial data scientists and urban planners interested in simulating map interventions, and to machine learning practitioners building change detection or segmentation datasets who need paired, co-registered imagery. Readers should come with background in diffusion models and conditioning mechanisms, since the paper places the relevant technical background in appendices and moves quickly through the architecture.

Authors’ abstract

Accurate and up-to-date geospatial data are essential for urban planning, infrastructure monitoring, and environmental management. Yet, automating urban monitoring remains difficult because curated datasets of specific urban features and their changes are scarce. We introduce OSMGen, a generative framework that creates realistic satellite imagery directly from raw OpenStreetMap (OSM) data. Unlike prior work that relies on raster tiles, OSMGen uses the full richness of OSM JSON, including vector geometries, semantic tags, location, and time, giving fine-grained control over how scenes are generated. A central feature of the framework is the ability to produce consistent before-after image pairs: user edits to OSM inputs translate into targeted visual changes, while the rest of the scene is preserved. This makes it possible to generate training data that addresses scarcity and class imbalance, and to give planners a simple way to preview proposed interventions by editing map data. More broadly, OSMGen produces paired (JSON, image) data for both static and changed states, paving the way toward a closed-loop system where satellite imagery can automatically drive structured OSM updates. Source code is available at https://github.com/amir-zsh/OSMGen.