EIDSeg: A Pixel-Level Semantic Segmentation Dataset for Post-Earthquake Damage Assessment from Social Media Images

AI summary

Overview

Research area: Computer vision for disaster response — semantic segmentation, dataset construction, and post-earthquake infrastructure damage assessment from ground-level social media imagery.

Technical level: Intermediate. The dataset design and annotation protocol are accessible to non-specialists, but the benchmarking section assumes familiarity with segmentation architectures (Mask2Former, BEiT, OneFormer, transformers) and metrics such as mIoU and FWIoU.

Scope: The paper introduces EIDSeg, the first large-scale pixel-level semantic segmentation dataset for post-earthquake damage assessment built from social media photographs, and benchmarks eight modern segmentation models on it.

What This Paper Is About

Rapid damage assessment after an earthquake is critical for rescue and resource allocation, but current methods rely on expensive satellite or drone imagery and expert interpretation, and typically produce only binary damaged/undamaged maps. Ground-level photos posted to social media within minutes of an event capture street-scale damage that aerial views miss, yet no large pixel-level annotated dataset exists for this kind of imagery. This paper builds that dataset, defines a labeling standard non-experts can apply consistently, and establishes baseline performance for future work.

Key Contributions

1. Dataset: EIDSeg, a benchmark of 3,266 post-earthquake social media images from nine major earthquakes (2008–2023), annotated with five-class pixel-level damage masks — Undamaged Building, Damaged Building, Destroyed Building (Debris), Undamaged Road, and Damaged Road.
2. Annotation guideline: A three-phase cross-disciplinary protocol that lets annotators with no civil engineering background produce consistent pixel-level labels, achieving over 70% inter-annotator agreement.
3. Benchmark: A systematic evaluation of eight state-of-the-art segmentation models, establishing EoMT as the strongest baseline at 80.8% mIoU and 90.3% pixel accuracy.
4. Quality control pipeline: An automated sharpness and resolution filter combined with embedded gold-standard images, threshold-based rejection, and expert adjudication, producing a dataset where every image has been reviewed at least twice.

Main Findings

• EoMT is the top performer: The Encoder-only Mask Transformer (ViT-L backbone, Cityscapes pre-training, 1024×1024 input) reaches 80.8% mIoU, 80.9% FWIoU, and 90.3% pixel accuracy, outperforming OneFormer (79.2%), BEiT-L (78.7%), and Mask2Former-L (77.4%).
• Higher resolution helps: EoMT's advantage is attributed partly to training at 1024×1024, which preserves spatial detail that smaller inputs lose.
• Complexity costs the least-capable models: DeepLabV3+ (ResNet-101) trails at 67.1% mIoU, roughly 13 points below EoMT, despite being the most memory-efficient model.
• Buildings are harder than roads or debris: Undamaged Building has the lowest per-class IoU across all models (EoMT: 70.1%), followed by Damaged Building (80.0%). Destroyed Building/Debris is easiest (up to 85.1%), since collapse rubble has distinctive visual texture.
• The five-class dataset is not harder than DSS's three-class version: SegFormer-B5 scored 72% mIoU on DSS's three-class task but 74.4% here on five classes, suggesting the expanded corpus and refined annotation reduce noise.
• Damaged pixels dominate: The dataset contains more damaged than undamaged pixels, reflecting that social media users are more likely to photograph and post visible destruction than intact scenes.
• Cross-event generalization is consistent: Leave-one-event-out testing with EoMT yields 80.76% average FWIoU and 88.64% average pixel accuracy across nine disasters. Chile (86.05%), Ecuador (84.21%), and Türkiye (83.07%) score highest; Haiti (74.46%) and Iraq–Iran (74.86%) lowest, reflecting greater visual variation and less transferable damage patterns.
• Deployment trade-offs differ sharply: On an A100 GPU, Mask2Former offers the best speed/memory balance (~37 ms latency, ~27 img/s). On CPU, DeepLabV3+ is fastest (3.06 img/s), while BEiT-L and EoMT drop below 0.3 img/s, making them impractical for edge deployment.
• Model errors follow patterns: Blurred or low-resolution regions cause systematic failures, and models often fragment a single building into disconnected predicted regions rather than recognizing whole structures.

Methodology in Plain English

The researchers assembled images from two existing sources: the EID dataset (image-level damage labels) and the DSS dataset (607 segment-labeled images). From EID they kept only images labeled as no-damage, mild, or severe damage, discarding irrelevant images such as ads and maps. They then applied an automated quality filter, discarding images whose sharpness (measured by Laplacian variance) fell below 50 or whose resolution was under 512×256 pixels. This left 6,646 images for annotation.

Labeling was done by six annotators with no civil engineering background, working in three independent pairs under two domain experts. The protocol had three phases. First, three weeks of guided training on a 300-image practice set, with weekly scoring against expert masks and iterative refinement of the guidelines. Second, full production labeling, where each round embedded 20–25 "gold standard" images among the assigned batches. Two agreement metrics were tracked: annotator-versus-expert mIoU on gold images, and inter-annotator mIoU on shared images. Annotators scoring below 0.60 expert agreement were retrained; images with inter-annotator agreement below 0.60 were dropped. Third, expert consolidation, in which every "Undesignated" label was manually resolved and images where ambiguity exceeded 2% of pixels were removed.

The damage taxonomy deliberately simplifies civil engineering standards. EMS-98 and HAZUS classify buildings by construction material and structural failure modes, which are usually invisible in user-generated photos. Instead, EIDSeg uses two infrastructure types — buildings and roads — each with two or three visually determined damage levels, plus an Undesignated class to absorb ambiguity. Roads get only two levels because they do not collapse the way buildings do; debris lying on a road is not treated as road damage since it often originates from surrounding terrain.

For benchmarking, the dataset was split 80/10/10 into training, validation, and test sets. Eight models were trained under identical conditions (AdamW, learning rate 1e-5, up to 100 epochs, single A100 GPU, batch size 4), with Cityscapes pre-trained weights for all models except BEiT, which used ADE20K.

Why This Matters

Impact on research: The paper fills a documented gap — no pixel-level segmentation benchmark existed for ground-level post-earthquake imagery, and aerial datasets like xBD and EarthquakeNet do not transfer directly to street-level perspectives. By publishing guidelines that non-experts can follow, it also lowers the barrier to building similar datasets for other disaster types.

Real-world applications:

• Emergency response triage. Automated segmentation of social media photos could route rescue teams and supplies to the hardest-hit blocks within hours of an earthquake, before aerial surveys are flown.
• Damage mapping at street scale. Pixel-level masks distinguish destroyed buildings from damaged roads, which binary damage proxies cannot do, enabling more precise infrastructure status maps.
• Crowdsourced rapid assessment. Since the protocol works with non-expert annotators, relief organizations could scale labeling across large volunteer pools during an active disaster.
• Insurance and recovery planning. Fine-grained damage categories support faster preliminary loss estimation and prioritization of reconstruction funds.

Industry relevance: The computational benchmark matters for deployment. Results show that the highest-accuracy model (EoMT) is slow and memory-heavy, while DeepLabV3+ is the only model practical on CPU-only hardware. Organizations building field-deployable tools must weigh accuracy against latency, and Mask2Former emerges as the best GPU compromise.

Future Directions

• Improve building damage discrimination. Undamaged and Damaged Building classes have the lowest IoU because background buildings are often blurry and the conservative labeling rule (visible cracking implies damage) creates visual overlap between classes. Better structural priors or higher-resolution inputs could help.
• Address image degradation and occlusion directly. All models fragment buildings into disconnected regions and miss background damage. Dedicated robustness techniques for blur, shadows, clutter, and occlusion are needed.
• Expand geographic and event coverage. Provenance is missing for many images in the source datasets, which limited the leave-one-event-out analysis to images with verifiable origins. Fuller metadata would enable stronger generalization studies.
• Reduce annotation cost. The three-phase protocol is thorough but labor-intensive. Semi-supervised or active learning approaches could preserve quality with less expert time.
• Validate with civil engineering ground truth. Comparing model outputs against field inspection data would clarify how well visual segmentation correlates with actual structural condition.

Target Audience

Researchers and practitioners in computer vision, remote sensing, and disaster informatics who work on damage assessment or benchmark dataset construction. Civil and structural engineers interested in how visual damage scales translate to street-level imagery will find the taxonomy discussion useful. Emergency management agencies and humanitarian organizations evaluating automated tools for rapid post-disaster response are a secondary audience, as are graduate students looking for a well-documented example of a cross-disciplinary annotation protocol.

Authors’ abstract

Rapid post-earthquake damage assessment is crucial for rescue and resource planning. Still, existing remote sensing methods depend on costly aerial images, expert labeling, and produce only binary damage maps for early-stage evaluation. Although ground-level images from social networks provide a valuable source to fill this gap, a large pixel-level annotated dataset for this task is still unavailable. We introduce EIDSeg, the first large-scale semantic segmentation dataset specifically for post-earthquake social media imagery. The dataset comprises 3,266 images from nine major earthquakes (2008-2023), annotated across five classes of infrastructure damage: Undamaged Building, Damaged Building, Destroyed Building, Undamaged Road, and Damaged Road. We propose a practical three-phase cross-disciplinary annotation protocol with labeling guidelines that enables consistent segmentation by non-expert annotators, achieving over 70% inter-annotator agreement. We benchmark several state-of-the-art segmentation models, identifying Encoder-only Mask Transformer (EoMT) as the top-performing method with a Mean Intersection over Union (mIoU) of 80.8%. By unlocking social networks' rich ground-level perspective, our work paves the way for a faster, finer-grained damage assessment in the post-earthquake scenario.