A Benchmark Dataset for Spatially Aligned Road Damage Assessment in Small Uncrewed Aerial Systems Disaster Imagery

AI summary

Overview

Research area: Computer vision and machine learning for disaster response, specifically road damage assessment from small uncrewed aerial systems (sUAS) imagery, plus benchmark dataset construction and spatial data alignment.

Technical level: Intermediate. Readers should be comfortable with semantic segmentation, Intersection over Union (IoU), F1 scores, and the general practice of training and evaluating segmentation models; the paper is accessible to practitioners in emergency management who want to understand what the models can and cannot do.

One-sentence scope: The paper releases a 657.25 km road damage assessment benchmark built on post-disaster sUAS imagery from 10 federally declared disasters, adds 9,184 spatial alignment annotations for a priori road lines, provides 18 baseline models, and operationally validates one of them during the 2024 responses to Hurricanes Debby and Helene.

What This Paper Is About

Automating road damage assessment from drone imagery could tell emergency managers which roads are flooded, obstructed, or destroyed, but prior public datasets were either small-scale or built on low-resolution imagery that cannot show the conditions emergency managers actually care about, and no known road damage model had been operationally validated. This paper builds a larger, practitioner-relevant benchmark (657.25 km of labeled roads, a 10-class schema developed with the U.S. Federal Emergency Management Agency and the Texas Department of Transportation) and adds a second, previously unaddressed ingredient: correcting the misalignment between a priori road lines and the actual roads visible in sUAS imagery. The authors then train 18 baselines and deploy one during real hurricane responses to measure both quantitative performance and practitioner feedback.

Key Contributions

1. Largest known road damage assessment dataset for sUAS imagery. 657.25 km of labeled roads across imagery from 10 federally declared disasters, drawn from 52 orthomosaics in the CRASAR-U-DROIDs collection.
2. First dataset to address spatial alignment errors with a priori road lines. 9,184 road line adjustment annotations were collected, correcting misalignment that is non-uniform and specific to sUAS imagery.
3. A practitioner-relevant 10-class road condition schema. Developed with input from federal and state agencies in the United States so that model outputs map onto operational decisions rather than only pavement health or flood extent.
4. 18 baseline models plus one operational validation. Nine architectures (eight trainable plus a random baseline), each evaluated on "Simple" and "Full" prediction tasks, with the Attention UNet configured for the "Simple" task deployed in Florida during Hurricanes Debby and Helene in 2024.

Main Findings

• Baseline performance is low in absolute terms. The best trainable model, UNet with Attention, reached 0.331 macro IoU_km and 0.393 F1_km on the "Simple" task using aligned data, and 0.091 IoU_km / 0.095 F1_km on the "Full" task. All trainable models beat the random baseline (0.135 IoU_km / 0.215 F1_km on "Simple"; 0.016 / 0.031 on "Full"), but the authors state performance remains low compared to prior work such as RescueNet and FloodNet.
• Performance degrades as the number of classes grows. The paper reports that model performance appears to degrade as the number of prediction classes increases from the 3-class "Simple" formulation to the exact-label "Full" formulation.
• Spatial misalignment measurably hurts models. When the 18 baseline models are deployed against real-world misaligned road lines, performance degrades on average by 5.596% Macro IoU. In the controlled test-set comparison, the top model degrades by 1.9 Macro IoU; averaging across model lines, adjusted-versus-unadjusted degradation is 0.014 IoU_km and 0.018 F1_km for "Simple," and 0.0003 IoU_km and 0.0004 F1_km for "Full."
• Misalignment has concrete data-quality costs. Approximately 59 km (9%) of road lines were misaligned off the actual road, and approximately 11 km (8%) of adverse conditions on road lines would be labeled incorrectly if spatial alignment is not considered. Applying alignment moves road line vertices by 396 meters across all 657.25 km of road lines (0.06%).
• Labels are heavily imbalanced. The label distribution is plotted on a log-scale y-axis, and the per-label results show why: in the "Simple" configuration, the UNet with Attention scored 0.825 IoU_km (unadjusted) and 0.839 (adjusted) on the Road Line class, while every specific damage class in the "Full" configuration scored 0.000 for most models. Many "Total" classes (Total Destruction, Total Flooding, Total Obstruction) and several "Partial" classes were not recovered at all by the trainable baselines.
• Operational deployment was useful despite error rates. The Attention UNet ("Simple") was deployed in Florida during Hurricane Debby (Category 1, August 2024) and Hurricane Helene (Category 4, September 2024); it processed an approximately 3 mi² orthomosaic in 5 minutes on a desktop with an NVIDIA RTX4090, and outputs were converted to KML files for dissemination. Practitioners tolerated false positives far better than false negatives, and acceptance was time-dependent: they tolerated more errors when outputs arrived earlier. Practitioners consistently said they would have preferred a full schema, and no single model performance value was deemed acceptable in practice.
• Manual review corrected substantial label noise. Of the tiles, 1,710 (9.7%) were rejected and re-annotated or corrected during individual review; spot-check corrections changed 78.5 km (12%) of road line labels; manually added road lines not sourced from OpenStreetMap accounted for 1.156 km (0.17%) of road lines.
• Annotation counts. Annotation of the dataset involved a pool of 130 annotators, and 656.094 km of road was annotated before the manual additions brought the total to 657.25 km.

Methodology in Plain English

The authors started from CRASAR-U-DROIDs, described in the paper as the largest known collection of sUAS orthomosaic imagery, comprising 52 orthomosaics from 10 federally declared disasters with resolutions ranging from 12.7 cm/px to 1.93 cm/px (the evaluation section states resolutions between 12.7 cm/px and 1.77 cm/px). They tiled each orthomosaic at 2048x2048 (45 orthomosaics) and 8500x8500 (7 orthomosaics), overlaid existing road lines from OpenStreetMap, and had 130 annotators label road conditions in Labelbox using the 10-class schema. The smaller tiles were annotated by the annotation pool; the 8500x8500 tiles were annotated by the authors.

Because annotator polygons were inconsistent in shape and sometimes did not correspond to any road line, the authors computed intersections between all road lines and all annotation polygons and generated new rectangular polygons parallel to the road line with a width of 7.2 meters, the reported upper limit on the width of two lanes of rural or urban road in the United States. This standardization both matched polygon dimensions to real road dimensions and created a consistent region for absorbing reasonable spatial misalignment.

Review happened in two stages: individual reviewers inspected each tile (rejecting 1,710 tiles, 9.7%), then a committee of three authors and one external reviewer examined annotations overlaid on orthomosaics and made spot-check corrections. Reviewers had operational experience as an emergency vehicle operator, an sUAS data manager, and an sUAS pilot. Missing road lines were manually added where reviewers judged them necessary.

For spatial alignment, the authors used the same vector field formulation as prior work but operated on individual road line vertices instead of building polygons, producing 9,184 road line adjustments. The annotation format deliberately separates road lines from annotation polygons, so road line labels can be generated by intersecting the two, decoupling the ground truth from any particular alignment.

For the models, road damage assessment is framed as segmentation. Road lines were masked by buffering all road lines by 40 pixels into a rectangular mask, and all trained models used two class-imbalance strategies at once: weighted sample presentation targeting uniform label propensity, and cross-entropy loss weighted by observed inverse label propensity. Two tasks were defined: "Simple" (three classes: no annotation, "Partial," or "Total," with "Not Able To Determine" folded into the "Total" grouping) and "Full" (the exact ground-truth label). The nine architectures are a Random Baseline, UNet without attention, UNet with attention, ResNet101+PSPNet, ResNet101+DeepLabv3plus, ViT+Segmenter, ViT (pretrained)+Segmenter, ViT+UperNet, and ViT (pretrained)+UperNet.

Evaluation uses IoU and F1 computed along the spatial dimension in kilometers rather than pixels, denoted IoU_km and F1_km, so that results are not biased by imagery at the extremes of the 12.7 cm/px to 1.77 cm/px resolution range. Statistical significance between models was assessed with a Hoeffding Bound at p < 0.001.

Why This Matters

Impact on research. The paper argues there is no current state of practice for post-disaster road damage assessment because prior public datasets were small-scale or reliant on low-resolution imagery, and none of the previously developed ML systems are known to have been operationally validated. It also introduces spatial misalignment of a priori road lines, a problem described as prevalent in operationally collected sUAS imagery and not seen in satellite imagery, as a first-class benchmark concern. Prior schemas (flooded/non-flooded, road blocked/road clear, no/minor/severe damage, and pavement-condition indices) are argued not to capture what emergency managers need.

Real-world applications:

• Routing and navigation of aid and evacuation through impacted areas during the response phase.
• Allocation of aid and resources based on which roads are passable.
• Avoiding spurious repair dispatches and reduced route efficiency caused by alignment-driven false positives.
• Field-deployable sUAS workflows, since the model processed a 3 mi² orthomosaic in 5 minutes on an NVIDIA RTX4090 and outputs were distributed as KML files.

Industry relevance. The work is relevant to drone analytics and remote sensing companies, emergency management agencies, transportation departments, and insurers or infrastructure operators who need rapid post-disaster road status. It also sets a benchmark that segmentation model developers can target, including the finding that models may need large receptive fields to classify labels such as partial obstruction and partial road condition, which contain pixels that would otherwise look like clear road.

Future Directions

• Collecting additional data, potentially including non-disaster imagery, to improve model performance.
• Analyzing the baseline models and experimenting to improve performance, and specifically exploring the performance degradations observed in the "Full" formulation.
• Evaluating the ethical implications of both false negatives and false positives in disaster operations.
• Addressing the two critical challenges the paper identifies for future ML and CV work: spatial alignment of a priori road lines and the distribution shifts present in real-world road damage assessment aerial imagery.

Target Audience

Computer vision and machine learning researchers working on segmentation, remote sensing, and distribution shift will benefit most, along with robotics researchers working on field-deployed sUAS systems. Disaster response practitioners, emergency managers, and transportation agencies will find the schema, the operational deployment findings, and the practitioner feedback on false positives versus false negatives directly relevant. Dataset and benchmark builders should read it for the annotation pipeline, two-stage review process, and alignment-annotation design.

Authors’ abstract

This paper presents the largest known benchmark dataset for road damage assessment and road alignment, and provides 18 baseline models trained on the CRASAR-U-DRIODs dataset's post-disaster small uncrewed aerial systems (sUAS) imagery from 10 federally declared disasters, addressing three challenges within prior post-disaster road damage assessment datasets. While prior disaster road damage assessment datasets exist, there is no current state of practice, as prior public datasets have either been small-scale or reliant on low-resolution imagery insufficient for detecting phenomena of interest to emergency managers. Further, while machine learning (ML) systems have been developed for this task previously, none are known to have been operationally validated. These limitations are overcome in this work through the labeling of 657.25km of roads according to a 10-class labeling schema, followed by training and deploying ML models during the operational response to Hurricanes Debby and Helene in 2024. Motivated by observed road line misalignment in practice, 9,184 road line adjustments were provided for spatial alignment of a priori road lines, as it was found that when the 18 baseline models are deployed against real-world misaligned road lines, model performance degraded on average by 5.596\% Macro IoU. If spatial alignment is not considered, approximately 8\% (11km) of adverse conditions on road lines will be labeled incorrectly, with approximately 9\% (59km) of road lines misaligned off the actual road. These dynamics are gaps that should be addressed by the ML, CV, and robotics communities to enable more effective and informed decision-making during disasters.