AegisBench: Auditing Aerial Search-and-Rescue Person Detection Under the Conditions It Is Deployed In

AI summary

Overview

Research area: Computer vision for aerial search-and-rescue (SAR) — specifically, robustness auditing of person detectors under disaster conditions.

Technical level: Intermediate. The paper assumes familiarity with object detection basics (recall, mAP, IoU, confidence thresholds) but explains its benchmark design and protocol in largely self-contained terms.

Scope (one sentence): The paper builds and releases AegisBench, a disaster-grounded corruption benchmark that evaluates three architecturally distinct person detectors across 168 conditions on two aerial SAR datasets using deployment-realistic frozen thresholds, and reports that robustness is highly uneven — with low light collapsing recall to exactly zero.

What This Paper Is About

Drone-based person detection is being adopted by emergency-management agencies based on benchmarks (HERIDAL and SARD) that were collected in calm, clear, well-lit daylight — while the missions that trigger deployment happen in floods, wildfires, storms, and at night. The authors measure that gap by applying nine physically motivated corruptions from four disaster families, at three calibrated severities each, to both datasets and scoring three detectors under one shared evaluator. The goal is not to propose a better detector but to produce a reusable audit instrument that shows how fielded systems actually degrade, and to give agencies a defensible reporting practice.

Key Contributions

1. A disaster-grounded corruption benchmark. Nine corruptions across four disaster families (flood, wildfire, storm, earthquake), each at three severities monotone in a declared physical or image statistic, deterministically seeded via SHA-256 hashes so corrupted test sets regenerate bit-identically from public source data.

2. A deployment-realistic audit protocol and its results. Frozen operating points (threshold selected once on clean validation data), one shared evaluator, bootstrap intervals, and a localization-stability axis, applied to Faster R-CNN, YOLOv11, and RT-DETR across 168 conditions, with every per-condition number released.

3. A cross-architecture low-light collapse, checked against the obvious alternative explanations (operating point, training seed, synthetic optics) and against targeted augmentation — which moves the synthetic floor at an architecture-dependent cost elsewhere.

4. Evidence that the failure is silent, plus a concrete condition-stratified reporting practice for systems used in disaster response, and a low-cost reusable audit instrument (MIT license, single 8 GB GPU, any detector behind a common interface).

Main Findings

• Clean accuracy looks fine. Clean recall is 0.745 to 0.792 on HERIDAL and 0.879 to 0.891 on SARD — every model above 0.74.

• Robustness is highly uneven. For YOLOv11 on SARD at severity 3, recall ranges from 0.855 under rain_streaks (a 3.6-point drop from the 0.891 clean baseline) to exactly 0.000 under low_light.

• Low light is a total collapse. A 4.5-stop exposure loss (4.47 stops at severity 3) drives recall to exactly 0.000 for all three detectors on both datasets; every one of 1,000 bootstrap resamples returns zero, so each interval is degenerate at [0.000, 0.000].

• Not a threshold artifact. Threshold-free mAP50 collapses in step: on HERIDAL at severity 1, RT-DETR falls from 0.835 clean to 0.001, YOLOv11 from 0.803 to 0.064, and Faster R-CNN from 0.785 to 0.537; by severity 3 no model on either dataset exceeds 0.010. The detections are absent, not hidden.

• Not a single-run fluke. Retraining SARD YOLOv11 at seeds 17, 42, and 123 gives clean recall of 0.891, 0.874, and 0.885, severity-1 recall of 0.184, 0.200, and 0.234, and exactly 0.000 at severity 3 in all three runs.

• A real night-imagery counterpart. The SARD-trained YOLOv11, at its frozen threshold and without retraining, found 2 of 759 annotated persons (recall 0.003) across 300 real night images from the VisDrone-DET test-dev split. The authors note part of this is ordinary domain shift, since VisDrone night scenes are urban and artificially lit.

• The failure is silent. For YOLOv11 on SARD, localization stability stays in a narrow band near the top (never below 0.78) while recall sweeps from 0.89 to 0.00; the count of persons detected in both conditions tracks recall exactly (202, then 8, then 0). Detections disappear rather than drift off the person, so an empty frame is indistinguishable from a correctly searched empty one.

• Flood is the mildest family everywhere. It is the one ordering that holds across the whole study; storm is worst on HERIDAL and earthquake dust worst on SARD. But water_glare fails partly by adding false alarms: on SARD at severity 3, Faster R-CNN's precision falls from 0.967 to 0.513 as false positives rise from 33 to 655.

• Spatial corruptions do not transfer across capture regimes. rain_streaks and motion_blur are mild on SARD (0.778 to 0.855 and 0.329 to 0.478 at severity 3) and near-total on HERIDAL (0.003 to 0.056 and 0.000 to 0.003). The reversal is not attributed to altitude: at severity 3 the blur kernel is 100 pixels on a 4000×3000 HERIDAL frame and 16 pixels on a 640×640 SARD frame, while most persons in both sets fall in the same 32²–96² pixel size band.

• Similar global statistics, different outcomes. smoke_haze and dust_haze reach nearly identical RMS contrast at severity 3 (0.0475 versus 0.0489) and similar optical depth, yet YOLOv11 on SARD keeps 0.391 recall under one and 0.058 under the other; dust_haze costs more recall in all 18 model, dataset, and severity comparisons.

• Targeted augmentation moves the synthetic floor, at a price. Fine-tuning on low_light severities 1–2 (severity 3 never seen in training) raises held-out severity-3 recall to 0.121 for Faster R-CNN, 0.059 for RT-DETR, and 0.188 for YOLOv11, with severity-3 mAP50 rising from below 0.002 to 0.152, 0.134, and 0.368. Across the 24 corrupted conditions outside low_light, Faster R-CNN improves (+0.037 mean), YOLOv11 loses a little (−0.019, 8 of 24 worse by more than 0.03), and RT-DETR pays a real tax (−0.043, 17 of 24 worse by more than 0.03). On the real night images, the augmented YOLOv11 found 4 of 759 rather than 2 — the synthetic fix does not move the real floor.

Methodology in Plain English

The authors take two existing aerial SAR datasets and digitally recreate the visual signatures of disasters on top of them. Each corruption is built on a documented optical model rather than an arbitrary image filter: the Koschmieder atmospheric scattering model for smoke, firelit haze, and mineral dust; a linear photometric treatment for low light (gamma 2.2 decode, photon scaling, twilight blue shift, shot noise and read noise, re-encode); an additive directional streak model for rain; a linear kernel for wind-induced platform shake. Sediment-laden water is handled separately as a blend toward mud chromaticity plus a standing-water layer and specular sun-glint, since it is not a scattering process.

Crucially, each corruption's three severities are calibrated against a declared, machine-checked image statistic that must move monotonically — for example, RMS contrast falling from 0.1170 to 0.0776 to 0.0475 for smoke, or mean luminance falling from 0.2355 to 0.1533 to 0.1175 for low light. Calibration is verified on a fixed random 20-image HERIDAL subsample. rain_streaks is the one exception, since streak density has no clean closed-form image measure; its severity is defined by generation parameters and checked visually.

Three detectors are fine-tuned from pretrained weights — one training run per model per dataset — and receive identical inputs (1024-pixel tiles with 256-pixel overlap on HERIDAL, so 20 tiles per frame; whole 640-pixel frames on SARD). Each model's confidence threshold is chosen once by maximizing F1 on clean validation data and then frozen across all 28 conditions for that model and dataset, matching how a system is fielded. All models are scored by a single evaluator (greedy score-ordered matching at IoU ≥ 0.5, class-agnostic, one prediction per ground-truth box) so no architecture is judged by its own framework's metric. Degradation is reported relative to each model's own clean score, with bootstrap 2.5th–97.5th percentile intervals over 1,000 resamples. Everything runs on one consumer GPU with 8 GB of memory.

Why This Matters

Impact on research. The paper reframes robustness reporting for safety-critical detection: a single averaged robustness score, the form a vendor is most likely to report, "would compress the whole finding into one number" and would hide both the zero-recall floor and architecture-dependent mitigation costs. It also shows that detection difficulty is not reducible to a global statistic like contrast, and that spatial robustness figures are not portable between imaging configurations.

Real-world applications:

• Emergency-management agencies procuring drone detection systems, who currently compare vendors on clean-condition accuracy that does not describe the missions they fly.
• Night and low-visibility search operations, where the measured failure is total and undetectable from the output — motivating flight-time ambient-light monitoring and thermal infrared as a complementary sensor.
• Flood and wildfire response, where water_glare adds hundreds of false positives (33 to 655 on SARD at severity 3) and spends operator attention even when recall holds up.
• Third-party auditors and regulators of consequential AI, since the released pipeline reports per-condition recall, a threshold-independent metric, image and target sizes, and precision for any detector wrapped behind its evaluator.

Industry relevance. Model cards already ask for results disaggregated by relevant conditions; for aerial SAR this paper supplies the disaggregated environment numbers that were previously missing, and the tooling to regenerate them on a single 8 GB GPU under an MIT license.

Future Directions

1. Target-relative geometry control. Does the rain and blur reversal between HERIDAL and SARD survive when spatial parameters are set relative to target size instead of frame size? The authors did not run this control and explicitly report the reversal as a property of the benchmark's geometry, not of the capture regimes.

2. Isolating the low-light mechanism. Which part of the dust model carries its effect — airlight chromaticity, transmission granularity, or near-lens grain — is not isolated. Likewise, whether the low-light zero reflects a detector limit or a visible-band sensor limit is not settled; severity 3 is a dim dusk exposure, and augmentation recovers only part of it.

3. A matched daytime VisDrone control. The real night check cannot separate darkness from domain shift because VisDrone night scenes are urban and artificially lit, with pedestrians differing from SARD's in scene and scale. The authors flag this as "the first check to add."

4. Compounding conditions and broader coverage. Corruptions are applied one at a time while a real night flood combines several; the seed and augmentation studies cover SARD only; and the authors note four open questions remain, of which the excerpt shows the geometry and low-light-mechanism items.

Target Audience

Most useful to: computer-vision researchers working on robustness benchmarking or small-object detection; engineers and procurement staff evaluating drone-based detection for emergency response; policy and standards people working on model cards and condition-stratified reporting for consequential AI; and practitioners who want a reproducible audit they can rerun on their own detector — the corruption engine, evaluator, configurations, datasheet, and per-condition results are released under an MIT license, and the benchmark runs on a single 8 GB GPU. Readers wanting a new architecture will not find one here; the paper deliberately proposes no detector.

Authors’ abstract

Drone-based person detection is being adopted by emergency-management agencies to find missing people faster than ground teams can. The benchmarks behind that adoption were collected in calm, clear daylight, while the missions that trigger deployment happen in floods, wildfires, storms, and at night. AegisBench measures that gap. It applies nine physically motivated corruptions from four disaster families, each at three severities monotone in a declared image statistic or physical quantity, to two aerial search-and-rescue datasets, HERIDAL and SARD. Three architecturally distinct detectors (Faster R-CNN, YOLOv11, RT-DETR) are scored across 168 conditions by one shared evaluator, at confidence thresholds frozen on clean validation data as a fielded system would run. Robustness is highly uneven. Low light is the extreme case: a 4.5-stop exposure loss drives recall to exactly zero for all three detectors on both datasets, a result that holds under threshold-free mAP and across three training seeds, and is mirrored on 300 real night drone images, where 2 of 759 people are found. Surviving detections largely stay on target, so the failure is silent: an empty frame looks the same as one whose occupants the system cannot see. Fine-tuning on milder low-light levels recovers 0.06 to 0.19 recall at the held-out severity, at an architecture-dependent cost on other conditions, and does not transfer to real night imagery. Spatial corruptions rank differently on the two datasets, which we cannot attribute to altitude alone because their parameters scale with frame size rather than person size. We release the corruption engine, evaluation pipeline, and all per-condition results as an audit instrument for anyone evaluating these systems.