DisasterInsight: A Multimodal Benchmark for Function-Aware and Grounded Disaster Assessment

AI summary

Overview

• Research area: Computer vision / remote sensing, specifically vision–language models (VLMs) for disaster response, with a focus on building-level multimodal evaluation.
• Technical level: Intermediate — the paper assumes some familiarity with VLMs, instruction tuning (LoRA), remote sensing datasets, and standard evaluation metrics, though the core idea (assessing individual buildings, not just whole scenes) is explained in accessible terms.
• Scope: The paper introduces DisasterInsight, a building-centric, multimodal instruction-following benchmark built on the xBD satellite dataset, enriched with OpenStreetMap building-function labels and Sentinel-1 SAR imagery, used to diagnose where general-domain and remote-sensing VLMs succeed and fail at disaster assessment.

What This Paper Is About

Existing remote-sensing disaster benchmarks mostly ask whether a whole scene is damaged, flooded, or burned, and they treat buildings as undifferentiated damage-bearing objects. But real disaster responders need to know which specific buildings are affected, what those buildings are used for (a hospital versus a residence matters differently), and how several damaged structures compare in response priority. DisasterInsight addresses this gap by turning xBD building instances into a large instruction-following benchmark with function labels, multi-instance reasoning tasks, counting tasks, and structured report generation, then measuring how well current VLMs handle each.

Key Contributions

1. An instance-centric disaster benchmark. The authors introduce DisasterInsight, which enriches xBD buildings with OpenStreetMap-derived functional labels so that building identity, function, damage state, and response relevance can be evaluated together rather than as a single scene-level label.
2. A multimodal extension with SAR. xBD is extended with co-registered Sentinel-1 SAR data, and a subset of scene-level diagnostic inputs pairs optical RGB imagery with post-disaster radar evidence to test complementary optical–radar reasoning.
3. A large, structured task suite. The benchmark contains 134,108 task-specific instruction records spanning 15 task types, organized into four families: instance-level assessment, scene-level counting, multi-instance reasoning, and structured report generation, with multiple input settings (single-view, multi-view, tight/context/global crops, RGB/SAR).
4. A unified evaluation of general and remote-sensing VLMs. The authors evaluate LLaVA-OneVision, Qwen2.5-VL-7B, Qwen3-VL-8B, Qwen3-VL-30B, and TEOChat under one protocol with LoRA instruction tuning, showing that tuning helps visible damage and temporal-change reasoning but does not close the building-centric gap.

Main Findings

• A persistent building-centric gap. Models perform better on visible damage cues than on building-function understanding, multi-instance reasoning, scene-level counting, and grounded reporting. After instruction tuning, temporal-change detection and damage-level estimation reach relatively high scores, while building-function macro-F1 remains limited.
• Instruction tuning helps unevenly. Qwen3-VL-8B + LoRA achieves the best building-function macro-F1 (37.0), function-selection Set-F1 (82.8), spatial-reasoning accuracy (70.7), and report-generation scores (BLEU-4 19.5, ROUGE-L F1 41.1, BERTScore F1 90.9). TEOChat + LoRA obtains the best damage MAE (0.14) and temporal-change scores (accuracy 90.6, macro-F1 53.4).
• Counting improves sharply with tuning. Total-count MAE drops from 175.6 for untuned Qwen3-VL-8B and 248.8 for untuned TEOChat to 25.2 and 36.4 respectively after LoRA tuning; key-wise function-count MAE falls to 2.7 for Qwen3-VL-8B + LoRA and 7.3 for TEOChat + LoRA.
• Untuned models are weak at counting and reporting. Untuned models show high counting errors and low report scores — for example, TEOChat reports 4.7 BLEU-4 and 21.5 ROUGE-L F1, and Qwen2.5-VL-7B reports 1.8 BLEU-4 and 16.3 ROUGE-L F1.
• Response-priority reasoning is among the weakest tasks. Priority accuracy stays low across models, ranging from 14.3 (TEOChat) to 38.1 (Qwen3-VL-8B + LoRA), showing difficulty in combining visible damage evidence with a building's functional role.
• Function labels are long-tailed. Residential buildings dominate the instruction pool, while response-critical classes such as medical, government/emergency, and recreational/leisure facilities have much lower support. Per-class F1 follows this imbalance: residential buildings are recognized well, while rare categories remain brittle and are often confused with more common classes.
• OSM matching is conservative. Approximately 48–55% of xBD buildings receive an accepted OSM match across splits, but only 12–15% remain after semantic filtering, corresponding to 14.5% of the training split and 14.4% of the test split. In raw counts, 79,865 of 148,175 training buildings obtain an accepted match with 21,422 retained; in the test split, 30,250 of 54,862 obtain an accepted match with 7,879 retained.
• Extra input formulations do not solve the problem. Auxiliary view diagnostics (tight, context, global, and single-view context variants) can help some damage-related predictions but do not consistently improve building-function recognition; SAR-aware scene prompts are described as a benchmark extension for future analysis rather than a demonstrated fix.

Methodology in Plain English

The authors start from xBD, a pre/post-disaster satellite dataset that supplies building footprints and ordinal damage labels but no information about what buildings are used for. To add that missing layer, they query the Overpass API for OpenStreetMap building elements near each scene's footprints inside a padded bounding box, then match each xBD footprint to OSM candidates geometrically. When polygons are available they use an asymmetric coverage score — the fraction of the xBD footprint covered by the OSM polygon — accepting the best candidate if coverage is at least 0.10; if no candidate meets that threshold, they accept the nearest candidate only when the haversine distance between geometry centers is at most 10 m. Matches are stored with confidence metadata (coverage ratio, center-point distance, number of candidates, acceptance mode, query status), and the authors note this is geometric metadata rather than a learned probability.

From accepted matches they extract semantic tags, prioritize function-bearing keys such as amenity, shop, and office over generic keys like building, discard generic or ambiguous tags (for example building=yes, unknown, road/path, structural-only, agricultural, and vague place labels), and map what remains into eight coarse categories: residential, commercial, educational, medical, government/emergency, industrial/utilities, religious/cultural, and recreational/leisure.

These enriched instances are converted into 15 instruction-following task types across four families. Instance-level tasks ask about a single marked building's function, damage level, temporal change, or both; scene-level tasks ask for total building counts and counts by damage level or function; report tasks ask for short or long structured reports; and multi-instance tasks ask a model to select, compare, rank, or prioritize among several marked buildings. Buildings can be shown at three spatial scales (tight, context, global) plus a single-view context variant, and scene-level diagnostics pair RGB with Sentinel-1 SAR.

For report references, the authors build compact structured scene summaries (disaster type, total footprints, damage-label availability, damage counts, affected and destroyed counts, known function counts, and coarse spatial damage patterns), with explicit separation between all detected buildings and those with known damage or function labels. Reference reports were generated offline using Llama 3.3-70B-Instruct after human-in-the-loop prompt refinement; at evaluation time the VLMs see only imagery, not the summaries.

Evaluation uses accuracy, macro-F1, ordinal MAE, exact match, Set-F1, MAE/RMSE with key-wise MAE for counts, and BLEU-4, ROUGE-L F1, and BERTScore F1 for reports. Multiple prompt templates are sampled per instance to reduce sensitivity to wording. A standardized balanced split contains 7,563 training, 809 validation, and 3,171 test conversations across the 15 task types, with test examples never used during tuning. LoRA instruction tuning was applied to Qwen3-VL-8B and TEOChat for one epoch with learning rate 2×10⁻⁵, dropout 0.05, cosine scheduling, and warmup ratio 0.03.

Why This Matters

• Research impact: The paper reframes disaster VLM evaluation from scene-level severity judgments to per-building, function-aware, grounded reasoning, and shows that this shift exposes weaknesses that standard benchmarks hide. It also supplies a diagnostic resource with a released instruction pool, task taxonomy, and OSM-enrichment pipeline that other researchers can build on.
• Real-world applications:
  • Triage and prioritization after earthquakes, floods, fires, tsunamis, or hurricanes, where responders must decide which damaged structures matter most.
  • Medical surge planning, since distinguishing a damaged hospital or clinic from a damaged residence changes how resources are allocated.
  • Shelter and search-and-rescue planning, where schools, shelters, and government/emergency buildings carry different operational weight.
  • Structured situation reporting for emergency coordination agencies, aligned with the INSARAG and FEMA assessment guidance the paper cites.
• Industry relevance: Satellite imagery providers, geospatial analytics companies, and humanitarian or civil-protection organizations building VLM-based damage-assessment tools can use the benchmark to test whether a model can ground claims in building-level evidence rather than producing fluent but unsupported summaries. The results suggest current off-the-shelf VLMs are not yet reliable for function-aware or priority-aware automated reporting.

Future Directions

• Long-tail-aware learning: The authors explicitly point to rare but operationally critical categories (medical, government/emergency, recreational/leisure) as brittle, motivating methods that handle OSM's long-tailed and geographically uneven label distribution.
• Stronger geospatial grounding: Improving how models connect visual evidence to semantic building information, since the paper argues part of the bottleneck lies in that connection during downstream reasoning.
• Multimodal adaptation and SAR use: The RGB/SAR scene setting is presented as a diagnostic extension; whether radar backscatter can be exploited effectively by VLMs remains an open question the paper does not resolve.
• Uncertainty-aware disaster VLMs: Reference generation explicitly separates known from unknown damage and function labels, and the paper calls for models that handle unavailable information faithfully rather than treating missing labels as no damage.

Target Audience

Researchers and practitioners working on vision–language models, remote sensing, or multimodal disaster assessment will benefit most, particularly those designing benchmarks or evaluating instruction-tuned models on satellite imagery. It is also relevant to geospatial and humanitarian-response teams considering VLM-based damage triage or automated reporting, and to students with intermediate familiarity with computer vision who want a concrete example of how benchmark design shapes what we learn about model capabilities.

Authors’ abstract

Vision--language models (VLMs) show promise for disaster-response remote sensing, but existing benchmarks mainly emphasize scene-level or damage-centric assessment. To study this building-centric gap, we introduce \method{}, a diagnostic benchmark built on xBD, a pre/post-disaster satellite dataset with building-level damage labels. \method{} enriches building instances with OpenStreetMap-derived functional labels and contains 134{,}108 task-specific instruction records across 15 task types, spanning instance-level assessment, scene-level counting, multi-instance reasoning, and structured report generation. The benchmark supports RGB pre/post-disaster imagery, single- and multi-view instance formulations, and scene-level RGB/SAR diagnostic inputs. Experiments with general-domain and remote-sensing VLMs show that models perform better on visible damage cues than on building-function understanding, multi-instance reasoning, counting, and grounded reporting. Instruction tuning improves performance on several tasks but does not close this building-centric gap.