DetectiumFire: A Comprehensive Multi-modal Dataset Bridging Vision and Language for Fire Understanding

AI summary

Overview

Research area: computer vision and multi-modal machine learning applied to fire detection, synthetic data generation, and vision-language reasoning.

Technical level: Intermediate.

Scope: This paper introduces DetectiumFire, a large-scale multi-modal fire dataset containing real and synthetic images, videos, bounding boxes, and text captions, and evaluates it on object detection, diffusion-based image generation, and vision-language fire reasoning.

What This Paper Is About

Fire-related computer vision datasets have been small, redundant, and limited to traditional tasks like classification and bounding-box detection, which makes them unsuitable for training or evaluating modern multi-modal models. The authors build a large, publicly released dataset that pairs fire imagery with rich text descriptions covering what is burning, the surrounding environment, and the severity of the fire. They then test whether this data improves detectors, helps diffusion models generate realistic fire images, and teaches a vision-language model to reason about fire scenes.

Key Contributions

1. A large-scale real-world fire dataset. DetectiumFire contains over 14.5k high-quality real-world fire images and 2.5k fire-related videos, with bounding box annotations and detailed captions. Together with 8k synthetic images, the overall collection exceeds 22.5k fire-related images.

2. A synthetic fire data pipeline with preference data. The authors fine-tune diffusion models (Stable Diffusion v1.5, Stable Diffusion 2, and Stable Diffusion XL-1.0) via Supervised Fine-Tuning and via Reinforcement Learning from Human Feedback using Diffusion-DPO, producing roughly 8k filtered synthetic fire images and a preference dataset of 12k labeled pairs.

3. A four-level fire severity taxonomy embedded in captions. Captions were generated with GPT-4o and then human-edited, limited to 75 tokens, and severity labels were manually corrected across No Risk, Low Risk, Moderate Risk, and High Risk.

4. Benchmarking across three task families. The paper evaluates object detection, diffusion-based generation quality (scored with GPT-4o and Elo ratings), and multi-modal fire reasoning with LLaMA-3.2-11B-Vision-Instruct.

Main Findings

• Duplication is substantially lower than in D-Fire. Using the imagededup tool, DetectiumFire had a PHash duplication ratio of 0.03 (205 of 7,549 images) versus 0.15 for D-Fire (883 of 5,822 images), and a CNN-based duplication ratio of 0.23 (1,756 of 7,549) versus 0.55 for D-Fire (3,201 of 5,822). The authors attribute this to sampling only 1 or 2 visually distinct frames per fire phase rather than extracting every frame.

• Models trained on DetectiumFire transfer better than the reverse. With YOLOv11m, training on DetectiumFire and testing on D-Fire gave 40.32 ± 1.53 mAP, 68.94 ± 1.43 mAP@50, and 63.72 ± 1.46 mAR, close to training and testing on D-Fire itself (40.28 ± 1.76 mAP, 71.66 ± 2.06 mAP@50, 65.12 ± 2.66 mAR). Training on D-Fire and testing on DetectiumFire dropped sharply to 24.88 ± 0.77 mAP, 48.04 ± 1.21 mAP@50, and 47.42 ± 1.62 mAR. In-domain DetectiumFire performance was 43.74 ± 0.64 mAP, 75.96 ± 0.62 mAP@50, and 69.80 ± 1.25 mAR.

• Baseline detectors on the DetectiumFire test set. On real-world training data, Faster R-CNN reached 41.13 ± 1.01 mAP, 80.13 ± 1.51 mAP@IoU50, and 51.60 ± 0.54 mAR@100; YOLO-World-M reached 41.28 ± 0.84 mAP, 76.50 ± 0.90 mAP@IoU50, and 67.32 ± 1.43 mAR@100. The authors highlight YOLO-World-M's promise for open-vocabulary fire detection.

• Synthetic data from DetectiumFire is usable, but slightly weaker than real data. With SFT-generated synthetic data, YOLOv11m scored 33.50 ± 0.35 mAP, 59.40 ± 0.75 mAP@IoU50, and 60.54 ± 2.26 mAR@100; with RLHF-generated data it scored 32.12 ± 0.29 mAP, 58.08 ± 0.14 mAP@IoU50, and 59.42 ± 2.35 mAR@100. The authors note RLHF data performed slightly worse than SFT data and hypothesize the RLHF pipeline reduced synthetic diversity.

• FLAME_SD synthetic data performed far worse. Models trained on the FLAME_SD set of 10k synthetic fire images scored only 2.10 ± 3.90 mAP, 6.71 ± 2.37 mAP@IoU50, and 9.38 ± 2.83 mAR@100 for YOLOv11m, and 3.42 ± 0.24 mAP, 7.11 ± 0.39 mAP@IoU50, and 10.56 ± 0.15 mAR@100 for Faster R-CNN.

• Combining real and synthetic data gives a small gain. YOLOv11m reached 44.52 ± 0.50 mAP, 76.26 ± 0.34 mAP@IoU50, and 69.92 ± 1.95 mAR@100; Faster R-CNN reached 42.56 ± 2.44 mAP, 81.41 ± 1.93 mAP@IoU50, and 52.18 ± 0.47 mAR@100, both above the real-data-only results.

• Fine-tuning on DetectiumFire greatly improves VLM fire reasoning. LLaMA-3.2-11B-Vision-Instruct improved from 62.73% to 87.37% on burning object identification, from 71.82% to 89.39% on environment recognition, and from 56.06% to 83.84% on fire severity classification.

• Controlled fires are deliberately included. Unlike prior datasets focused on destructive events, DetectiumFire includes candle flames (1,076 images), lighter flames (1,075), matches (163), controlled cooking fires (54), stove/fireplace fires (30), and controlled campfires (347), alongside uncontrolled categories such as kitchen fires (212), electrical fires (213), vehicle fires (750), forest/wildfires (1,045), and house/residential fires (1,159).

• Indoor and outdoor coverage is unbalanced by design category. The dataset contains 3,374 indoor fire images and 4,175 outdoor fire images. Smaller categories include ship fires (42), plane fires (14), trash bin fires (26), LPG/gas tank fires (159), burning debris (190), other outdoor fires (418), pure flame crops (25), other indoor accidental fires (551), and other indoor (unspecified counts for stove/fireplace listings as given).

• Curated hard negatives were included. The 7k non-fire images include deliberately chosen false-positive triggers such as fire-like clouds, red-hued sunsets, and strong light sources in dark scenes. Video data includes 1.7k fire clips plus 757 non-fire clips, with each clip at least 10 seconds long.

Methodology in Plain English

The authors gathered fire images, videos, and short clips from Google, Twitter, YouTube, and TikTok using text queries in English and other languages, including Chinese, and added images captured with IoT devices during controlled fire demonstrations and samples from existing benchmarks. They removed exact and near-duplicate images with the imagededup tool, then had human annotators review what remained and discard low-quality or irrelevant items. Fire safety professionals then drew bounding boxes in Roboflow and reviewed captions that had first been drafted by GPT-4o and constrained to 75 tokens focused on what is burning, the setting, and severity.

To test synthetic data, the team fine-tuned three Stable Diffusion variants with LoRA for 4,000 steps at a learning rate of 1e-4 using an AdamW8bit optimizer and fp16 mixed precision. For the human-feedback stage, they picked 4k prompts, generated between 2 and 9 images per prompt across the three fine-tuned models, had annotators choose preferred images, kept 12k label pairs after removing ties, and then fine-tuned Stable Diffusion v1.5 with Diffusion-DPO for 3,000 steps at 1e-8 learning rate with a DPO beta of 5,000. Human annotators filtered the resulting generated images down to roughly 8k and added boxes.

For detection, the researchers ran 5-fold cross-validation on YOLOv11m, holding out 10% as a fixed test set and splitting the rest into 70% training and 20% validation, training for 300 epochs with mini-batch 16. They also trained Faster R-CNN with a ResNet-101 FPN backbone (learning rate 0.0025, batch size 2, 300 epochs) and evaluated YOLO-World-M. All training used a single NVIDIA Tesla T4 GPU, and results are reported as mAP, mAP@50IoU, and mAR@100.

For generation quality, they wrote 770 unique evaluation prompts with GPT-4o, distinct from the dataset captions, generated images with the original Stable Diffusion v1.5 and the two fine-tuned variants, and had GPT-4o score pairwise comparisons along general preference, visual appeal, and prompt alignment, converting win rates into Elo scores. For reasoning, they fine-tuned LLaMA-3.2-11B-Vision-Instruct with LoRA and Fully Sharded Data Parallelism on 2× A100 GPUs at a learning rate of 1e-5 for 3 epochs, then used GPT-4o to judge whether outputs matched the human-verified ground-truth attributes.

Why This Matters

The paper argues that fire is a high-impact but under-resourced domain in AI for disaster safety, where existing benchmarks are small, redundant, and restricted to detection. By releasing a large, annotated, multi-modal dataset publicly, the work lowers the barrier for researchers who previously could not access high-quality fire data, which the authors note is often held privately by companies. It also demonstrates that synthetic data generated from this dataset can meaningfully augment scarce real data, and that vision-language models can learn fine-grained distinctions such as a candle flame versus a spreading curtain fire.

Real-world applications the authors point to:

• Automated alarm filtering that reduces false positives from controlled flames, sunsets, or fire-like clouds.
• Post-incident analysis using text descriptions of what burned and in what environment.
• Early-stage risk assessment that classifies fire severity to prioritize response.
• Intelligent safety and decision-support systems for disaster response and industrial monitoring.

Industry relevance: the dataset and results are directly relevant to fire safety, insurance, smart building monitoring, wildfire management, and any sector deploying camera-based alarm systems where false alarms are costly. The authors also suggest the methodology transfers to adjacent domains such as disaster response, industrial monitoring, and rare event modeling where data is scarce and stakes are high.

Future Directions

• Extending the dataset to additional languages beyond English and Chinese to further reduce geographic and cultural bias.
• Adding richer scene-level annotations to support fire progression modeling, human presence detection, and multi-step incident report generation.
• Improving synthetic data generation to increase diversity, since RLHF-generated images underperformed SFT-generated ones in downstream detection.
• Exploring controllable fire video generation, fine-grained fire assessment, and AI agent-based safety response systems built on the dataset.

Target Audience

Researchers and practitioners in computer vision, multi-modal learning, and safety-critical AI who work on fire detection, disaster response, or rare-event modeling, as well as engineers building real-world alarm and monitoring systems. It is also useful for anyone studying synthetic data augmentation or human-feedback fine-tuning of diffusion models, since the paper reports both pipelines on the same task.

Authors’ abstract

Recent advances in multi-modal models have demonstrated strong performance in tasks such as image generation and reasoning. However, applying these models to the fire domain remains challenging due to the lack of publicly available datasets with high-quality fire domain annotations. To address this gap, we introduce DetectiumFire, a large-scale, multi-modal dataset comprising of 22.5k high-resolution fire-related images and 2.5k real-world fire-related videos covering a wide range of fire types, environments, and risk levels. The data are annotated with both traditional computer vision labels (e.g., bounding boxes) and detailed textual prompts describing the scene, enabling applications such as synthetic data generation and fire risk reasoning. DetectiumFire offers clear advantages over existing benchmarks in scale, diversity, and data quality, significantly reducing redundancy and enhancing coverage of real-world scenarios. We validate the utility of DetectiumFire across multiple tasks, including object detection, diffusion-based image generation, and vision-language reasoning. Our results highlight the potential of this dataset to advance fire-related research and support the development of intelligent safety systems. We release DetectiumFire to promote broader exploration of fire understanding in the AI community. The dataset is available at https://kaggle.com/datasets/38b79c344bdfc55d1eed3d22fbaa9c31fad45e27edbbe9e3c529d6e5c4f93890