Hierarchical Reasoning with Vision-Language Models for Incident Reports from Dashcam Videos

AI summary

Overview

Research area: Computer vision and autonomous driving, specifically vision-language model (VLM) pipelines for generating human-readable incident reports from dashcam video.

Technical level: Intermediate. The pipeline itself is conceptually straightforward, but it assumes familiarity with VLMs, captioning metrics (CIDEr-D, METEOR, SPICE), and challenge-style evaluations.

Scope: The paper describes a three-stage hierarchical VLM pipeline, an ensembling step, and a human A/B selection protocol that together earned 2nd place out of 29 teams in the 2COOOL challenge at ICCV 2025.

What This Paper Is About

Autonomous driving models are trained mostly on safe, normal driving logs, so they handle rare hazards and near-misses poorly. The 2COOOL challenge asks participants to go beyond simple hazard detection and automatically write incident reports from dashcam videos — narratives that explain what happened and why. This paper presents the Turing Inc. team's solution: a hierarchical reasoning framework built on vision-language models that breaks the report-writing problem into smaller, focused stages.

Key Contributions

1. A hierarchical reasoning framework with three stages — frame-level captioning, incident frame detection, and incident captioning — that decomposes incident report generation instead of feeding whole videos to a VLM.
2. Use of gaze heatmaps as an additional visual input, created by vertically concatenating each reference frame with its corresponding gaze heatmap, alongside metadata extraction about incident-related objects such as pedestrians and animals.
3. An ensembling strategy that collects multiple candidate reports generated under different inference settings and has an LLM rewrite them into a single coherent final report.
4. A Blind A/B Scoring protocol in which reports from different methods or settings are shown in random order with their origin concealed, evaluated by multiple annotators with majority vote on factual correctness, readability, and trustworthiness.

Main Findings

• Leaderboard placement: The method ranks 2nd among 29 teams on the official 2COOOL open leaderboard, with a final score of 0.1498 (average of SPICE 0.1822, METEOR 0.2605, and CIDEr-D 0.0067).
• Best CIDEr-D: The submission achieved the best CIDEr-D score on the open leaderboard (0.0067), compared with 0.0040 for the top-ranked team (NotSoDeep), 0.0046 for Awais, 0.0036 for Jane Doe, and 0.0028 for iAmAbIrD.
• Close margins at the top: Top teams were separated by very small differences in final score — 0.1518 (1st), 0.1498 (2nd), and 0.1497 (3rd).
• Candidate report comparison: Three candidate reports were evaluated. Candidate III scored highest on all three automatic metrics (SPICE 0.1822, METEOR 0.2605, CIDEr-D 0.0067; final score 0.1498) and was tied for rank 1 in A/B scoring alongside candidate II (SPICE 0.1739, METEOR 0.2547, CIDEr-D 0.0063; final 0.1449). Candidate I scored lowest (SPICE 0.1717, METEOR 0.2489, CIDEr-D 0.0054; final 0.1420) and ranked 2 in A/B.
• Metric–human agreement: SPICE, METEOR, and CIDEr-D were generally aligned with human judgments, though notable differences remained, which the authors say motivates combining quantitative metrics with human evaluation.
• Selection by human preference: The authors selected the report that received the highest rating in blind A/B scoring as their final submission.
• Qualitative success and failure: The method produced factually accurate and coherent reports across a wide range of scenarios, but occasionally made errors in relative spatial expressions such as confusing left and right — a known limitation of current VLMs in spatial reasoning.
• Compute profile: All experiments ran on 8 NVIDIA H100/H200 GPUs, with each video processed in only a few minutes across all stages.

Methodology in Plain English

Feeding an entire dashcam video into a VLM is expensive and often causes the model to miss the critical moment, so the authors split the job into three steps.

First, they sample the video every k frames (using k = 10 in this stage) and take the last frame of each segment as a reference frame. Each reference frame is captioned individually, and the model also extracts metadata about incident-related objects such as pedestrians and animals. To bring in driver attention, each reference frame is augmented by vertically concatenating the raw frame with its gaze heatmap. This keeps the number of visual tokens down while still capturing what matters.

Second, the captions and metadata from those reference frames are formatted and given to a large language model, which predicts the incident frame i. Because reference frames around an incident usually mention hazardous factors, this narrows down the likely time range efficiently.

Third, the predicted incident frame acts as an anchor, and frames within a start/end offset t around it are sampled according to the set F(i, k, t) = {i + mk | m ∈ Z, −t ≤ m ≤ t}. These frames go into a VLM to produce the incident report. For candidate generation, the authors varied the settings: in Stage 3 the frame interval was k ∈ {2, 6, 11, 12} and the offset was t ∈ {6, 8, 10}.

Models used: GLM-4.5V in Stage 1, GPT-OSS-120B in Stage 2, and GLM-4.5V plus Qwen3-VL-235B-A22B-Thinking in Stage 3. Candidate reports were then consolidated by Qwen3-Next-80B-A3B-Instruct into one final report, and the best candidate was chosen using Blind A/B Scoring rather than relying on automatic metrics alone.

Why This Matters

Impact on research: The work shows that structuring a hard video-understanding task into hierarchical VLM stages can beat a single end-to-end pass, and it pairs automatic metrics with human judgment. It also highlights an unresolved weakness — VLM spatial reasoning — that affects safety-critical applications.

Real-world applications:

• Automated insurance claims processing that turns dashcam footage into structured incident narratives.
• Fleet safety monitoring, where reports of near-misses and accidents can be generated at scale for driver coaching.
• Post-incident forensic analysis and traffic-safety research on rare, long-tail events.
• In-vehicle or cloud driver-assistance systems that summarize what happened for the driver or for regulators.

Industry relevance: Autonomous driving and fleet operators need systems that can explain out-of-distribution events, not just detect them. The paper's code is publicly released at https://github.com/riron1206/kaggle-2COOOL-2nd-Place-Solution, and the pipeline runs on commodity-scale GPU nodes (8 H100/H200 GPUs) with a few minutes per video, making it practically reproducible.

Future Directions

• Fixing spatial reasoning: The authors explicitly flag left/right confusion in relative spatial expressions as an open challenge for future research, citing prior work on VLM spatial limitations.
• Better metric–human alignment: Since SPICE, METEOR, and CIDEr-D only generally aligned with human judgment, work is needed on evaluation that predicts human preference more reliably.
• Reducing pipeline complexity: The three-stage design plus multi-model ensembling requires several large models; simplifying while preserving quality is a natural next step.
• Broader hazard coverage: The authors frame hierarchical VLM reasoning as promising for "broader understanding of safety-critical traffic events," suggesting extension beyond the 2COOOL setting.

Target Audience

Researchers and engineers working on autonomous driving, vision-language models, and video understanding — especially those interested in out-of-distribution hazard handling and human-interpretable model outputs. It is also useful for practitioners building applied video-to-text reporting systems, and for challenge participants looking for a concrete reference pipeline. Readers without background in VLMs or captioning metrics will need to consult the cited benchmarks and evaluation papers first.

Note on completeness: the paper content provided does not report the total dataset size, the number of clips or videos, or training/fine-tuning details, so those cannot be summarized here.

Authors’ abstract

Recent advances in end-to-end (E2E) autonomous driving have been enabled by training on diverse large-scale driving datasets, yet autonomous driving models still struggle in out-of-distribution (OOD) scenarios. The COOOL benchmark targets this gap by encouraging hazard understanding beyond closed taxonomies, and the 2COOOL challenge extends it to generating human-interpretable incident reports. We present a hierarchical reasoning framework for incident report generation from dashcam videos that integrates frame-level captioning, incident frame detection, and fine-grained reasoning within vision-language models (VLMs). We further improve factual accuracy and readability through model ensembling and a Blind A/B Scoring selection protocol. On the official 2COOOL open leaderboard, our method ranks 2nd among 29 teams and achieves the best CIDEr-D score, producing accurate and coherent incident narratives. These results indicate that hierarchical reasoning with VLMs is a promising direction for accident analysis and for broader understanding of safety-critical traffic events. The implementation and code are available at https://github.com/riron1206/kaggle-2COOOL-2nd-Place-Solution.