BD-HazardVLM: Probing Vision-Language Models for Latent Defensive-Driving Hazards in Bangladesh Road Scenes

AI summary

Overview

Research area: Computer vision and multimodal evaluation — specifically vision-language model (VLM) benchmarking for driving-scene safety reasoning, with a geographic focus on Bangladesh mixed-traffic roads.

Technical level: Intermediate. Readers should be comfortable with VLM terminology (zero-shot prompting, quantization, structured output schemas) and standard classification metrics (accuracy, macro-F1, Jaccard similarity), but the paper is primarily an empirical diagnostic study rather than a methods paper.

Scope (one sentence): The paper introduces BD-HazardVLM, a 500-image Bangladesh-centric benchmark with structured hazard annotations, and uses it to run a controlled zero-shot evaluation of three open VLMs (Qwen2.5-VL-7B, InternVL3-8B, MiniCPM-V-4.6) on fine-grained defensive-driving reasoning.

What This Paper Is About

VLMs are increasingly proposed for interpretable driving-scene understanding, but describing visible road users is not the same as reasoning defensively about them. A safe driver must decide which element is the primary hazard, whether the risk is immediate or still latent, how severe the consequence could be, and what preventive action to take — and existing benchmarks rarely test these judgments in weakly structured Global South traffic. The authors build a diagnostic benchmark from Bangladesh road imagery and use it to test whether coarse hazard recognition in current VLMs actually implies reliable fine-grained defensive reasoning.

Key Contributions

1. A new diagnostic benchmark. BD-HazardVLM contains 500 Bangladesh road-scene images — 375 from RSUD20K (75.0%) and 125 from TFP-BD (25.0%) — each assigned an identifier from BDHV_0001 to BDHV_0500 and annotated at the scene level rather than for detection or segmentation.

2. A structured defensive-driving taxonomy. Every image is labeled for hazard presence, primary hazard category (twelve mutually exclusive categories, including a control category with no major hazard), risk timing (immediate / latent / both / none), severity (low / medium / high), a set of defensive actions drawn from a nine-item vocabulary, visible evidence, plus scene-level hallucination trap and Western-road-bias trap flags.

3. A controlled three-model comparison. Qwen2.5-VL-7B, InternVL3-8B, and MiniCPM-V-4.6 all process the same 500 images under an identical semantic task definition and output schema, with per-image statistical testing (exact McNemar tests for category, timing, and severity; paired Wilcoxon signed-rank tests for action Jaccard; Holm correction within each task).

4. A diagnosis of specific failure modes. The paper isolates category mode collapse, near-absent latent-hazard anticipation, zero or near-zero high-severity recall, weak context-specific action selection, and near-zero self-reported trap recognition — failures that a single aggregate score or a binary hazard-detection number would hide.

Main Findings

• Binary hazard presence is uninformative. Model accuracy ranges from 88.0% to 89.2%, while a trivial majority-class classifier that always predicts hazard-present reaches 89.0%, essentially matching all three models, because 445 of the 500 images (89.0%) are hazard-positive.

• Fine-grained category recognition stays below 40%. Qwen2.5-VL-7B reaches 38.2% category accuracy, InternVL3-8B 35.0%, and MiniCPM-V-4.6 27.6%; macro-F1 is 9.9%, 9.4%, and 10.5% respectively. MiniCPM-V-4.6 has the lowest exact accuracy but the highest macro-F1, indicating broader though still weak coverage across the taxonomy.

• Qwen exhibits severe category mode collapse. It assigns the rickshaw/easybike hazard category to 426 of 500 images (85.2%), although that category appears in only 205 ground-truth images (41.0%), so locally salient vehicles dominate its reading of scenes whose primary hazard is pedestrian behavior, occlusion, road damage, or an uncontrolled intersection.

• Risk timing is the weakest capability. Accuracy is 12.4% for InternVL3-8B, 9.2% for Qwen2.5-VL-7B, and 1.6% for MiniCPM-V-4.6. Ground truth contains 404 latent (80.8%), 41 immediate (8.2%), and 55 no-risk (11.0%) cases, with no image labeled "both"; MiniCPM-V-4.6 instead predicts "both" for 283 images and "immediate" for 217, never predicting latent or none, and is correct on only eight images.

• The 1.6% timing score is a genuine collapse, not a parsing artifact. A raw-output audit confirmed every final risk_timing value matched its parsed value; in 215 initial responses MiniCPM-V-4.6 wrote the semantically equivalent phrase "both immediate and latent" rather than the permitted token, and the single formatting-only retry returned schema-valid output.

• Aggregate severity accuracy hides a safety-critical failure. InternVL3-8B has the highest aggregate severity accuracy at 61.2% but 0.0% recall on the 58 high-severity cases. High-severity recall is 5.2% for Qwen2.5-VL-7B and 22.4% for MiniCPM-V-4.6. Both InternVL3-8B and Qwen2.5-VL-7B predominantly collapse toward medium severity.

• Exact defensive-action matching is near zero. Exact set match is 1.6% for Qwen2.5-VL-7B, 0.4% for InternVL3-8B, and 1.0% for MiniCPM-V-4.6. Partial-overlap measures are more forgiving: any-overlap is 66.6%, 70.6%, and 73.2%, and mean Jaccard is 22.0%, 25.8%, and 29.9%. Models favor broadly safe actions such as slow_down and prepare_to_brake but struggle with context-specific ones such as monitor_occlusion, avoid_overtaking, keep_distance, or stop_or_yield.

• Trap self-reporting is close to absent. Human annotation marks 324 images (64.8%) as hallucination-prone and 285 (57.0%) as Western-road-bias-prone, yet unconditional positive-flag rates are 0.0%, 0.4%, and 0.6% for hallucination and 3.8%, 2.8%, and 0.2% for Western bias across Qwen2.5-VL-7B, InternVL3-8B, and MiniCPM-V-4.6. Trap recall is 0.0%, 0.3%, and 0.0% (hallucination) and 1.1%, 1.8%, and 0.0% (Western bias). For example, Qwen2.5-VL-7B emits 19 Western-bias flags but only three fall among the 285 ground-truth-positive images, and all three of MiniCPM-V-4.6's hallucination flags are false positives.

• No model wins everywhere. Qwen2.5-VL-7B leads on category accuracy, InternVL3-8B on timing and severity accuracy, and MiniCPM-V-4.6 on action Jaccard. The Qwen–InternVL category gap is not significant (p = 0.101), while both outperform MiniCPM-V-4.6 (p < 0.001); timing and severity differences are also reported as significant at the stated levels, and all action-Jaccard pairwise differences are significant (p < 0.001).

• Qualitative cases show incoherence across fields. The four examples in Figure 1 show false hazard activation on a control scene, latent risk converted into immediate or combined risk, high-severity underestimation driven by attention to a salient rickshaw rather than a vulnerable pedestrian, and a correct bus/truck-occlusion category paired with incorrect timing and severity.

Methodology in Plain English

The authors did not collect new photographs. They sampled existing Bangladesh driving imagery — 375 images from RSUD20K and 125 from TFP-BD — choosing scenes deliberately for diversity of hazard type rather than to represent how often each hazard occurs, so the benchmark is a diagnostic stress test, not a prevalence survey. An earlier 100-image pilot subset is included within the final 500.

Four undergraduates (three Computer Science and Engineering students and one Physics student) each labeled one disjoint 125-image subset using a shared guideline and taxonomy. Two team members — one CSE and one Physics undergraduate — reviewed all 500 annotations for label consistency, taxonomy compliance, evidence quality, and formatting, with ambiguous cases resolved by discussion, and the project lead making final calls when consensus failed. Because both reviewers were also annotators and each reviewed their own subset, and because images were not independently double-annotated, the authors report no formal inter-annotator agreement statistic and treat the adjudicated labels as the reference rather than a unique human-consensus ground truth.

Each model received the same zero-shot prompt containing the Bangladesh mixed-traffic context, the hazard taxonomy, the action vocabulary, and the output schema, with an explicit instruction to rely only on visible evidence and not to invent lane markings, traffic signals, zebra crossings, sidewalks, or stop signs. Responses had eight fields: hazard_present, primary_hazard_category, risk_timing, severity, defensive_actions, evidence, possible_hallucination, and possible_western_road_bias. Outputs were parsed into the predefined label space; the evidence field was kept for qualitative auditing and not scored.

Inference was greedy with sampling disabled. Hardware memory limits forced different configurations: Qwen2.5-VL-7B in 4-bit NF4 with double quantization and FP16 compute, a 200,704–401,408 pixel visual budget, and 180 max tokens; InternVL3-8B in 8-bit with 448×448 aspect-preserving tiling (up to six tiles plus a thumbnail) and 256 max tokens; MiniCPM-V-4.6 in FP16 with 4x downsampling, up to nine slices, and 190 max tokens. The authors disclose these differences and caution against over-reading small performance gaps.

Scoring uses accuracy against the majority baseline for hazard presence; exact-match accuracy and macro-F1 for category; exact-match accuracy for timing and severity; and exact set match, any-overlap, and mean Jaccard for actions treated as unordered sets. The trap fields are scored as positive-flag rate and recall against human trap labels, which measures whether a model admits unsupported reasoning is possible — not whether its full response actually hallucinated.

Why This Matters

Impact on research. The paper shows that a headline binary number can be almost entirely explained by class imbalance: the 89.0% majority baseline matches every evaluated model's 88.0%–89.2% hazard-presence accuracy. It argues for structured, decomposed labels — hazard identity, timing, severity, action — because InternVL3-8B's leading 61.2% severity accuracy coexists with 0.0% recall on the 58 high-severity cases, a contradiction invisible in the aggregate. It also adds a geographically specific evaluation point to a literature dominated by nuScenes, CARLA, and BDD-X, complementing datasets like IDD, IDD-3D, RSUD20K, and TFP-BD whose annotations target detection, tracking, or traffic flow rather than defensive reasoning.

Real-world applications:

• ADAS and driver-assistance warning design in dense mixed traffic where the relevant cue is occlusion, an unpredictable CNG stop, or a roadside crowd rather than a formal signal.
• Safety auditing of VLM-based driving assistants before deployment in South Asian cities, using field-level diagnostics instead of a single pass/fail score.
• Benchmark and dataset curation for geography-diverse multimodal evaluation, including explicit hallucination and Western-infrastructure-bias susceptibility labels.
• Driver-training and simulation content, where latent-hazard anticipation is the pedagogical target and single-frame labels can supply scenario metadata.

Industry relevance. Teams building or procuring driving VLMs can use the result that coarse recognition does not transfer to fine-grained defensive judgment as a procurement and regression-testing criterion. The benchmark's release plan — manifest, identifiers, annotations, prompts, predictions, and evaluation scripts, with annotations under CC BY-NC 4.0 and code under MIT and no image redistribution — is designed for reproducible third-party auditing without violating source-dataset licenses.

Future Directions

• Independent duplicate annotation. The authors explicitly plan to add duplicate annotation to establish a human-agreement ceiling for subjective fields such as risk timing, severity, and action sets, which the current single-annotation-plus-review protocol cannot provide.

• Video and motion-based evaluation. Single frames cannot capture ego speed, acceleration, trajectories, or time-to-collision, so risk timing and severity are image-level judgments; extending to video would let timing labels reflect actual dynamic risk.

• Broader baselines and subgroup analysis. The paper limits its claims to three open models under disclosed inference configurations — noting that quantization and preprocessing differ across them — and calls for larger and proprietary VLMs plus subgroup analysis; the released data could also support separate evidence-based human adjudication of raw responses to measure actual hallucination rates rather than self-reported trap flags.

• Taxonomy refinement for multi-hazard scenes. Assigning one primary category simplifies scenes with several interacting risks, and the intentional class imbalance (445 hazard-positive, 404 latent, 0 "both") leaves open how to evaluate balanced or multi-label formulations.

Target Audience

Most useful to researchers and engineers working on multimodal model evaluation and driving safety — particularly those building benchmarks for autonomous driving, VLM trustworthiness, or hallucination and geographic bias. It also serves practitioners evaluating VLMs for deployment in South Asian or other under-represented traffic environments, dataset curators extending detection-oriented corpora like RSUD20K and TFP-BD into reasoning tasks, and policy or safety reviewers who need a concrete demonstration of why aggregate accuracy alone is insufficient evidence of defensive-driving competence. Readers new to VLMs will find the taxonomy and error analysis accessible; readers seeking a new model architecture will not, since the contribution is diagnostic.

Authors’ abstract

Vision-language models are increasingly used for driving-scene understanding, yet their defensive reasoning under weakly structured Global South traffic remains unclear. We introduce BD-HazardVLM, a diagnostic benchmark of 500 Bangladesh road-scene images, comprising 375 from RSUD20K and 125 from TFP-BD, annotated for hazard presence, primary hazard category, risk timing, severity, defensive actions, visible evidence, and hallucination- and Western-road-bias traps. We evaluate Qwen2.5-VL-7B, InternVL3-8B, and MiniCPM-V-4.6 using a shared zero-shot task definition and structured output schema. Binary hazard-presence accuracy ranges from 88.0 percent to 89.2 percent, essentially matching the 89.0 percent majority-class baseline. Fine-grained reasoning is substantially weaker: category accuracy remains below 40 percent, risk-timing accuracy ranges from 1.6 percent to 12.4 percent, and exact defensive-action matching remains below 2 percent. Although InternVL3-8B achieves the highest severity accuracy at 61.2 percent, it has zero recall on the 58 high-severity cases. Qwen exhibits severe category mode collapse, while all models show weak latent-hazard anticipation and near-zero self-reported trap recognition. For the evaluated models, these findings show that coarse hazard recognition does not imply reliable fine-grained defensive-driving reasoning and motivate geographically diverse safety evaluation.