Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysis

AI summary

Overview

Research area: Computer vision, specifically video-language models applied to traffic safety (fine-grained video captioning and visual question answering).

Technical level: Intermediate. The paper builds on existing video large language models and LoRA fine-tuning; readers benefit from familiarity with captioning metrics and VQA, but the framework itself is conceptually simple.

Scope: A dual-model system that pairs VideoLLaMA3-7B for captioning with Qwen2.5-VL-7B for visual question answering, trained separately, and evaluated on the WTS dataset in the 2025 AI City Challenge Track 2.

What This Paper Is About

Traffic safety analysis needs video systems that can both describe what pedestrians and vehicles are doing in fine detail and answer specific safety questions about a scene. Single models tend to be good at one of these tasks and weak at the other, and training one model on both tasks at once causes interference. This paper argues that splitting the work across two specialized models — each trained only on its own task — produces better overall results than any single baseline.

Key Contributions

1. A framework that systematically integrates VideoLLaMA's temporal reasoning with Qwen-VL's visual reasoning for traffic safety analysis, rather than trying to make one model do both.
2. An empirical validation that separate task-specific training for captioning and VQA significantly outperforms joint training on both tasks.
3. Extensive experiments on the WTS dataset across multiple metrics, resulting in a 10th-place finish in the AI City Challenge 2025 Track 2.
4. An ablation study isolating the effect of captioning-only, combined, and separate training regimes on captioning quality and VQA accuracy.

Main Findings

• Complementary model strengths: VideoLLaMA3-7B produced the strongest captioning scores on the internal split (BLEU-4: 0.2569, METEOR: 0.4528, ROUGE-L: 0.4512, CIDEr: 1.1001), while Qwen2.5-VL-7B produced the strongest VQA accuracy at 60.80% (reported as 60.7980).
• Combined system result: The VideoLLaMA3-7B and Qwen2.5-VL-7B pairing reached an S2 score of 45.7572. The paper reports this as a 1.09-point improvement over the strongest baseline; Table 1 lists the best single model (VideoLLaMA3-7B) at an S2 of 44.7329.
• Leaderboard placement: The approach ranked 10th in Track 2 of the 2025 AI City Challenge with a score of 45.7572, behind ninth place (Tyche, 52.1481) and ahead of no other listed team; the top entry was CHTTLIOT at 60.0393.
• Baseline comparison: LLaVA-NeXT-Video-7B scored an S2 of 40.3807, InternVL3-8B 41.9799, Qwen2.5-VL-7B 42.5136, Video-LLaVA-7B 35.0327, and MiniCPM-V-2_6 40.8556.
• Trade-off in single models: LLaVA-NeXT-Video performed comparatively well on captioning (CIDEr 1.0122) but reached only 49.32% VQA accuracy, while InternVL3-8B showed strong VQA accuracy (59.93%) but weaker captioning metrics (CIDEr 0.5753).
• Separate training beats joint training: Ablation results show separate training reaching VQA accuracy of 58.6121 (VideoLLaMA3-7B) and 60.7980 (Qwen2.5-VL-7B), compared with 53.50 and 57.50 under combined training, and 47.50 and 49.00 under captioning-only training. The abstract states the separate strategy outperforms joint training by 8.6% in VQA accuracy while maintaining captioning quality.
• Captioning is preserved under separate training: CIDEr reached 1.1001 for VideoLLaMA3-7B and 0.5769 for Qwen2.5-VL-7B with separate training, versus 1.0350 and 0.5450 under combined training, and 1.0625 and 0.5500 under captioning-only training.
• External split results are higher: On the external split, VideoLLaMA3-7B scored BLEU-4 0.2814, METEOR 0.4844, ROUGE-L 0.4658, and CIDEr 1.2579; Qwen2.5-VL-7B scored BLEU-4 0.1475, METEOR 0.3483, ROUGE-L 0.3414, and CIDEr 0.6304.
• Qualitative hallucination difference: In one safety-critical example where a pedestrian uses a smartphone while standing still, VideoLLaMA3-7B captured details such as a black T-shirt and stationary behavior, while Qwen2.5-VL misidentified the clothing as a "brown jacket and green slacks" and the behavior as "rushing out at high speed."

Methodology in Plain English

The researchers take two existing video-language models and give each one job. VideoLLaMA3-7B is trained only to write detailed captions describing pedestrian and vehicle behavior. Qwen2.5-VL-7B is trained only to answer multiple-choice safety questions about the video. Each model gets its own set of LoRA adapters — small trainable weight updates added to the frozen base model — so the captioning adapters and the VQA adapters never share parameters and never compete with each other. At inference time, the caption model handles description and the question-answering model handles reasoning.

The input is a video split into behavioral phases, and the outputs are captions from both a pedestrian viewpoint and a vehicle viewpoint, plus answers to multiple-choice questions about the scene.

Training details vary by model. Qwen2.5-VL was trained with a batch size of 4, gradient accumulation over 8 steps, 10 epochs, learning rate 1×10⁻⁴, and a cosine scheduler with a 0.1 warm-up ratio. VideoLLaMA3 used learning rates of 1×10⁻⁵ for the LLM and projector and 2×10⁻⁶ for the vision encoder, weight decay 0.01, gradient accumulation of 4 steps, gradient checkpointing, a cosine scheduler with 0.03 warm-up ratio, and logging every 5 steps. The evaluation combines BLEU-4, METEOR, ROUGE-L, and CIDEr into a caption score, averages that with VQA top-1 accuracy, and reports the result as the S2 metric.

Why This Matters

Impact on research: The paper tests the assumption that one multimodal model should handle everything. It shows that task interference is measurable — joint training lowered captioning quality while separately trained models kept both captioning and VQA performance high — and it offers a simple, reproducible recipe for combining heterogeneous video LLMs.

Real-world applications:

• Proactive pedestrian safety monitoring at intersections, where detailed behavioral descriptions can flag risky situations before a collision.
• Automated incident reporting for traffic agencies, converting multi-camera footage into structured narratives instead of manual post-incident reconstruction.
• Analysis of dashcam and ego-vehicle footage for driver-assistance and autonomous driving development, including reasoning about pedestrian intent.
• Insurance and legal review workflows that need both a narrative description and answers to specific factual questions about an event.

Industry relevance: The system targets the kind of safety-critical video understanding that intelligent transportation systems, fleet operators, and autonomous vehicle developers need. The paper also acknowledges that model synchronization overhead in multi-camera deployments limits real-time application — a practical constraint for deployment teams — and that single models still hallucinate in safety-critical scenes.

Future Directions

• Model distillation: The authors explicitly state that future work will explore distillation to maintain performance while reducing computational requirements for real-world deployment.
• Hallucination mitigation: The qualitative example shows Qwen2.5-VL fabricating clothing and behavior details; the paper identifies hallucination mitigation as a persistent open challenge in this domain.
• Closing the performance gap: The 10th-place score of 45.7572 versus the top score of 60.0393 indicates substantial room for improvement, though the paper does not detail the methods used by higher-ranked teams.
• Real-time and multi-camera deployment: The paper notes synchronization overhead in multi-camera settings and the difficulty of achieving real-time performance on edge hardware, leaving the question of deployment-ready efficiency open.

Target Audience

Researchers and graduate students working on video-language models, video captioning, and visual question answering; practitioners in intelligent transportation systems and traffic safety analytics; participants in the AI City Challenge and similar benchmarks; and industry engineers in autonomous driving, fleet safety, or urban mobility who need models that both describe and reason about traffic scenes.

Authors’ abstract

Traffic safety analysis requires complex video understanding to capture fine-grained behavioral patterns and generate comprehensive descriptions for accident prevention. In this work, we present a unique dual-model framework that strategically utilizes the complementary strengths of VideoLLaMA and Qwen2.5-VL through task-specific optimization to address this issue. The core insight behind our approach is that separating training for captioning and visual question answering (VQA) tasks minimizes task interference and allows each model to specialize more effectively. Experimental results demonstrate that VideoLLaMA is particularly effective in temporal reasoning, achieving a CIDEr score of 1.1001, while Qwen2.5-VL excels in visual understanding with a VQA accuracy of 60.80\%. Through extensive experiments on the WTS dataset, our method achieves an S2 score of 45.7572 in the 2025 AI City Challenge Track 2, placing 10th on the challenge leaderboard. Ablation studies validate that our separate training strategy outperforms joint training by 8.6\% in VQA accuracy while maintaining captioning quality.