StateBridge: Phase- and Track-Grounded Reasoning for Synthetic-to-Real Traffic Video Captioning and VQA

AI summary

Overview

• Research area: Computer vision and vision-language modeling for traffic safety, specifically video captioning and visual question answering (VQA) under a synthetic-to-real transfer protocol.
• Technical level: Advanced. The work assumes familiarity with vision-language models (VLMs), LoRA fine-tuning, object track summaries, and captioning metrics such as BLEU-4, METEOR, ROUGE-L and CIDEr.
• Scope: A single paper describing StateBridge, a synthetic-only framework submitted to AI City Challenge 2026 Track 2, which combines phase-aware clip construction, track-grounded prompting, fact-locked caption fusion, and evidence-diverse VQA consensus.

What This Paper Is About

The paper tackles a scenario in which a model may only learn from synthetic "Digital Twin" traffic videos (SynWTS) but is scored on real-world WTS footage, with no real captions or answers available for training. The goal is to produce phase-level pedestrian and vehicle captions (covering location, attention, behavior and context) plus multiple-choice VQA answers (covering attributes, environment, geometry, visibility, action and vehicle state) that stay mutually consistent. StateBridge's core idea is that geometry, phase metadata, actor roles and annotation schema transfer across domains far more reliably than appearance, pose, gaze, gait or lighting, so the system should carry structured evidence rather than simulator looks.

Key Contributions

1. Reframes the task as asymmetric structured-state transfer: the authors formalize Track 2 as a setting where the video distribution changes between source and target, but the annotation schema over phase, role, geometry, tracks and checklist attributes stays the same (Equation 2).
2. Phase- and track-grounded evidence interface: multi-view clips are cut to phase intervals, and pedestrian and vehicle trajectories are summarized as first, middle, last and mean boxes rescaled to a shared 0–1000 coordinate grid, so motion cues are resolution-independent.
3. Fact-locked caption fusion: a conservative role-phase selection rule that only accepts a candidate caption field when it improves a proxy score by a margin, keeps token overlap with a trusted field, and passes a binary consistency lock over supported facts.
4. Evidence-diverse VQA consensus plus a full pipeline: a weighted vote over synthetic-trained adapters that see different evidence constructions (clips, text boxes, overlays, interaction crops, joint caption–VQA adaptation), together with a complete synthetic-only pipeline for dataset construction, model adaptation, inference and official submission.

Main Findings

• Official score: StateBridge achieves an official S2 score of 55.4679 on AI City Challenge 2026 Track 2 using only synthetic training data, and ranks 8th in the competition (team ID 127).
• Leaderboard context: the top three entries score 60.0853 (rank 1), 57.3307 (rank 2) and 56.7949 (rank 3); ranks 4 through 10 range from 55.5901 down to 55.3330, placing StateBridge in a tight cluster.
• Component ablation: the joint Qwen3-VL baseline scores 55.0643 S2 with 81.2424 VQA accuracy; adding VQA consensus alone raises S2 to 55.3330 and accuracy to 81.2930, and also lifts CIDEr from 0.6590 to 0.7502.
• Caption fusion ordering: progressively constrained fusion improves S2 across the ablation — rewrite model with fact locking 55.4235, fusion with completion repair 55.4574, fixed-budget fusion with 45 fields 55.4657, role-specific fusion 55.4659, and conservative fact-locked fusion 55.4679.
• Metric trade-off: fact-locked fusion improves METEOR (0.4248 vs 0.4199) and ROUGE-L (0.4446 vs 0.4331) over the direct caption source, but lowers CIDEr (0.7252 vs 0.7502) and BLEU-4 (0.2438 vs 0.2470); the direct caption source holds the best BLEU-4 and CIDEr of the reported fusion variants.
• VQA accuracy is stable across caption variants: all Table 1 fusion variants report 81.2930 accuracy, so the differences between them isolate caption behavior, not VQA.
• Qualitative behavior: in a real public-test sequence (scenario 20230922_7_SN1_T1) the model produces role-specific captions and correct phase-level VQA answers; in a separate example (scenario 20230707_11_SY3_T1) the fact-lock repair removes an incomplete dangling phrase while preserving supported geometry, visibility, speed, appearance and road context.
• Consensus corrects a dissenting view: on a validation question (scenario 20230707_18_CY23_T1, avoidance phase), three synthetic-trained predictors answer "c. Notices the vehicle" while the visual-grounding variant answers "b", and the consensus selects the majority-supported answer.

Methodology in Plain English

The team starts from the challenge's own annotations rather than from raw appearance. Each scenario is indexed into a manifest recording the split, scenario identifier, view, video path, phase interval, caption fields, VQA items and available tracks. Every video is clipped to the five behavioral phases — pre-recognition, recognition, judgment, action and avoidance — so training examples look like test examples and neighboring phases do not leak in.

For each actor in each phase, the pixel boxes are converted into relative coordinates on a 0–1000 grid and reduced to four summary boxes (first, middle, last, mean), alongside the actor role, view, phase and number of tracked frames. These compact summaries are placed in the prompt, giving the model a description of coarse displacement, scale change and relative geometry that does not depend on resolution.

Public multimodal instruction models are then fine-tuned with LoRA using LLaMA-Factory, with Qwen3-VL-8B-Instruct as the main backbone. Three task mixtures are trained (caption-only, VQA-only, joint), plus evidence variants that differ only in what the model sees: serialized text boxes, rendered track overlays, pedestrian–vehicle interaction crops, or a combination of raw clips and grounding views. Caption rows request a JSON object with caption_pedestrian and caption_vehicle; VQA rows request only the option letter.

At inference, captions are not rewritten wholesale. Candidate fields from several synthetic-trained models (Qwen3-VL adapters plus Qwen3.5-9B and InternVL3-8B adapters) are scored by token overlap with a trusted field, four-gram preservation, agreement with supported facts, length compatibility, and penalties for repetition, contradictions or incomplete endings. A candidate replaces a field only if it clears a score margin, a minimum overlap threshold and a consistency lock; in the reported implementation accepted fields must be between 0.84 and 1.22 times the trusted field length and score no worse than 0.05 below it. VQA instead uses a vote across adapters with equal weights (w=1), with the strongest joint Qwen3-VL adapter breaking ties.

Why This Matters

The paper argues that the hard part of synthetic-to-real traffic understanding is not fluent text but knowing which generated field to trust when the visual evidence degrades. By shifting transfer onto phase, role, geometry and checklist semantics, it offers a template for deploying video-language models in domains where labeled real data is scarce or expensive.

Real-world applications:

• Intersection and crosswalk safety auditing, where cameras already record pedestrian–vehicle interactions and analysts need per-phase descriptions of who was where, whether they were attentive, and what the vehicle did.
• Automated incident and near-miss reporting, since the phase structure (including avoidance) maps directly onto how safety analysts document evolutions.
• Simulation-to-deployment pipelines for autonomous driving, where models can be trained in a Digital Twin and evaluated on recorded road footage without collecting new labels.
• Natural-language interfaces for traffic operations, where an operator asks multiple-choice or descriptive questions about archived footage instead of scrubbing video.

Industry relevance: the work targets an official challenge benchmark with a hidden test set, uses off-the-shelf open backbones, and reports that its most conservative fusion strategy wins — a result directly useful to teams that must ship predictions without any target-domain feedback.

Future Directions

• Improving synthetic fidelity for hard cues: the paper identifies pose, gaze, gait, contact dynamics, illumination, clothing detail and small objects as the attributes simulation reproduces poorly; whether better rendering or targeted augmentation closes this gap is untested here.
• Tuning the gate without target labels: the required score margin and minimum overlap threshold are written as δ and τ and are not reported numerically, so how sensitive the fact lock is to these settings — and how to select them without any target feedback — remains open.
• Weaker reliance on challenge-provided annotations: the evidence interface uses the boxes and phase annotations supplied by the challenge; replacing them with the system's own detector and tracker under the same synthetic-only protocol is a natural next step.
• Broadening beyond one benchmark: the evaluation covers a single track with one hidden test set, so generalization to other traffic datasets, camera geometries and sensor configurations is not established by the reported results.

Target Audience

Researchers and engineers working on vision-language models for transportation and safety-critical video, particularly those entering AI City Challenge-style competitions or building systems where only synthetic or simulated supervision is available. It also suits practitioners interested in constrained, fact-preserving decoding and multi-model consensus as alternatives to full caption rewriting, and readers who want a concrete example of designing a transfer interface around what actually survives a domain shift.

Authors’ abstract

Traffic safety videos provide rich evidence for understanding interactions between pedestrians and vehicles, but synthetic-to-real evaluation makes this setting challenging because models must learn from simulated scenes and generalize to real-world traffic. In this work, we present StateBridge, a traffic video-language framework designed to operate under this synthetic-only training protocol. Our approach uses phase-aware clip construction, pedestrian and vehicle track summaries, and structured prompts to ground vision-language models in stable traffic evidence rather than appearance alone. We fine-tune public vision-language models on synthetic captioning and visual question answering examples, then combine their outputs through fact-locked caption fusion and evidence-diverse VQA consensus to improve consistency across actor roles, behavioral phases, and question types. StateBridge achieves an official S2 score of 55.4679 on AI City Challenge 2026 Track 2 using only synthetic training data. These results demonstrate the effectiveness of structured evidence transfer for traffic safety captioning and visual question answering.