RoadSceneVQA: Benchmarking Visual Question Answering in Roadside Perception Systems for Intelligent Transportation System

AI summary

Overview

• Research area: Computer Vision / Multi-modal Large Language Models (MLLMs), specifically Visual Question Answering (VQA) for roadside traffic perception in Intelligent Transportation Systems (ITS). arXiv:2511.18286v2 [cs.CV].
• Technical level: Advanced. The paper combines benchmark construction (annotation pipelines, dataset statistics) with MLLM architecture design (a kernel-based linear cross-attention fusion module), chain-of-thought distillation, and multi-task loss formulation.
• One-sentence scope: The paper introduces a 34,736-sample roadside-scenario VQA dataset (RoadSceneVQA), an agile Collaborative Human–Machine Annotation system (CH-MA), and a baseline MLLM, RoadMind, built on CogniAnchor Fusion (CAF) and Assisted Decoupled Chain-of-Thought (AD-CoT).

What This Paper Is About

Existing roadside perception systems mainly perform instance-level tasks such as detection, tracking, trajectory prediction, and traffic-flow forecasting, which produce fixed outputs (for example bounding boxes) and do not support natural-language interaction or reasoning about whether traffic participants are behaving legally or how they interact in context. The paper builds a large-scale roadside VQA benchmark whose questions require both explicit recognition and implicit commonsense reasoning about traffic rules, and it proposes a baseline multi-modal model designed so that lightweight, edge-deployable MLLMs can answer those questions more accurately and efficiently.

Key Contributions

1. RoadSceneVQA: A large-scale VQA dataset for roadside perception containing 34,736 question–answer pairs, spanning perception and reasoning tasks across varying weather, illumination, and traffic conditions, with questions covering object attributes as well as the intent, legality, and interaction patterns of traffic participants.
2. CH-MA (Collaborative Human–Machine Annotation): An agile human–machine collaborative annotation framework for VQA datasets, intended to improve annotation efficiency while enhancing accuracy and quality.
3. CogniAnchor Fusion (CAF): A vision–language fusion module inspired by human scene cognition (language-driven pre-anchoring of candidate regions) that is meant to improve reasoning while keeping computation efficient.
4. RoadMind: A multi-modal large language model powered by Assisted Decoupled Chain-of-Thought (AD-CoT) for roadside traffic perception and reasoning, evaluated at 0.9B, 2B, and 8B scales.

Main Findings

• State-of-the-art on the new benchmark: In Table 2, RoadMind-8B (Qwen 2.5) reaches the best reported GPT-Score (0.554), ROUGE-L (0.425), METEOR (0.411), SPICE (0.221), Exact Match (0.157), and CIDEr (1.836) among the compared models; RoadMind-0.9B records the highest BLEU-4 among RoadMind variants (0.179) and RoadMind-2B records exact match 0.142, ROUGE-L 0.405, METEOR 0.347, CIDEr 1.705, SPICE 0.219, and GPT-Score 0.489.
• Small models beat much larger ones: RoadMind (0.9B) achieves a GPT-Score of 0.440, exceeding the zero-shot MiniCPM-o 2.6 (8B, GPT-Score 0.428) and the fully fine-tuned MobileVLM v2 (1.7B, GPT-Score 0.417), which the authors attribute to CAF and AD-CoT.
• Perception is easier than reasoning: Table 3 shows RoadMind-8B performs substantially better on perception-type samples than reasoning-type ones. For example, the perception category's weather and traffic-light related samples reach METEOR 0.437 and 0.348 and GPT-Score 0.601 and 0.583, while the reasoning categories score METEOR 0.177–0.253 and GPT-Score 0.338–0.429. The authors note performance is relatively lower for understanding traffic participants and roadside infrastructure.
• Generalization to an ego-vehicle benchmark: On CODA-LM (20,495 train / 2,123 test samples), RoadMind-8B records a General Text-Score of 48.50 and Suggestion Text-Score of 54.28, surpassing InternVL 1.5-20B (GTS 38.38, STS 41.18). Regional perception numbers reported for RoadMind-8B include Vehicle 70.65, VRU 74.25, Sign 47.43, and 59.78 for the "All" entry.
• CAF transfers and is efficient: Table 6 shows CAF + Concat uses 0.924K parameters and 61.08M FLOPs with ROUGE-L 0.425, METEOR 0.411, SPICE 0.221, versus plain Concat (ROUGE-L 0.366, METEOR 0.397, SPICE 0.187) and versus CA + Concat (1.063M parameters, 495.41M FLOPs, ROUGE-L 0.418, METEOR 0.422, SPICE 0.217). The authors describe this as a precision-versus-computational-cost trade-off.
• Components improve a different backbone: Table 5 reports GPT-Score migration performance on MiniCPM-o 2.6: NSFT 0.428, LoRA 0.452, SFT 0.527, SFT + CAF 0.533, SFT + CAF + AD-CoT 0.549.
• AD-CoT beats alternatives: Table 7 shows the full AD-CoT setting scoring METEOR 0.420 / GPT-Score 0.568 on perception and METEOR 0.339 / GPT-Score 0.445 on reasoning, outperforming "Only OQ Input", "Only GT (train)", "Only CA (train)", and MCoT.
• Dataset positioning: Table 1 lists RoadSceneVQA at 34.7k QA items, 26 scenes, reasoning supported, roadside domain, generated by LLM + Manual annotation — the only roadside entry in the table marked as supporting reasoning (TUM-VideoQA is listed at 87.3k items, 1k scenes, and no reasoning).

Methodology in Plain English

Dataset construction. The questions and answers build on the Rope3D roadside detection dataset and are created through a three-stage CH-MA pipeline. In Stage A, a strong MLLM (QwenVL-Max) is given a tailored prompt and a roadside image and proposes four candidate QA pairs, each with an estimated reasoning contribution score; a human annotator reviews them and picks the best one as a reference. In Stage B, annotators revise and refine that pair to fix factual errors, contextual mismatches, and hallucinations while keeping a formal, neutral tone. In Stage C, a panel of seven annotators reviews all samples and only those with majority approval enter the dataset; rejected samples go back to Stage B.

Model design. RoadMind encodes an image into visual patches plus a downsampled global view, processes them with an InternViT visual encoder, applies Pixel Shuffle, and aligns them through a lightweight MLP adapter. The question is augmented with a Chain-of-Thought prompt generated by GPT-4o. The two streams are then combined by CogniAnchor Fusion, which uses text as the query to selectively attend to visual features ("anchoring" candidate regions), with a linear-attention formulation (inspired by InLine Attention) so that the computation grows linearly rather than quadratically with the number of tokens. The fused, text-conditioned visual tokens are concatenated with the language embeddings and fed to Qwen 2.5, which produces the answer.

Training strategy. AD-CoT uses GPT-4o as a "soft" teacher: it generates an assisted reasoning context (perceive-and-reason process plus a conclusive answer). The reasoning process is concatenated with the original question as enriched input, while a multi-task loss combines a hard loss on the human ground-truth answer with a KL-divergence term matching the model's distribution to GPT-4o's, with learnable uncertainty weights for each term.

Experimental setup. RoadSceneVQA is split into 30,058 training and 4,677 test samples; CODA-LM into 20,495 training and 2,123 test samples. Images are resized to 448 × 448 px; models are fine-tuned for one epoch at an initial learning rate of 1e-5 with the visual encoder frozen and the LLM and MLP projector unfrozen, max sequence length 16384, AdamW with weight decay 0.05, cosine scheduler, warm-up ratio 0.03, on 4 A100 GPUs with batch size 1 per GPU. The attention head number for CAF is set to 8. Metrics are Exact Match, ROUGE-L, BLEU-4, METEOR, CIDEr, SPICE, and GPT-Score. Compared models include QwenVL, InternVL, MiniCPM series, MobileVLM v2, TinyLLaVA, and EM-VLM4AD.

Why This Matters

Impact on research. The paper argues that existing traffic VQA benchmarks emphasize explicit localization or captioning and therefore cannot test whether a model understands implicit traffic rules (for example, whether a pedestrian is violating rules given the current signal state and crosswalk layout). RoadSceneVQA reframes roadside evaluation around regulation-aware reasoning, and the dataset plus code are released at https://github.com/GuanRunwei/RS-VQA.

Real-world applications.

• Automated traffic-rule compliance monitoring at intersections, where a system reasons about signal state, crosswalk location, and participant behavior together.
• Natural-language query interfaces for traffic operators, letting them ask questions about a scene instead of manually inspecting bounding-box outputs.
• Edge-deployed roadside perception units, since the paper targets lightweight MLLMs smaller than 8B and reports efficiency gains from CAF.
• Traffic behavior analysis and interaction modeling feeding into traffic management, congestion control, and planning systems.

Industry relevance. Road

Authors’ abstract

Current roadside perception systems mainly focus on instance-level perception, which fall short in enabling interaction via natural language and reasoning about traffic behaviors in context. To bridge this gap, we introduce RoadSceneVQA, a large-scale and richly annotated visual question answering (VQA) dataset specifically tailored for roadside scenarios. The dataset comprises 34,736 diverse QA pairs collected under varying weather, illumination, and traffic conditions, targeting not only object attributes but also the intent, legality, and interaction patterns of traffic participants. RoadSceneVQA challenges models to perform both explicit recognition and implicit commonsense reasoning, grounded in real-world traffic rules and contextual dependencies. To fully exploit the reasoning potential of Multi-modal Large Language Models (MLLMs), we further propose CogniAnchor Fusion (CAF), a vision-language fusion module inspired by human-like scene anchoring mechanisms. Moreover, we propose the Assisted Decoupled Chain-of-Thought (AD-CoT) to enhance the reasoned thinking via CoT prompting and multi-task learning. Based on the above, we propose the baseline model RoadMind. Experiments on RoadSceneVQA and CODA-LM benchmark show that the pipeline consistently improves both reasoning accuracy and computational efficiency, allowing the MLLM to achieve state-of-the-art performance in structural traffic perception and reasoning tasks.