Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception

AI summary

Overview

Research area: Computer Vision / Vision-Language Models applied to automated driving perception, with a benchmark contribution (Visual Question Answering) at the intersection of autonomous driving safety and multimodal model evaluation.

Technical level: Intermediate. The paper assumes familiarity with VLM architecture (vision encoder, projector, LLM backbone), VQA benchmarks, and autonomous driving stacks, but the core question — can small models correctly answer simple visual questions about traffic scenes at distance — is accessible.

Scope in one sentence: The paper introduces DTPQA, a distance-annotated, perception-only VQA benchmark for traffic scenes, and uses it to measure how well state-of-the-art small VLMs (under 4 billion parameters) perceive objects at distances from 5 to 50 meters compared with human performance.

What This Paper Is About

Vision-Language Models are increasingly proposed as components of automated driving systems because of their reasoning and generalisation abilities, yet a model cannot be trusted in a safety-critical setting unless its perception is reliable. Traffic scenes frequently contain critical objects at a distance, so the authors argue a usable system must not be "shortsighted" and must perceive well at both close (up to 20 meters) and long (30+ meters) range. The paper builds a benchmark of trivial but driving-relevant visual questions with distance annotations, and asks whether small VLMs can answer them and how their accuracy changes as the object in question gets farther away.

Key Contributions

1. DTPQA benchmark. The authors introduce Distance-Annotated Traffic Perception Question Answering, described as the first VQA benchmark containing perception-only, trivial but driving-crucial questions for traffic scenes with distance annotations. It has two parts: DTP-Synthetic, built with the CARLA simulator, and DTP-Real, built on nuScenes.
2. Evaluation of state-of-the-art small VLMs. The nine publicly available models from the top 10 of the Open VLM Leaderboard (March 2025) in the under-4B parameter category were evaluated on DTPQA. BlueLM-V-3B, ranked first, was excluded because its weights are not publicly available.
3. Human ceiling measurement. Human performance was measured on the benchmark to establish the achievable ceiling, using 23 participants who each answered 62 samples.
4. Robustness to question phrasing. Three semantically equivalent variants of every question were created and all experiments repeated, to test whether small wording changes affect model performance on simple visual tasks.

Main Findings

• Small VLMs lag far behind humans. Despite the simplicity of the questions, the best-performing small VLM reached roughly 60% average accuracy versus roughly 85% for human participants. The authors note the human sample was relatively small, which imposes statistical limitations.
• Left-versus-right discrimination is a specific weak point. The paper identifies distinguishing left from right as a perception task that remains particularly challenging for these models.
• Distance-dependent degradation is the benchmark's central axis. DTPQA places the same question at 5, 10, 20, 30, 40 and 50 meters, plus negative samples where the object is absent, so degradation with distance can be measured directly. Per-model, per-distance and per-category accuracy numbers appear in Table 4 and Figure 4 of the paper, but those specific values are not included in the content provided here, so they are not reported in this summary.
• Question-phrasing experiments were run but their outcomes are not in the provided content. Three variants per question were tested — one replacing the subject with a synonym, one replacing the verb, and one rephrasing the whole question — with results in Section 6.2, which falls outside the truncated text.
• Deployment constraints motivate the small-model focus. The NVIDIA Jetson AGX Orin, a common self-driving hardware platform, provides 64 GB of unified memory shared between CPU and GPU in its most powerful AGX variant. InternVL3-78B would need approximately 156 GB of VRAM just to load its parameters in bfloat16 precision, which exceeds that capacity.
• Perception weaknesses matter even for known-unsafe scenarios. The authors give the SOTIF example of a pedestrian crossing at long range (30–50 meters): if the model fails to detect the pedestrian, no amount of reasoning ability helps the system decide to brake.

Methodology in Plain English

The researchers first checked whether an existing benchmark met three needs: questions that require perception only and no reasoning, questions tied to traffic scenes and driving decisions, and distance annotations for every sample. No benchmark satisfied all three, so they built one.

The synthetic half, DTP-Synthetic, uses the CARLA simulator to generate 8,288 unique images at 1920x1080 resolution and six questions, producing 9,368 image-question samples. Scenes are repeatable, so each scene is rendered multiple times with the object at different distances. The real half, DTP-Real, draws on nuScenes and contains 7,438 unique images at 1600×900, the native nuScenes resolution, with four questions and 9,781 samples, generated automatically from nuScenes annotations. Because real distances are not exact, samples are assigned to six distance bins (5m, 10m, 20m, 30m, 40m, 50m).

Each category keeps a balanced number of samples for every possible answer at every distance, so a model cannot gain an advantage from language bias alone. For example, Cat.4-Real at 30 meters has 396 samples split into three equal sets of 132 across the possible answers. The one stated exception is Cat.5-Synth at 10 meters, which has 12 samples for "Red" and 11 each for "Yellow" and "Green"; the authors judge this statistically insignificant.

Ten categories in total were chosen, six in DTP-Synthetic and four in DTP-Real, spanning coarse questions such as whether a pedestrian is present and fine-grained ones such as which blinker on the vehicle ahead is on. Category 2 (direction a pedestrian is walking) illustrates the driving relevance: whether a crossing pedestrian moves left or right can determine whether the driver yields or proceeds.

For evaluation, questions were given to models in multiple-choice format with the instruction to reply with a single word, using greedy decoding. Answers were scored as correct (exact match), unclear (ground truth contained in a longer response), or incorrect. Unclear cases were passed to Qwen2.5-72B-Instruct together with the ground truth for classification, and the authors manually verified a subset of these classifications.

Human evaluation used 23 participants answering 62 samples each — one sample per category per distance for both the synthetic and real benchmarks, chosen randomly — with answers in multiple-choice format and no further instructions. Statistical significance was tested with a Monte Carlo simulation: 1 million simulations per model per category, with differences treated as non-significant if a model matched or exceeded human accuracy on identically sampled subsets more than 5% of the time (one-sided p > 0.05).

Hardware used: a single NVIDIA GeForce RTX 4070 Ti SUPER with 16GB of VRAM for the small VLMs, except DeepSeek-VL2-Tiny, which is incompatible with FlashAttention-2 and required an NVIDIA A100-SXM4-40GB. InternVL3-78B was run on four NVIDIA H100 80GB HBM3 GPUs as a large-model comparison.

Why This Matters

Research impact. Existing VLM benchmarks usually mix perception, reasoning, and world knowledge, which makes it hard to tell which capability caused a failure. DTPQA isolates perception and adds a distance axis the authors state is absent from prior literature, giving a cleaner diagnostic for a capability that autonomous driving depends on. It also extends perception-focused evaluation, previously dominated by large models like GPT-4o, Gemini 1.5 Pro and InternVL2-76B, down to the sub-4B models that in-vehicle hardware can actually run.

Real-world applications.

• On-vehicle perception screening: checking whether a candidate model can detect pedestrians, people, vehicles and signs at 30–50 meters before it is trusted with driving decisions.
• High-level planning modules: VLMs proposed as natural-language planners depend on correctly reading the scene first, so distance-aware perception scores inform whether that use is safe.
• Benchmark-driven selection under hardware limits: the results help engineers choose among sub-4B models that fit platforms such as the Jetson AGX Orin.
• Prompt and interface design: the question-variant experiments probe whether a deployed VQA or driving assistant gives stable answers when a question is worded slightly differently.

Industry relevance. The work is co-funded by Valeo Vision Systems alongside Science Foundation Ireland (Grant 13/RC/2094_P2), the European Regional Development Fund, and Lero, the Science Foundation Ireland Research Centre for Software, indicating direct automotive-tier interest in knowing where small VLMs break down. The hardware comparison against a 78B model makes the practical argument explicit: large SOTA VLMs cannot be loaded on typical automotive compute, so the capability gap of small models is the gap that matters in production.

Future Directions

• Improving perception in small VLMs for traffic scenes. The authors state their hope is that the findings encourage further research into making small VLMs more suitable for automated driving.
• Robustness to prompt structure. The paper studies only modifications to the main question, not the

Authors’ abstract

Vision-Language Models (VLMs) are becoming increasingly powerful, demonstrating strong performance on a variety of tasks that require both visual and textual understanding. Their strong generalisation abilities make them a promising component for automated driving systems, which must handle unexpected corner cases. However, to be trusted in such safety-critical applications, a model must first possess a reliable perception system. Moreover, since critical objects and agents in traffic scenes are often at a distance, we require systems that are not "shortsighted", i.e., systems with strong perception capabilities at both close (up to 20 meters) and long (30+ meters) range. With this in mind, we introduce Distance-Annotated Traffic Perception Question Answering (DTPQA), the first Visual Question Answering (VQA) benchmark focused solely on perception-based questions in traffic scenes, enriched with distance annotations. By excluding questions that require reasoning, we ensure that model performance reflects perception capabilities alone. Since automated driving hardware has limited processing power and cannot support large VLMs, our study centers on smaller VLMs. More specifically, we evaluate several state-of-the-art (SOTA) small VLMs on DTPQA and show that, despite the simplicity of the questions, these models significantly underperform compared to humans (~60% average accuracy for the best-performing small VLM versus ~85% human performance). However, it is important to note that the human sample size was relatively small, which imposes statistical limitations. We also identify specific perception tasks, such as distinguishing left from right, that remain particularly challenging for these models.