From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models

AI summary

Overview

Research area: Computer Vision / Vision-Language Models for autonomous driving, with a focus on video temporal understanding and agentic AI benchmarking.

Technical level: Intermediate. The paper is readable without deep model-training knowledge, but assumes familiarity with VLMs, benchmarks, and basic driving-perception concepts (ego vehicle, trajectory, temporal localization).

Scope: The paper introduces TAD, a question-answering benchmark for temporal understanding in autonomous driving video, evaluates 9 VLMs in 30 system configurations against it, and proposes two training-free methods (Scene-CoT and TCogMap) that improve VLM temporal reasoning.

What This Paper Is About

Vision-Language Models are increasingly used as the perception-and-reasoning component of autonomous driving agents, where misjudging the order, timing, or duration of events can be dangerous. Existing video benchmarks cover sports, cooking, or movies, and existing driving benchmarks mostly use single images or focus on spatial perception, so no benchmark isolated temporal understanding for both short and long autonomous-driving clips. The authors build that benchmark (TAD) from NuScenes video, measure how far current models fall short of humans, and then propose two add-on methods that improve temporal reasoning without any retraining.

Key Contributions

1. TAD benchmark: The first question-answering benchmark exclusively for autonomous driving that evaluates temporal understanding on both short-form (segment-level) and long-form (scene-level) video, comprising 5,861 QA pairs across 7 tasks and 150 NuScenes videos.
2. New action annotation dataset: Fine-grained, human-annotated action labels for ego and non-ego vehicles across the 150 NuScenes videos, totaling 4,481 vehicle action annotations, released as a standalone resource for vehicle action recognition.
3. Comprehensive model evaluation: A benchmarking study of 9 unique VLMs across 30 system configurations, spanning open-source generalist, closed-source, and AD-specialist models, plus human and chance-level baselines.
4. Two training-free methods: Scene-CoT (Chain-of-Thought over video segments) and TCogMap (an ego-centric temporal cognitive map produced by a deterministic trajectory-analysis tool used as an agentic wrapper around the VLM), improving average TAD accuracy by up to 17.72% and STSBench accuracy by up to 10.35%.

Main Findings

• A large human-to-model gap exists. Human-level average accuracy on TAD is 74.72%, versus 34.37% for chance level. The best closed-source results reported are GPT-5-mini at 52.04% and Gemini-2.5-Flash at 52.10% average, and the best open-source baseline is InternVL3-38B at 56.49%.
• TCogMap gives the largest gains. Adding TCogMap to Qwen2.5-VL-7B raises average accuracy from 44.46% to 62.18%; on InternVL3-38B it reaches 65.66%. TCogMap outperforms the baseline-plus-raw-ego-pose variant by as much as 17.74% (Qwen2.5-VL-7B) and by 8.68% with InternVL3-38B, even though both receive identical raw information.
• Raw pose data is not enough; structure is what helps. Because the baseline + ego pose and TCogMap variants get the same underlying information, the gap shows the VLM cannot digest raw pose data but can use a structured motion summary.
• Scene-CoT helps smaller models more. Qwen2.5-VL-7B with Scene-CoT improves 4.25% over baseline, but for larger models Scene-CoT does not offer additional average gains, which the authors hypothesize is due to their stronger inherent capabilities. On Qwen2.5-VL-32B, Scene-CoT actually lowered average accuracy to 48.78% from 55.20% baseline.
• Scale still matters for baselines. Qwen2.5-VL improves by more than 10% in average performance from the 7B to the 32B variant.
• In-domain AD training is not a cure. The AD-specialist baselines RoboTron (42.58%) and Cosmos-Reason (41.21%) are on par with Qwen2.5-VL-7B (44.46%), yet the training-free extensions still add up to 11.38% average accuracy to AD-specific models.
• Cosmos-Reason struggles with frame-indexed tasks. It is significantly outperformed by RoboTron on Temporal Action Localization and Temporal Object Localization, likely because it receives whole videos rather than frame sequences and therefore lacks explicit frame-index awareness.
• Ego questions are easier than non-ego questions. For InternVL3-8B, baseline accuracy is 55.15% on ego questions versus 43.24% on non-ego questions. Scene-CoT drops ego accuracy slightly (0.52%) but improves non-ego by more than 5%; TCogMap boosts ego by 13.02% and non-ego by 4.57%.
• The cognitive map carries real temporal signal. In the blind test with InternVL3-8B, the Blind setting (question only) approaches chance level, and the TCogMap Only setting (no frames) yields higher accuracies in five tasks than Image Only. Combining frames with TCogMap is best across most tasks.
• Gains transfer to another benchmark. On STSBench, Scene-CoT raises InternVL3-8B average from 43.00% to 44.94%, while TCogMap raises it to 53.35%, a 10.35% boost, with the largest gains on ego-centric tasks.
• Humans find Temporal Action Localization hardest. Human accuracy there is 50.21%, which the authors attribute to similar motion categories (e.g., stopped vs. stopping) and minor misalignments between segment-level ground truth and frame-level human responses.

Methodology in Plain English

The authors started from the NuScenes validation split: 150 videos of roughly 20 seconds each, with sensor data and 3D bounding boxes but no action labels. They focused on seven fine-grained vehicle classes (car, bus, bicycle, construction vehicle, motorcycle, trailer, truck) and ignored other objects. Each video was split into ten uniformly distributed, overlapping five-second segments, since most vehicle atomic actions complete within five seconds. Segments were filtered to vehicles within 50 m of the ego vehicle, and vehicles with negligible trajectory displacement were automatically labeled "stopped" and excluded from manual labeling. Human annotators then watched segments alongside a bird's-eye-view visualization with trajectories and assigned one dominant action label per vehicle per segment, drawing from eight categories (straight at constant speed, stopping, stopped, starting, turning left, turning right, changing lanes left, changing lanes right).

Question-answer pairs were then generated from templates, with logical checks for ambiguity and a final human verification stage, producing 5,861 QAs across seven tasks: Exact Answer Action Recognition, Multiple Choice Action Recognition, Action Duration, Temporal Ordering, Temporal Action Localization, Relative Temporal Action Localization, and Temporal Object Localization.

For the methods, both are training-free wrappers. Scene-CoT splits a video into five-second segments with roughly 50% overlap, samples four frames per segment, and prompts the VLM to produce a four-step reasoning trace per segment (scene description, ego motion, nearby vehicle motion, JSON-style summary). The traces are passed in temporal order, along with the question, to an LLM (Qwen2.5-14B-Instruct-1M) that produces the final answer. TCogMap instead feeds the VLM an ego-centric "temporal cognitive map": a deterministic tool reads the ego vehicle's pose sequence (translation, quaternion, timestamp) from NuScenes, computes frame-wise velocities by finite differences, transforms them into the ego's local frame, and applies a hierarchical rule-based classifier to output one of eight motion labels per segment. Those labels, formatted as strings tied to specific frames, are given to the VLM together with the full frame set and the question. Evaluation used accuracy for multiple-choice questions, temporal mean intersection over union (mIoU) for frame-list outputs, and binary exact match for text outputs. Baselines received about 40 scene frames; the two segment-level action recognition tasks used about ten frames from the relevant segment. Human performance was computed on a randomly sampled 10% subset of each question type.

Why This Matters

Impact on research: TAD gives the community a dedicated, AD-exclusive testbed that separates temporal reasoning (when, in what order, how long) from spatial perception, and the reported human-versus-model gap establishes a clear target. The two methods also demonstrate that structured, tool-generated context can substitute for part of what fine-tuning would otherwise be expected to provide.

Real-world applications:

• Safer autonomous driving stacks that use a VLM as one component, since temporal-reasoning failures propagate directly to the agent's behavior.
• Driving assistants and VQA systems that must explain what the vehicle is doing and why, over multiple seconds.
• Incident review and fleet analytics that need to localize and order events within recorded driving footage.
• Evaluation pipelines for any embodied agent operating from an ego-centric camera, since the methods transfer to STSBench.

Industry relevance: The work comes from Huawei Technologies Canada and Huawei Cloud and connects to a crowded field of industry VLM-for-driving benchmarks (LingoQA, DriveLM, VLADBench, SURDS, Ego3D-Bench, CODA-LM, MetaVQA). The fact that the proposed methods are training-free and give up to 11.38% average gains on AD-specialist models makes them directly applicable to deployed VLMs without retraining costs. The authors explicitly caution, however, that TAD should be one component of a broader validation framework rather than a definitive measure of deployment readiness.

Future Directions

• Extending temporal cognitive maps beyond the ego vehicle to non-ego objects, which TCogMap currently omits deliberately to avoid overloading the VLM.
• Integrating the methods into closed-loop planning agents rather than question-answering pipelines.
• Creating a fine-tuning dataset for VLM-based agentic AD systems, since the current methods are training-free.
• Exploring more sophisticated video segmentation strategies, which the authors leave for future work, and clarifying the boundary between action definitions used in the benchmark.

Target Audience

Researchers and engineers working on Vision-Language Models, video understanding, and autonomous driving perception, especially those building or evaluating agentic driving systems that depend on temporal reasoning. It is also useful for benchmark designers in embodied AI who need a model of how to construct fine-grained action annotations and how to test whether a model is genuinely using temporal context rather than question-level shortcuts.

Authors’ abstract

Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances. Reliable temporal understanding is essential for such agents to anticipate events, attribute causes, and act safely in dynamic environments, yet this remains a significant challenge even for state-of-the-art (SoTA) VLMs. Prior video benchmarks have emphasized other content (sports, cooking, etc.), yet no existing benchmark focuses exclusively on temporal understanding for both short- and long-form AD footage. To fill this gap, we present the Temporal Understanding in Autonomous Driving (TAD) benchmark, comprising nearly 6000 question-answer (QA) pairs across 7 tasks, and evaluate 9 closed- and open-source generalist as well as AD-specialist models. Current SoTA models perform substantially below human accuracy on TAD. To improve the temporal reasoning of VLM-based driving agents, we propose two novel training-free solutions: Scene-CoT, which uses Chain-of-Thought (CoT) reasoning, and TCogMap, which incorporates an ego-centric temporal cognitive map produced by a trajectory-analysis module that operates as an agentic tool around the VLM. Integrated with existing VLMs, our methods improve average accuracy on TAD by up to $17.72\%$ and by up to $10.35\%$ on STSBench. By introducing TAD, benchmarking SoTA models, and proposing effective enhancements, this work aims to catalyze further progress on temporal understanding for agentic AD systems operating in the wild. The benchmark and evaluation code are available at ${\href{https://huggingface.co/datasets/vbdai/TAD}{\text{Hugging Face}}}$ and ${\href{https://github.com/vbdi/tad_bench}{\text{GitHub}}}$, respectively.