GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Models

AI summary

Overview

Research area: Computer vision, specifically spatial-temporal intelligence of Vision-Language Models (VLMs), evaluated through a new benchmark that combines geographic maps with multi-camera video.

Technical level: Intermediate. The task definitions and metrics are easy to grasp, but full appreciation of the benchmark construction pipeline (camera calibration, homography projection, trajectory transformation) and the model comparison assumes some familiarity with VLMs and multi-camera tracking.

One-sentence scope: The paper introduces GTR-Bench, a 420-question benchmark built from 364 real videos across outdoor (vehicle) and indoor (pedestrian) camera networks, and shows that even the best proprietary VLM reaches only 34.9% accuracy versus 78.61% for humans on geographic temporal reasoning.

What This Paper Is About

Existing spatial-temporal benchmarks for VLMs either test egocentric (first-person) reasoning from images or video, or test geographic reasoning using maps and other graphical context alone. Neither combination matches real-world settings such as traffic management or emergency response, where a model must combine a map with video from many cameras that do not overlap. The paper defines this missing capability as Geo-Temporal Reasoning (GTR) and builds GTR-Bench to measure it, covering moving targets (vehicles outdoors, pedestrians indoors) in large-scale camera networks where the model must reason about places and times it never directly observes.

Key Contributions

1. A new problem formulation. The authors propose Geo-Temporal Reasoning: geographic temporal reasoning about moving targets inside a large-scale camera network, requiring joint use of visual context (images/video) and graphical context (maps), multiple perspective switches between map and cameras, joint reasoning across videos with non-overlapping fields of view, and inference over spatial-temporal regions unobserved by any video.

2. The GTR-Bench benchmark. A hierarchical, multi-level evaluation framework with seven tasks and 420 questions derived from 364 distinct video clips (average duration 10.64 seconds). Three basic tasks (Geo-Location, Arrival Time-Interval, Motion-State) isolate core abilities; four combinatorial tasks (Causal Reordering, Next Spot Forecasting, Trajectory Forecasting, Multi-Target Trajectory Forecasting) test how models integrate them. Questions are balanced at 60 per task (7 × 30 for each of the two sub-datasets, GTR-Outdoor and GTR-Indoor).

3. A new evaluation metric. Beyond standard multiple-choice accuracy (MCQ Acc), the authors introduce Spatial-Temporal Intersection over Union (ST-IoU), which multiplies an indicator for correct predicted camera ID by the temporal IoU between predicted and ground-truth time intervals. This forces forecasting tasks to be right in both space and time.

4. A diagnostic analysis of current VLMs. Evaluation of more than 10 models reveals three major deficiencies: imbalanced utilization of spatial, temporal, and motion-state context; weak temporal forecasting; and poor alignment of map data with multi-view video inputs.

Main Findings

• A large human-model gap. The best model tested, Gemini-2.5-Pro, scores 34.9% (34.93) overall, while human performance is 78.61%. The best open-source model, InternVL3-38B, reaches 30.76%. Average scores reported are OS Avg 23.82 and PM Avg 33.15.

• Top proprietary models cluster tightly. Gemini-2.5-Pro (34.93), GPT-5 (34.05), and Claude-4-Sonnet (34.03) occupy the top three ranks, followed by GPT-4o (33.53). Top open-source results are InternVL3-38B (30.76), Qwen2.5-VL-32B-Instruct (30.45), and GLM-4.1V-9B-Thinking (28.51). The weakest listed model is Qwen2-VL-2B-Instruct at 17.65.

• Forecasting tasks are extremely hard. Next Spot Forecasting, Trajectory Forecasting, and Multi-Target Trajectory Forecasting use ST-IoU and produce very low scores, including many near-zero results for open-source models. For example, GPT-4o scores 0.00 ST-IoU on outdoor Trajectory Forecasting and Claude-4-Sonnet scores 0.00 on indoor Multi-Target Trajectory Forecasting.

• Spatial performance far exceeds spatial-temporal performance. Models identify locations reasonably well but collapse when time is required. GPT-4o achieves 53.33 MCQ accuracy on outdoor Next Spot Forecasting but only 20.53 ST-IoU, a 32.80 point gap. Gemini-2.5-Pro shows 45.45 MCQ accuracy versus 13.16 ST-IoU on outdoor Trajectory Forecasting, a 32.29 point difference. GLM-4.1V-9B-Thinking reaches 76.67 MCQ accuracy on outdoor Multi-Target Trajectory Forecasting but only 25.38 ST-IoU, a 51.29 point difference.

• Outdoor usually beats indoor, with one notable exception. Most models perform better outdoors, consistent with clearer spatial cues and more regular motion patterns. Claude-3.7-Sonnet, for instance, scores 25.75, 8.90, and 21.97 outdoors on NSF, TF, and MTTF versus 9.48, 9.51, and 3.56 indoors. Gemini-2.5-Pro is counterintuitive, scoring higher indoors (25.11, 28.09, 14.37) than outdoors (19.13, 13.16, 19.18), which the authors suggest reflects advanced models engaging deeper reasoning in the more complex indoor setting.

• Context usage is imbalanced. A prompt-based self-assessment where the model rates its own use of spatial, temporal, and motion-state context (0 or 1 each) shows Gemini-2.5-Pro using context types in a balanced way, while open-source models show skew—for example, InternVL3-38B has strong spatial and motion understanding but weaker temporal reasoning.

• A traditional ReID baseline is competitive on forecasting. The ReID-based baseline scores 45.72 overall, higher than every evaluated VLM, with particularly strong forecasting numbers (66.67, 38.33, 66.67 outdoors; 43.33, 40.00, 52.17 indoors), though it does not apply to all task types.

• Distinctive error patterns. The failure case study identifies Topology Error (using direct linear distance and ignoring wall obstructions), Motion State Error (failing during world-direction transformation), FoV Alignment Error (equating location with viewpoint instead of mapping camera field of view to the spatial layout), and Time Interval Error (ignoring distance and speed constraints when estimating traversal time).

• Benchmark scale and structure. GTR-Bench has 420 questions, 364 unique videos, 7.0 average images per question, average question length 185.1, and average choice length 132.4. GTR-Outdoor covers vehicles across 31 cameras with an average inter-camera distance of 984.77 m (max 2389.42 m); GTR-Indoor covers pedestrians across 16 cameras with an average distance of 30.59 m (max 60.93 m). The paper describes the outdoor scenario as approximately 32 times larger in spatial scale. Human per-task scores range from 51.24 (outdoor Trajectory Forecasting) to 98.20 (indoor Geo-Location).

Methodology in Plain English

The authors built the benchmark from two existing multi-camera tracking datasets: CityFlow (outdoor vehicles, 345 target IDs and 40 cameras) and MTMMC (indoor pedestrians, 3669 target IDs and 32 cameras). Video frames are cropped around each annotated bounding box to isolate individual target instances. For each camera they compute a homography matrix—a 3×3 transformation derived from manually annotating at least eight correspondence points between the image and a top-down map—so that pixel coordinates can be projected onto real-world map positions in meters. From those projected trajectories they derive distance, velocity, and direction, then clean and validate the data and use a large language model to turn the numbers into a narrative "Motion Summary."

Questions are then generated from templates in four steps: sample valid trajectory segments, integrate the trajectory with map and video data, formulate the question and ground-truth answer, and generate plausible but wrong distractors (from architecturally distinct indoor areas, synthetic outdoor cameras, or randomized camera IDs) so models cannot succeed through superficial heuristics. Tasks are grouped into Long, Medium, and Short geo-temporal complexity tiers based on trajectory length and duration, and the final set passed a two-stage manual review—first selecting context for diversity in spatial span and temporal duration and removing questions with large trajectory errors, then having human experts pick answers and select 30 questions per task.

For evaluation, 13 state-of-the-art models were tested. Proprietary models (Claude-3.7-Sonnet, Claude-4-Sonnet, GPT-4o, GPT-5, Gemini-2.5-Pro) were accessed through official APIs; open-source models (InternVL3-2B/8B/38B, Qwen2-VL-2B/7B, Qwen2.5-VL-2B/7B/32B, GLM-4.1V-9B-Thinking) were deployed with LMDeploy on 8 NVIDIA V100 GPUs. Videos were uniformly sampled so that total frames across multiple videos stayed within 20, with temperature 0.1 and max_new_token 16,384. Human performance was measured on the same questions, and a traditional ReID-based baseline was added as a reference point.

Why This Matters

Impact on research. The paper shifts spatial-temporal evaluation away from egocentric, single-camera perception toward a geographic, multi-camera setting that requires map-video alignment and reasoning about unobserved regions. It also contributes a new metric (ST-IoU) that penalizes models for getting location right while getting timing wrong—a failure mode invisible under standard multiple-choice scoring.

Real-world applications:

• Traffic management: tracking and forecasting vehicle movement across a city's camera network to anticipate congestion.
• Emergency response: predicting where a target will be next, and when, to direct responders.
• Multi-camera person tracking: replacing purely visual-feature-based Re-identification pipelines with reasoning that uses geographic and temporal context.
• Urban and building operations: monitoring pedestrian flow in multi-level indoor environments, where the benchmark's indoor subset shows models behave differently from outdoors.

Industry relevance. The evaluated models include widely deployed commercial and open-source systems, and the gap to human performance is large across the board. The finding that a traditional ReID baseline scores 45.72 overall—above every VLM tested—is a practical signal that current VLMs are not yet ready to replace conventional tracking pipelines in surveillance-scale deployments.

Future Directions

• Fix temporal reasoning specifically. Since the largest scores come from MCQ-based spatial tasks and the smallest from ST-IoU-based forecasting tasks, targeted training objectives for temporal forecasting, rather than generic spatial pretraining, are the obvious next step.
• Improve map-video alignment. The error taxonomy (Topology Error, Motion State Error, FoV Alignment Error, Time Interval Error) points to concrete failure modes—such as treating shortest straight-line distance as the actual route—that could motivate explicit geometric and field-of-view-aware model components.
• Explain and correct context imbalance. The self-assessed context-utilization analysis suggests some models underuse the temporal channel; understanding why, and whether it is a data or an architecture issue, remains open.
• Extend scale and scenario coverage. The benchmark deliberately pairs a large outdoor vehicle network with a smaller indoor pedestrian network; expanding to more camera counts, more diverse environments, and longer time horizons would test whether the observed patterns generalize. The authors state that the benchmark and code will be released at https://github.com/X-Luffy/GTR-Bench.

Target Audience

This paper is most useful to researchers and engineers working on Vision-Language Models, multi-camera tracking and re-identification, video understanding, and spatial-temporal reasoning; to practitioners in intelligent transportation, video surveillance, and emergency response evaluating whether VLMs can replace or augment conventional pipelines; and to benchmark designers interested in the multi-perspective, map-plus-video task format and the ST-IoU metric. Readers looking for a quick orientation will find the task table and results table sufficient, while those interested in replication will need the construction pipeline details covering calibration and distractor generation.

Authors’ abstract

Recently spatial-temporal intelligence of Visual-Language Models (VLMs) has attracted much attention due to its importance for autonomous driving, embodied AI and general AI. Existing spatial-temporal benchmarks mainly focus on egocentric (first-person) perspective reasoning using images/video contexts, or geographic reasoning with graphical context (e.g., maps), thus fail to assess VLMs' geographic spatial-temporal intelligence that requires integrating both images/video and graphical context, which is crucial for real-world scenarios such as traffic management and emergency response. To address the gaps, we introduce Geo-Temporal Reasoning benchmark (GTR-Bench), a novel challenge for geographic temporal reasoning of moving targets in a large-scale camera network. GTR-Bench is more challenging as it requires multiple perspective switches between maps and videos, joint reasoning across multiple videos with non-overlapping fields of view, and inference over spatial-temporal regions that are unobserved by any video context. Evaluations of more than 10 popular VLMs on GTR-Bench show that even the best proprietary model, Gemini-2.5-Pro (34.9\%), significantly lags behind human performance (78.61\%) on geo-temporal reasoning. Moreover, our comprehensive analysis on GTR-Bench reveals three major deficiencies of current models for geo-temporal reasoning. (1) VLMs exhibit imbalanced utilization of spatial and temporal context during reasoning. (2) they show weak temporal forecasting ability, leading to poorer performance on temporally focused tasks. (3) they lack the capability to effectively align and integrate map data with multi-view video inputs. We believe GTR-Bench offers valuable insights and opens up new opportunities for research and applications in spatial-temporal intelligence. Benchmark and code will be released at https://github.com/X-Luffy/GTR-Bench.