RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios

AI summary

Overview

Research area: multimodal large language models (MLLMs), spatial understanding and reasoning benchmarks, urban and road-scene computer vision.

Technical level: Advanced. The paper assumes familiarity with MLLM evaluation practice, Bird's-Eye View (BEV) versus First-Person View (FPV) imagery, and metrics such as Hamming Loss, RMSE, and Fréchet Distance.

Scope: RoadBench is a systematically constructed benchmark of 3,040 manually verified test cases across eight tasks that measures how well MLLMs understand and reason about fine-grained urban spatial elements, using road markings as the representative example.

What This Paper Is About

Existing urban-scene benchmarks for MLLMs mostly test whole-image understanding or isolated object recognition (for example GeoQA, landmark recognition, vehicle detection), leaving fine-grained spatial elements largely unmeasured. The authors argue that road markings, which are thin, image-spanning, and semantically interdependent, expose three specific gaps: recognizing fine-grained structure at global scale, jointly understanding multiple fine-grained structures, and combining image information with traffic domain knowledge. RoadBench is built to fill that gap by testing MLLMs on road markings and the road networks they form, from both BEV and FPV viewpoints.

Key Contributions

1. A systematic benchmark named RoadBench with eight tasks spanning BEV and FPV image inputs, organized to move from local perception to global contextual reasoning, plus cross-view tasks requiring joint reasoning across perspectives.
2. A dataset of 3,040 test cases, built from 2,137 unique BEV images and 721 unique FPV images collected from five Chinese cities with relatively consistent traffic conventions, with all labels rigorously manually verified. Data sources include satellite imagery, a tier-1 online map service provider's database, OpenStreetMap, and crowd-sourced in-vehicle camera photos.
3. A systematic evaluation of 20 mainstream closed-source and open-source MLLMs, plus rule-based and random baselines, and additional fine-tuning of Qwen3-VL models at 2B, 4B, and 8B parameters on the RoadBench training set.
4. Ablation-style analyses of reference-line prompting strategies in BEV tasks and of adverse scene conditions (poor lighting, obscured markings) in FPV tasks, reported in the paper's appendix tables.

Main Findings

• RoadBench is a highly challenging benchmark for MLLMs. In BEV lane counting, the best model, GPT-5-Mini, reached an F1-Score of only 0.355, with RMSE between predicted and ground-truth lane counts of 1.162.
• Models struggle to produce precise coordinates. In BEV road network correction, the paper reports the best model at an RMSE@20% of 0.132 for junction points and an FD@50% of 0.232 for road segment polylines, both within the same order of magnitude as the upper thresholds. At the small 10% threshold, results are almost entirely contributed by the distance upper bound, indicating large shifts from ground truth.
• MLLMs often fail to beat simple rule-based baselines. In both lane designation recognition tasks, most MLLMs could not outperform a baseline mapping table designed from traffic common sense (rule-based baseline: Hamming Loss 0.149 and accuracy 0.587 in BEV; 0.127 and 0.605 in FPV). Outside the FPV road type classification task, some MLLMs also failed to beat random-choice baselines.
• FPV is easier than BEV for the same task. MLLMs performed significantly better on FPV images than BEV images, both in absolute metric values and relative to baseline, which the authors attribute to larger spatial elements in FPV being easier to understand.
• Cross-view reasoning remains difficult. Most models performed noticeably worse on Cross-View Lane Counting and Cross-View Lane Designation than on single-view counterparts, with clear drops even for GPT-5 and Gemini-2.5.
• Fine-tuning substantially helps small Qwen3-VL models. Tuned Qwen3-VL-2B, 4B, and 8B improved consistently across most tasks, and Qwen3-VL-8B (tuned) became the best of that group (for example F1 0.433 on FPV lane counting, accuracy 0.708 on FPV lane designation recognition, accuracy 0.728 on cross-view lane designation recognition, accuracy 0.830 on FPV road type classification), while still trailing leading closed-source models.
• More parameters is not a universal solution. GPT-5-Mini outperformed GPT-5 on most tasks, and Qwen2.5-VL-32B outperformed Qwen2.5-VL-72B in the FPV lane recognition task.
• Closed-source models hold an advantage. The GPT-5 series and Gemini-2.5 series generally led, and among open-source models only GLM-4.5V ranked highly in the comprehensive evaluation.
• Prompt format matters and differs by model. GLM-4.5V benefited most from combined textual and arrow-based image prompts, with image prompts dominant; removing image prompts caused a noticeable decline, while removing text prompts caused only minor fluctuation. GPT-5-Mini preferred color-based direction prompts, and prompt strategy made only marginal difference in BEV lane designation recognition.
• Adverse conditions degrade performance, but unevenly. The authors identified 175 test cases with adverse lighting conditions and 46 test cases with obscured road markings; poor lighting hurt performance but the impact was significantly alleviated in models with stronger image understanding. Shifting the basis for the answer from obscured markings to other cues such as signage substantially improved GPT-5 and Gemini-2.5-Pro, indicating weaker ability on fine-grained spatial elements than on other capabilities.

Methodology in Plain English

The authors chose road markings as a test case because they are small, elongated, span whole images, and must be read together to make sense. They built tasks around them in two viewing modes: satellite-style top-down (BEV) images, and in-car camera (FPV) images. In BEV tasks, a red polyline with arrowheads is drawn on the satellite image to mark the road of interest, and the model must answer about lanes, lane directions, or correct the road network. In FPV tasks the reference line is removed, so the model must work out the spatial relationship between the camera and the surrounding roads. Two further tasks give the model geographically paired BEV and FPV images of the same scene and require joint reasoning; a final task asks only whether the vehicle is on a main road or a service road.

Data construction ran in three stages. In preparation, the team downloaded OpenStreetMap data in areas of relatively high satellite image resolution, extracted junctions, and manually reviewed satellite imagery from Google Maps to discard fake, hard-to-identify, or severely obscured views, while deliberately retaining a proportion of challenging cases such as nighttime conditions. In processing, bounding boxes were fed into a tier-1 online map service provider's database to pull ground truth for road centerlines, junction locations, lane counts, and lane directions; matching against OpenStreetMap produced the deliberately inaccurate reference lines used for the road network correction task. In quality control, five domain experts reviewed and refined all samples on the LabelU platform, excluding ambiguous or fundamentally erroneous samples rather than re-annotating them.

Evaluation covers 20 MLLMs plus baselines. Lane counting uses weighted Precision, Recall, F1, and RMSE; lane designation recognition uses Hamming Loss and accuracy; road network correction matches predictions to ground truth by nearest neighbor (unmatched items mapped to infinity) and scores junction points with RMSE and polylines with Fréchet Distance under normalized distance upper bounds of 10%, 20%, and 50%; FPV road type classification uses accuracy on a balanced binary task. Failed API calls, empty responses, and wrong formats trigger retries up to a maximum of six attempts, after which the outcome is recorded as zero or empty. All coordinates are normalized to [0,1] before metric calculation. Because quality control retained only data humans could identify correctly, no human baseline is provided.

Why This Matters

This work opens a specific evaluation axis—fine-grained, highly interdependent spatial structures—that prior urban benchmarks such as CityBench, UrBench, CityEQA, DriveBench, NuPlanQA, and MapDR did not isolate, as the paper's comparison table argues. It also supplies evidence that current MLLMs can be beaten by trivial rule-based or random guessing on some urban spatial tasks, which reframes how strong their spatial reasoning really is.

Real-world applications:

• High-definition map auto-generation, where lane counts, lane directions, and junction geometry must be extracted reliably.
• End-to-end autonomous driving, where lane designation and road type affect planning and routing.
• Map service and navigation correction, since missed junctions in a road network can send users on incorrect routes.
• Urban and transportation analysis, plus embodied intelligence and remote-sensing-adjacent systems that depend on spatial grounding.

Industry relevance: the data pipeline draws directly on a tier-1 online map service provider's production database and on OpenStreetMap, and the tasks were chosen for practical value (road network correction is described as having significant practical value). That makes the benchmark directly legible to map providers, autonomous driving teams, and anyone building spatial assistants.

Future Directions

• Closing the gap in fine-grained spatial understanding, since models performed much better when answers could be derived from coarse cues such as signage than from fine-grained markings themselves.
• Improving precise structured numeric output, given that generated points and polylines shifted substantially from ground truth even for strong models.
• Making cross-view fusion work, since even GPT-5 and Gemini-2.5 dropped noticeably when BEV and FPV views had to be aligned.
• Determining which training factors drive these failures, since model scale alone did not predict performance (GPT-5-Mini beat GPT-5 on most tasks; Qwen2.5-VL-32B beat the 72B model on FPV lane recognition), and prompt-format preferences varied between models.

Target Audience

Researchers and engineers working on multimodal large language models, spatial reasoning evaluation, and urban or road-scene perception. It is most useful to those building or auditing benchmarks in autonomous driving, high-definition mapping, and navigation services, and to MLLM developers looking for a hard, manually verified test set with a documented data pipeline, released code at https://github.com/tsinghua-fib-lab/RoadBench, and supplementary material containing example data, prompts, evaluation scripts, and raw evaluation results.

Authors’ abstract

Multimodal large language models (MLLMs) have demonstrated powerful capabilities in general spatial understanding and reasoning. However, their fine-grained spatial understanding and reasoning capabilities in complex urban scenarios have not received significant attention in the fields of both research and industry. To fill this gap, we focus primarily on road markings as a typical example of fine-grained spatial elements under urban scenarios, given the essential role of the integrated road traffic network they form within cities. Around road markings and urban traffic systems, we propose \textbf{RoadBench}, a systematic benchmark that comprehensively evaluates MLLMs' fine-grained spatial understanding and reasoning capabilities using Bird's-Eye View (BEV) and First-Person View (FPV) image inputs. This benchmark comprises eight tasks consisting of 3,040 strictly manually verified test cases, constructed from 2,137 unique BEV images and 721 unique FPV images collected from five Chinese cities with relatively consistent traffic conventions. These tasks form a systematic evaluation framework that bridges understanding at local spatial scopes to global reasoning. They not only test MLLMs' capabilities in recognition, joint understanding, and reasoning but also assess their ability to integrate image information with domain knowledge. After evaluating 20 mainstream MLLMs, we confirm that RoadBench is a challenging benchmark for MLLMs while revealing significant shortcomings in existing MLLMs' fine-grained spatial understanding and reasoning capabilities within urban scenarios. In certain tasks, their performance even falls short of simple rule-based or random selection baselines. These findings, along with RoadBench itself, will contribute to the comprehensive advancement of spatial understanding capabilities for MLLMs.