UrbanIng-V2X: A Large-Scale Multi-Vehicle, Multi-Infrastructure Dataset Across Multiple Intersections for Cooperative Perception

AI summary

Overview

Research area: Computer vision for autonomous driving — specifically cooperative perception (Vehicle-to-Everything, or V2X), 3D object detection, and multi-modal sensor fusion.

Technical level: Intermediate. Readers need basic familiarity with 3D object detection, LiDAR point clouds, and benchmark metrics (mAP, IoU), but the paper's central arguments about dataset design and generalization are accessible to a broader audience.

Scope: The paper introduces UrbanIng-V2X, a real-world cooperative perception dataset spanning three urban intersections in Ingolstadt, Germany, with multiple connected vehicles and infrastructure sensor poles, and establishes baseline benchmarks showing that state-of-the-art fusion models degrade substantially when tested at intersections they have never seen.

What This Paper Is About

Autonomous vehicles have a hard time seeing around occlusions — a truck blocking a pedestrian, a building blocking a cyclist. Cooperative perception addresses this by letting vehicles share sensor data with each other and with roadside infrastructure. The problem is that existing real-world datasets for this task are too narrow: most cover only one intersection or only one vehicle, so algorithms can score high by memorizing a single layout rather than genuinely learning to fuse information. This paper builds a dataset that deliberately spans three different intersections with two vehicles and multiple infrastructure poles, forcing models to prove they can generalize.

Key Contributions

1. A multi-intersection, multi-agent dataset. UrbanIng-V2X is the first real-world cooperative perception dataset combining multiple connected vehicles with extensive infrastructure sensing at three distinct urban intersections. Prior real-world V2X datasets either cover one intersection or include only one vehicle.

2. The largest cooperating sensor fleet in a real-world V2X dataset. Each scenario involves two vehicles (12 RGB cameras, 2 LiDARs) cooperating with up to 17 thermal cameras and 12 LiDARs mounted on 7 infrastructure poles across the three sites. Thermal cameras are a modality largely absent from prior cooperative datasets, and all sensors are spatially calibrated and temporally aligned.

3. Large-scale, multi-task annotations. 34 sequences of 20 seconds each, annotated at 10 Hz with 3D bounding boxes across 13 object classes, yielding roughly 712,000 annotated instances — with tracking IDs, six attribute types, traffic-light labels, and per-object occlusion states.

4. New generalization-focused evaluation protocols and baselines. The paper proposes Equal Intersection Split (EIS) and Separate Intersection Split (SIS), then benchmarks five state-of-the-art fusion methods under both, exposing a large generalization gap. It also releases a developer toolkit (converters for OpenCOOD and nuScenes), HD maps in Lanelet2 format, and a geo-referenced CARLA digital twin of the collection environment.

Main Findings

• Generalization collapses on unseen intersections. Using CoBEVT, performance averaged 38.2 mAP@0.5 on EIS (known intersections) but dropped to 24.2 mAP@0.5 averaged across SIS configurations (unseen intersections) — a 14.0 mAP@0.5 gap that the authors frame as the dataset's headline challenge.

• Intermediate fusion wins, late fusion loses. On the SIS 1/2 vs. 3 split, intermediate fusion methods (AttFuse, CoBEVT, F-Cooper, V2X-ViT, Where2Comm) reached 23.8–25.5 mAP@0.5, beating no-fusion (18.3), early fusion (22.8), and late fusion (11.1). Late fusion's poor showing points to genuine difficulty in associating object lists produced independently by different agents.

• AttFuse generalizes best; V2X-ViT leads on seen intersections. AttFuse achieved the top score on the unseen-intersection split (29.4/25.5 mAP), while V2X-ViT scored highest on the single EIS split (41.9/33.8 mAP), suggesting different architectural strengths for in-domain versus out-of-domain performance.

• Class difficulty tracks object size and shape variance. The Vehicle superclass (cars, vans) is detected most reliably. Pedestrians are hardest because they are small, and Two-Wheelers and Heavy Vehicles are difficult because each superclass merges three or more original annotation categories with wide variation in physical dimensions.

• Scene complexity differs measurably by intersection. Frames contain an average of 103.9 objects at Intersection 1, 128.6 at Intersection 2, and 78.2 at Intersection 3. Intersection 1 yields the densest point-level visibility per object, Intersection 2 the most dynamic and crowded scenes, and Intersection 3 the sparsest perceptual coverage.

Methodology in Plain English

The researchers equipped two cars with a high-end 128-beam LiDAR, six cameras covering a full 360 degrees, and a GPS/IMU unit accurate to about one centimeter. At each of three intersections they mounted two to three poles holding thermal cameras and a mix of 64-beam and 32-beam LiDARs. All devices share a common time reference derived from UTC: the cars use GPS-synced Precision Time Protocol, and the infrastructure poles use dedicated time servers and GPS receivers on the LiDARs. Cameras on the vehicles are triggered by hardware the moment the spinning LiDAR beam sweeps through their field of view, which minimizes latency between the two modalities. Thermal cameras run freely at 30 frames per second and are matched afterward to the closest LiDAR scan, with at most 16.6 milliseconds of misalignment.

Calibration happens in two stages. Camera intrinsics come from standard checkerboard procedures, with fisheye models for the wide-angle side cameras. For extrinsics — the precise position and orientation of every sensor relative to the vehicle or intersection frame — the team placed a cone with a reflective marker at multiple known GPS-measured positions, manually labeled it in both images and point clouds, and numerically optimized transformation matrices to minimize reprojection error.

Because a spinning LiDAR captures each point at a slightly different moment, and because the vehicles are moving, the team applies per-point motion compensation before fusing any point clouds into a shared global frame. This makes it possible to project any LiDAR point into any camera image. The authors quantify the residual error: assuming objects moving at up to 50 km/h, up to 0.7 meters of spatial misalignment within a single frame is unavoidable.

From roughly eight hours of raw recording, they curated 34 twenty-second scenarios covering varied lighting (day, cloudy, moderate light, late evening, night), then had a professional annotation company label fused point clouds with 3D boxes through three separate review cycles. Faces and license plates across 163,200 RGB images were blurred with a 75×75 Gaussian kernel.

For benchmarking, they trained five cooperative detection models on a PointPillars backbone for 60 epochs and evaluated them across two families of data splits — one mixing all intersections but separating sequences, and one holding out entire intersections to test transfer to wholly unfamiliar locations.

Why This Matters

Impact on research. The 14-point generalization drop is a concrete, reproducible measurement of a problem the field has largely asserted rather than demonstrated. By providing held-out intersections with a shared annotation standard, UrbanIng-V2X gives researchers a benchmark where overfitting to a single layout cannot be hidden. The inclusion of thermal imagery also opens a modality that most cooperative datasets skip entirely, which matters for night and low-visibility conditions.

Real-world applications:

• Urban intersection safety systems. Cities deploying roadside sensor poles can use these baselines to estimate how much a new infrastructure investment actually improves detection before committing to a full rollout.
• Occlusion-aware collision avoidance. The dataset directly targets the scenario where a vehicle cannot see a pedestrian or cyclist because of a bus or building, and infrastructure sensing fills the gap.
• Cross-city deployment planning. The SIS protocols simulate deploying a model trained in one city at intersections in another, which is exactly what fleet operators face when scaling beyond a pilot region.
• Synthetic-to-real transfer. The released CARLA digital twin of all three intersections enables generating labeled synthetic data and studying how well it transfers to real recordings.

Industry relevance. Automotive OEMs, Tier-1 suppliers, and infrastructure operators all need to know whether cooperative perception generalizes beyond a single instrumented intersection before building business cases around it. The released nuScenes and OpenCOOD format converters lower the barrier for teams already using those pipelines, and the digital twin supports simulation-driven development cycles where real data collection is expensive.

Future Directions

• Breaking the 14-point generalization gap. The paper's own results show no existing architecture solves cross-intersection transfer. Whether the answer lies in domain adaptation, more diverse training data, or fundamentally different fusion architectures remains open.

• Expanding geographic and environmental diversity. All three intersections are in one German city. Extending to other cities, topologies, and countries — and adding rain, fog, and snow — is the authors' stated next step, but no such data currently exists at comparable scale.

• Exploiting thermal and multi-modal fusion. The paper only benchmarks LiDAR-only fusion. How much the 17 thermal cameras actually contribute to detection at night or in adverse conditions is untested and is an obvious follow-up experiment.

• Temporal and predictive tasks. Tracking, trajectory prediction, and localization are supported by the annotations but not benchmarked here. Whether cooperative sensing improves trajectory prediction as much as it improves detection is an unanswered question.

Target Audience

Researchers and engineers working on cooperative perception, V2X communication, and multi-agent autonomous driving — particularly those building or evaluating 3D object detection and fusion models. It is also valuable to practitioners planning smart-infrastructure deployments, since the SIS results give a realistic estimate of how a model trained in one place will behave in another. Graduate students entering the field will find the dataset comparison table and the EIS/SIS split design useful as reference points for their own evaluation methodology.

Authors’ abstract

Recent cooperative perception datasets have played a crucial role in advancing smart mobility applications by enabling information exchange between intelligent agents, helping to overcome challenges such as occlusions and improving overall scene understanding. While some existing real-world datasets incorporate both vehicle-to-vehicle and vehicle-to-infrastructure interactions, they are typically limited to a single intersection or a single vehicle. A comprehensive perception dataset featuring multiple connected vehicles and infrastructure sensors across several intersections remains unavailable, limiting the benchmarking of algorithms in diverse traffic environments. Consequently, overfitting can occur, and models may demonstrate misleadingly high performance due to similar intersection layouts and traffic participant behavior. To address this gap, we introduce UrbanIng-V2X, the first large-scale, multi-modal dataset supporting cooperative perception involving vehicles and infrastructure sensors deployed across three urban intersections in Ingolstadt, Germany. UrbanIng-V2X consists of 34 temporally aligned and spatially calibrated sensor sequences, each lasting 20 seconds. All sequences contain recordings from one of three intersections, involving two vehicles and up to three infrastructure-mounted sensor poles operating in coordinated scenarios. In total, UrbanIng-V2X provides data from 12 vehicle-mounted RGB cameras, 2 vehicle LiDARs, 17 infrastructure thermal cameras, and 12 infrastructure LiDARs. All sequences are annotated at a frequency of 10 Hz with 3D bounding boxes spanning 13 object classes, resulting in approximately 712k annotated instances across the dataset. We provide comprehensive evaluations using state-of-the-art cooperative perception methods and publicly release the codebase, dataset, HD map, and a digital twin of the complete data collection environment.