SimBEV2X: A Large-Scale Dataset and Data Generation Tool for Multi-Task Vehicle-to-Everything Cooperative Perception

AI summary

Overview

Research area: Computer vision for autonomous driving — specifically cooperative perception via vehicle-to-everything (V2X) communication, with a focus on bird's-eye view (BEV) spatial representations.

Technical level: Advanced. The work assumes familiarity with BEV perception, multi-agent sensor fusion, 3D detection, semantic segmentation, and occupancy prediction.

Scope: The paper introduces a CARLA-based synthetic data generation tool, releases the largest V2X perception dataset to date with multi-modal, multi-task annotations from vehicles and roadside units, and proposes a new fusion architecture evaluated as a baseline on that dataset.

What This Paper Is About

A single autonomous vehicle cannot see through occlusions or beyond its sensor range, so researchers want vehicles to share what they perceive with each other and with roadside infrastructure. Progress on this is blocked by a shortage of large, multi-modal datasets that support several perception tasks at once, because collecting and annotating synchronized real-world data from many agents at the same time is prohibitively expensive. The paper addresses this by generating that data synthetically at scale and by providing a strong baseline model to build on.

Key Contributions

1. SimBEV2X, a data generation tool built on the CARLA simulator that automatically creates randomized driving scenarios and records multi-modal sensor data from both connected vehicles and roadside units (RSUs).
2. Multi-task ground truth generation, including 3D bounding boxes with unique track IDs, HD map information, BEV segmentation maps, and semantic occupancy voxel grids — from vehicles and RSUs alike.
3. The SimBEV2X dataset, described as the largest V2X perception dataset to date at an order of magnitude larger than existing V2X datasets, spanning 258 scenes with up to 8 connected vehicles and up to 4 RSUs across varied road networks.
4. CoBEVFusion, a novel architecture combining CoopDet3D with fused axial attention (FAX) for context-aware multi-agent feature aggregation, along with a CoopDet3D baseline established on the dataset.

Main Findings

• Dataset scale: SimBEV2X contains 102,200 frames, 588,520 lidar point clouds, more than 3 million images, and over 27 million bounding boxes, plus additional annotations. The abstract describes this as an order of magnitude larger than existing V2X datasets.
• Multi-agent configuration: Each of the 258 scenes involves up to 8 connected vehicles and up to 4 RSUs, and the scenes are spread across a variety of road networks.
• Annotation breadth: The dataset supports multiple task types simultaneously through 3D bounding boxes with unique track IDs, HD map information, BEV segmentation maps, and semantic occupancy voxel grids.
• Model performance: The authors establish a CoopDet3D baseline and report that CoBEVFusion, which adds fused axial attention to CoopDet3D, achieves superior performance. The abstract does not report specific metrics, margins, or comparisons, so the size of that improvement is not stated.
• Root cause addressed: The paper attributes the limited size and scope of prior V2X datasets to the cost of collecting and annotating synchronized real-world multi-agent data, and positions simulation as the remedy.

Methodology in Plain English

The authors build a tool on top of CARLA, a driving simulator, that generates randomized driving scenarios automatically rather than requiring a human to design each one. Within those scenarios, the system places multiple connected vehicles and roadside units, each with sensors, and records what they perceive from their different vantage points. Alongside the raw sensor data, the tool computes the correct answers a model should learn to produce — where every object is in 3D and how it moves over time, the layout of the road, a top-down map of what class each region belongs to, and a 3D grid describing which volumes of space are occupied. Because everything is simulated, these labels come for free rather than through manual annotation. The authors then train a cooperative detection model called CoopDet3D on this data to serve as a baseline, and propose CoBEVFusion, which augments that model with fused axial attention so the network can weigh and combine features contributed by different agents in a context-aware way.

Why This Matters

Impact on research: The paper targets a data bottleneck rather than a modeling bottleneck. If a simulation tool can produce synchronized multi-agent, multi-modal data with several annotation types at once, V2X perception research is no longer constrained by the cost of real-world data collection, and tasks like BEV segmentation and occupancy prediction can be studied cooperatively alongside 3D detection.

Real-world applications:

• Cooperative detection of vehicles, pedestrians, and other agents that a single car cannot see due to occlusion.
• Intersection safety, where roadside units provide an overhead view that complements vehicle sensors.
• Perception under sensor-range limits, such as highway scenarios where distant hazards are outside any single vehicle's field of view.
• Occupancy and drivable-area prediction for planning, using the dataset's semantic occupancy voxel grids and BEV segmentation maps.

Industry relevance: Companies developing autonomous driving and intelligent transportation infrastructure need training and validation data at a scale that real-world collection cannot economically supply. A configurable generator plus a large pre-built dataset lowers the barrier to prototyping cooperative perception stacks, and the released baseline and architecture give teams a starting point for comparison.

Future Directions

• Closing the simulation-to-reality gap: The abstract does not address whether models trained on synthetic SimBEV2X data transfer to real-world sensor data, which is the central open question for any simulator-derived dataset.
• Benchmarking across tasks: The paper establishes a detection baseline; the dataset's segmentation maps, occupancy grids, and HD maps suggest multi-task and joint-training benchmarks are a natural next step.
• Scaling simulation diversity: Whether the randomization scheme covers enough road networks, weather, lighting, and traffic behavior conditions to generalize is not answered in the abstract.
• Architecture development: CoBEVFusion is presented as a starting point for context-aware multi-agent fusion, leaving room for further work on how agents should weight and aggregate each other's features, including under communication constraints.

Target Audience

Researchers and engineers working on cooperative or multi-agent perception, BEV-based autonomous driving models, and V2X systems; developers of autonomous driving simulation and synthetic data pipelines; and graduate students or practitioners who need a large, multi-task dataset with detection, segmentation, and occupancy labels for training and benchmarking. Readers primarily interested in real-world dataset collection methodology will find the paper's premise — that synthetic generation replaces that process — is the point being argued rather than a method being demonstrated in the abstract.

Authors’ abstract

Cooperative perception through vehicle-to-everything (V2X) communication can overcome the inherent physical limitations of individual autonomous vehicles, such as occlusions and limited sensor range. However, the development of robust V2X algorithms, particularly those relying on unified spatial representations like bird's-eye view (BEV) representation, is hampered by the lack of large-scale, multi-modal, multi-task datasets. Moreover, collecting and annotating a large set of synchronized, real-world multi-agent data is prohibitively expensive. This has resulted in a landscape where existing V2X datasets are notably limited in both size and scope. To overcome this, we introduce SimBEV2X, an advanced synthetic data generation tool built on the CARLA simulator. SimBEV2X automatically creates randomized driving scenarios to collect multi-modal sensor data alongside various types of ground truth including 3D bounding boxes with unique track IDs, HD map information, BEV segmentation maps, and semantic occupancy voxel grids from both vehicles and RSUs. We also present the SimBEV2X dataset, the largest V2X perception dataset to date. The dataset comprises 258 scenes, each involving up to 8 connected vehicles and up to 4 RSUs across a variety of road networks. The SimBEV2X dataset is an order of magnitude larger than existing V2X datasets and contains 102,200 frames, 588,520 lidar point clouds, more than 3 million images, over 27 million bounding boxes, and a comprehensive set of other annotations. Finally, we establish a strong baseline on the SimBEV2X dataset using CoopDet3D and propose CoBEVFusion, a novel architecture that combines CoopDet3D with fused axial attention (FAX) for context-aware multi-agent feature aggregation, resulting in superior performance. SimBEV2X, the SimBEV2X dataset, and CoBEVFusion are available at https://simbev2x.org and https://github.com/GoodarzMehr/SimBEV2X.