A2RL V\textsubscript{max}: The A2RL autonomous racing dataset for long-range, high-speed perception and multi-vehicle interaction

AI summary

Overview

Research area: Autonomous driving perception — specifically LiDAR/RADAR-based 3D object detection and multi-object tracking for high-speed autonomous racing.

Technical level: Intermediate. Readers should be familiar with basic 3D perception concepts (point clouds, bounding boxes, detection and tracking metrics), but the paper is written to be usable by practitioners applying off-the-shelf models.

Scope: The paper introduces and benchmarks A2RL Vmax, a large-scale, professionally annotated LiDAR and RADAR perception dataset captured during the 2024 Abu Dhabi Autonomous Racing League (A2RL) at the Yas Marina F1 Circuit, covering single-vehicle, multi-vehicle, and full four-vehicle race scenarios at speeds exceeding 200 km/h.

What This Paper Is About

Most autonomous driving perception datasets (KITTI, Waymo Open Perception, nuScenes, Argoverse 2) focus on well-structured urban environments at inner-city and highway speeds. The authors argue that rare, high-speed, high-risk situations are exactly where autonomous systems fail, yet almost no public dataset captures them with professionally annotated 3D data. The goal of this work is to release a large, open, deep-learning-ready dataset from a real F1-circuit race between full-scale autonomous racecars, and to establish baseline detection and tracking results on it.

Key Contributions

1. The A2RL Vmax dataset: 28,791 professionally annotated LiDAR point clouds (drawn from 305,531 recorded point clouds) with 38,545 vehicle bounding boxes across 297 scenes and 466 opponent tracks, plus 386,006 RADAR point clouds, contributed by all eight teams that competed in A2RL 2024. The authors state it is the first large-scale dataset in autonomous racing to feature professionally annotated LiDAR point clouds.
2. A reproducible benchmark for 3D object detection and tracking as a reference point for assessing the dataset, using the nuScenes evaluation toolkit with an extension that reports metrics by relative distance range (0–80 m, 80–130 m, beyond 130 m).
3. Implementation and evaluation of off-the-shelf baseline methods for detection (SECOND, PointPillars, PV-RCNN, PV-RCNN++, CenterPoint pillar and voxel, DSVT pillar and voxel, Voxel-Mamba, VoxelNeXt, LION) and for tracking (AB3DMOT, following the nuScenes benchmark).
4. Open-source access to the dataset, baseline source code, auxiliary material, and a dataset website, with the stated plan to further enrich the dataset through additional A2RL competitions and teams.

Main Findings

• Close-range detection is strong; long-range detection is weak. All detectors achieve similar Average Precision (AP) in the 0–80 m range, roughly matching the car-class performance of their original publications — SECOND about 0.7 AP, and more recent methods around 0.8 AP. However, in the 80–130 m and beyond-130 m ranges these improvements largely disappear, with little to no advancement over older methods.
• Annotation distances require long-range perception. 53.5% of annotated data lies within 80 m and 78.4% within 130 m, meaning a substantial portion extends beyond 80 m.
• Point density collapses with distance. The paper reports a steep decline in LiDAR points per annotated bounding box up to approximately 80 m; beyond that range most targets contain fewer than 100 points, and targets at 200 m receive as few as 10 points. The paper also notes a drop in point density between 15 and 25 m due to blind spots in close proximity.
• Latency: only one baseline meets the real-time criterion. The authors define real-time capability as 30 ms or less of processing time. SECOND is the only method that qualifies, at 27.7 ± 0.3 ms. PointPillars (52.2 ± 0.5 ms), CenterPoint pillar (34.2 ± 0.5 ms), CenterPoint voxel (33.6 ± 0.2 ms), and VoxelNeXt (36.3 ± 0.4 ms) come close; PV-RCNN (185.8 ± 0.3 ms), PV-RCNN++ (69.9 ± 0.3 ms), DSVT pillar (73.6 ± 0.1 ms), DSVT voxel (75.6 ± 0.2 ms), Voxel-Mamba (86.2 ± 0.2 ms), and LION (116.2 ± 0.2 ms) do not.
• Orientation error grows with distance. All methods achieve a similar Average Orientation Error (AOE) of approximately 0.18 radians (about 10 degrees) at close range, increasing to roughly 0.27 radians (16 degrees) at greater distances. The authors do not consider this critical because orientation matters most in close proximity, such as during head-to-head interactions.
• Localization of detected objects is accurate and stable across range. Average Translation Error (ATE) is about 0.1 m for all methods across all distance ranges, which the authors attribute to all ground-truth objects having identical dimensions.
• A large domain gap exists between urban driving and racing. VoxelNeXt detectors trained on nuScenes and Argoverse achieved near-zero AP on A2RL Vmax.
• Tracking is surprisingly strong with perfect detections. Using perfect detections as an upper bound to isolate the tracker, AB3DMOT with the default 2 m matching distance and 300 m max range achieves AMOTA 0.814, AMOTP 0.357, 260 identity switches (IDS), 336 fragmentations (FRAG), 2178 false positives, and 3732 false negatives, with 0.3 ± 0.4 ms latency.
• Adjusting the matching distance improves tracking substantially. Increasing the maximum track-to-detection matching distance from 2 m to 8 m (because racecars travel further between frames) raises AMOTA to 0.891 at 300 m, with AMOTP 0.303, 83 IDS, 291 FRAG, 1432 FP, and 2559 FN.
• Longer range disproportionately hurts tracking. Restricting to 130 m yields AMOTA 0.897, AMOTP 0.292, 61 IDS, 177 FRAG, 910 FP, 1749 FN. Restricting to 80 m yields AMOTA 0.929, AMOTP 0.223, 32 IDS, 81 FRAG, 455 FP, 952 FN. The authors note IDS and FRAG almost double when extending from 80 m to 130 m, even though the volume of data increases only by 20%.
• Few vehicles need tracking per frame. A single racecar is present in 18,137 frames, two racecars in 8,086 frames, and three opponent racecars in 1,412 frames. The authors attribute this to large performance gaps between teams, which simplifies data association but makes state estimation harder because differential speeds are far greater than in regular vehicle tracking.
• Tracking failure concentrates in corners. Tracking is relatively simple on straight segments but becomes considerably harder in high-speed corners, where rapid ego-vehicle yaw changes induce large apparent motion in tracked objects — especially far ones — causing the AB3DMOT Kalman filter to produce identity switches and fragmentation. Tracker runtime is under one millisecond per frame.

Methodology in Plain English

The authors did not run an experiment in the traditional sense; they organized the collection and release of a dataset and then stress-tested existing algorithms on it.

1. Data capture in a live competition. All eight teams competing in A2RL 2024 shared sensor data recorded on the same Yas Marina F1 Circuit. Competition sessions included a single-vehicle time trial, a two-vehicle attack-and-defend head-to-head format, and an 8-lap four-vehicle race. The event was preceded by a two-month development and practice period involving more than 100 participants.
2. Standardized racecar platform with heterogeneous team configurations. The car is built on a 2023 Dallara Super Formula chassis with a turbocharged 2.0-liter four-cylinder engine producing approximately 550 horsepower, and computes on an AMD EPYC 7313P CPU with an NVIDIA RTX 6000 ADA GPU. Sensors include three Seyond Falcon K LiDARs (front, rear-left, rear-right), four ZF ProWave 4D-RADAR units, seven Leopard Imaging cameras, a dual-antenna Vectornav VN-310 INS, and a Kistler SF Motion ground-speed sensor. Teams were free to use subsets of sensors and configure them differently, so the data is non-homogeneous across teams — Table I lists which sensors and at which frequencies each of the eight teams contributed.
3. Formatting for reuse. Data is provided in the nuScenes format so the nuScenes devkit and libraries such as OpenPCDet work with minimal modification. Frames are grouped into logs per team and session; only scenes where opponent vehicles are visible are exported, and scenes end when other vehicles leave view (shortened to under 10 seconds). Labels use a single class, car, are produced on LiDAR point clouds, and do not include velocity (unlike nuScenes). Each frame is a single combined point cloud from all three LiDARs, in a right-hand coordinate system originating at the midpoint of the rear axle.
4. Benchmarking detectors. Well-known 3D detectors were trained three times each on the dataset with minor parameter adjustments and no extensive tuning. Evaluation follows the nuScenes detection task using AP, AOE, and ATE for the car class, extended by the authors to report metrics separately for 0–80 m, 80–130 m, and beyond 130 m. Latency was measured on the same GPU used in the racecar, with batch size 1, one warm-up pass, and five iterations over the validation split; published implementations were used without TensorRT or similar optimization.
5. Benchmarking tracking. Tracking follows the nuScenes tracking-by-detection setup using AB3DMOT. To isolate the tracker from detection errors, the authors feed it perfect detections as an upper bound. They switch the distance metric to Euclidean distance between 3D boxes (because not all teams provided camera calibration) and raise the maximum matching distance from 2 m to 8 m. Because the ground truth lacks velocity, they set the Kalman filter's predicted velocity to 0 during evaluation.

Why This Matters

Impact on research. The paper identifies long-range 3D detection as an understudied problem: existing detectors are trained for 50–80 m ranges, yet modern LiDAR hardware can return points up to 500 m, and the state of the art on the longer-range Argoverse 2 benchmark is characterized by the authors as an mAP of just 0.4 for LION. The prior high-speed dataset, RACECAR, lacks professionally annotated data and limits vehicle interaction to two vehicles under predefined rules. A2RL Vmax addresses both gaps with professionally labeled LiDAR and genuine multi-vehicle racing scenarios. Its nuScenes-compatible format also lowers the barrier to entry, letting researchers reuse existing code, models, and evaluation tooling.

Real-world applications:

• Highway and high-speed automated driving, where detecting vehicles beyond 80–130 m at low latency is safety-critical for braking and overtaking decisions.
• Long-range perception system design, informing sensor placement, point-cloud aggregation strategy, and whether sparse-convolution or linear-RNN backbones are viable under real-time budgets.
• Robust multi-object tracking under high differential speeds, applicable to highway traffic where relative motion and latency between detections cause large positional shifts.
• Simulation and validation of safety-critical systems, using the race as a controlled but extreme stress test for the full perception-to-control stack.
• Localization research, since the track deliberately exposes GNSS failure modes in the hotel section (turns 13 and 14) and under multiple bridges.

Industry relevance. The dataset is contributed by all eight competing A2RL teams with institutional participation spanning Karlsruhe Institute of Technology, Technical University of Munich, University of Modena and Reggio Emilia, Széchenyi István University, Politecnico di Milano, Constructor University, Beijing Institute of Technology, Nanyang Technological University, and Code 19 Racing. The paper frames competition-driven data collection as a paradigm in which competitive benchmarking and open scientific dissemination are intentionally integrated, and explicitly positions the dataset as a system-robustness benchmark under racing-grade constraints.

Future Directions

• Motion forecasting. Because the dataset annotates the entire race, the authors plan to integrate motion forecasting in future dataset iterations, which they consider particularly crucial for head-to-head racing where predicting an opponent's maneuvers enables more responsive, strategic decision-making.
• High-precision ground-truth positioning. GNSS signals are frequently disrupted by bridges and tunnels on this track, so the authors plan to enhance the dataset with high-precision ground-truth positioning to support more robust localization research for high-speed autonomous driving.
• Dataset expansion. The authors state they intend to further enrich the dataset through more A2RL competitions with more teams.
• Specialized long-range methods. The consistent collapse of detection and tracking performance beyond 80–130 m, the near-zero AP of urban-trained detectors when transferred, and the failure of current trackers in high-speed corners all point to a need for methods designed specifically for long-range, low-latency, high-differential-speed operation rather than adapted urban drivers.

Target Audience

Researchers and engineers working on 3D perception for autonomous driving, particularly those interested in long-range LiDAR detection, multi-object tracking under extreme dynamics, and domain transfer from urban to high-speed settings. It is also directly relevant to autonomous racing teams and competition organizers seeking a benchmarking and stress-testing resource, and to practitioners who want a nuScenes-compatible dataset they can plug into existing detection and tracking pipelines without rewriting tooling. Readers with no background in point-cloud perception will find the baseline tables dense, but the dataset description and challenge analysis are accessible.

Authors’ abstract

In autonomous driving development, a perception dataset is crucial, as it provides fundamental data for training, testing, and validating algorithms for an autonomous vehicle's multimodal perception systems. So far, most research has concentrated on providing datasets for well-structured urban environments. This work introduces the A2RL V\textsubscript{max} open-source dataset, specifically designed for perception tasks in high-speed autonomous driving and multi-vehicle interaction. The dataset was captured during the 2024 Abu Dhabi Autonomous Racing League (A2RL), held at the Yas Marina F1 Circuit, with participation from all competing teams. It contains diverse scenarios, including single-vehicle data at varying speeds, multi-vehicle sessions, and the full final four-vehicle race. The dataset contains almost 30,000 professionally annotated LiDAR point clouds, along with RADAR point clouds. In particular, it is the first large-scale dataset in autonomous racing to feature professionally annotated LiDAR point clouds, enabling deep learning-based perception research. The data is provided in a developer-friendly format, enabling easy implementation and evaluation in future research. We provide implementation and evaluation for off-the-shelf 3D detection and tracking methods. Although baseline methods show promising results for both 3D detection and tracking, specialized methods are required to address the unique challenges of high-speed autonomous driving. For a detailed description of the dataset, please visit the \href{https://tum-avs.github.io/A2RL_Dataset_website/}{A2RL V\textsubscript{max} Dataset Website}