DrivIng: A Large-Scale Multimodal Driving Dataset with Full Digital Twin Integration

AI summary

Overview

Research area: Autonomous driving perception — large-scale multimodal datasets, 3D object detection benchmarking, and real-to-sim digital twins.

Technical level: Intermediate. The paper is readable without deep specialization, but familiarity with 3D detection benchmarks (nuScenes, KITTI, MMDetection3D) and sensor terminology helps.

Scope: This paper introduces DrivIng, a multimodal driving dataset recorded over an approximately 18 km route with 6 RGB cameras, 1 LiDAR, and high-precision ADMA localization, paired with a fully geo-referenced CARLA digital twin and baseline 3D detection benchmarks.

What This Paper Is About

Most large driving datasets for autonomous driving consist of many short, disconnected sequences and have no high-fidelity digital replica of the environment they were recorded in. This makes it impossible to replay a real traffic scene in simulation or test how a perception model would behave under modified conditions. The authors build DrivIng to close that gap: a continuous real-world dataset whose entire recorded route has a matching geo-referenced digital twin, so real traffic can be transferred into simulation one-to-one while preserving agent interactions.

Key Contributions

1. A continuous real-world multimodal dataset: Three continuous, uninterrupted sequences along an approximately 18 km route (unique track length approximately 16 km) covering highway, suburban, urban roads, and construction zones, recorded with six RGB cameras providing 360° coverage, one roof-mounted LiDAR, and an ADMA-based localization system under Day, Dusk, and Night conditions.
2. High-frequency annotations: 3D bounding boxes with track IDs annotated at 10 Hz across 12 object classes, yielding approximately 1.2 million annotated instances over more than 63k frames.
3. A fully integrated digital twin and validation testbed: A geo-referenced CARLA reconstruction of the whole route, anchored by an HD map and enriched with over 1.2k hand-crafted buildings, more than 10k traffic signs, and over 20k additional environmental objects, supporting two reconstruction modes: high-fidelity kinematic replay and live interactive re-simulation.
4. Benchmark evaluations and public release: Baselines for a camera-only model (PETR) and a LiDAR-only model (CenterPoint) implemented in MMDetection3D, plus a nuScenes-format converter, codebase, dataset, and digital twin released publicly.

Main Findings

• Dataset scale and composition: DrivIng contains over 63k annotated frames, corresponding to about 378k RGB images and 63k LiDAR frames. The Day sequence has 23,092 frames (approximately 38.5 min), Dusk has 20,246 frames (approximately 33.7 min), and Night has 19,705 frames (approximately 32.8 min).
• Annotation distribution: Of the roughly 1.2 million annotated objects, about 560k are in Day, 336k in Dusk, and 268k in Night. Cars are the most frequently annotated class; Animals and OtherPedestrian appear least often. The relative class distributions are consistent across the three sequences.
• Object density per frame: The Night sequence averages 12.8 objects per frame, Dusk 15.0, and Day 20.6, with numerous frames across all sequences containing more than 50 objects.
• Distance coverage: Roughly 60% of all annotations lie within the first 50 m, approximately 90% within 100 m, and the dataset still contains tens of thousands of annotations beyond 100 m.
• Object orientation diversity: Most objects, particularly vehicles, are oriented along the primary cardinal directions, but the dataset includes a substantial number of objects observed at non-typical traffic angles.
• Track and size statistics: Bus, Truck, and Trailer show the longest average uninterrupted track lengths and contain the highest average number of LiDAR points within their 3D bounding boxes. Track ID counts range from 8,502 for Car down to 5 for OtherPedestrian, 6 for Animal, and 25 for E-Scooter. OtherPedestrian has no annotations in the Night sequence and Animal has none in the Dusk sequence.
• Daytime detection performance (nuScenes metrics): PETR achieves ATE 0.70, ASE 0.19, AOE 0.27, AVE 1.54, NDS 40.4, mAP 35.9. CenterPoint achieves ATE 0.20, ASE 0.16, AOE 0.16, AVE 3.72, NDS 71.7, mAP 79.3.
• Dusk detection performance: PETR achieves ATE 0.73, ASE 0.28, AOE 0.36, AVE 2.46, NDS 38.0, mAP 35.7. CenterPoint achieves ATE 0.16, ASE 0.17, AOE 0.27, AVE 3.22, NDS 66.9, mAP 72.5.
• Night detection performance: PETR achieves ATE 0.99, ASE 0.28, AOE 0.41, AVE 2.16, NDS 27.2, mAP 22.4. CenterPoint achieves ATE 0.23, ASE 0.17, AOE 0.22, AVE 3.87, NDS 63.1, mAP 66.1.
• Camera versus LiDAR gap: The LiDAR-based CenterPoint consistently achieves at least twice the mAP of the camera-based PETR, and nearly three times the value at night. PETR shows lower AVE, indicating better velocity estimates.
• Per-class behavior: CenterPoint outperforms PETR on small classes such as Bicycle and Pedestrian. PETR struggles with large articulated objects such as trailers — for example, its trailer AP reaches only 10.6 in Day, 17.8 in Dusk, and 0.0 in Night — due to localization and orientation errors, sparse or partially visible structures, and high appearance variability.
• Day-to-night degradation: Both models degrade from day to night. The authors attribute CenterPoint's degradation partly to fewer nearby objects in the sequences, and PETR's to low illumination and weaker geometric cues.
• Digital twin fidelity: Using only timestamps and 3D annotations, the digital twin reconstructs scenes by placing surrogate vehicle models at the same global coordinates as their real-world counterparts, producing exact correspondences for both dynamic and static objects. In kinematic replay mode, any positional discrepancy arises solely from CARLA's transform precision and reflects simulator limitations rather than trajectory reconstruction error.
• Stated limitation: In both digital twin modes, the visual fidelity of agents is constrained by the finite set of vehicle models provided by the simulator.

Methodology in Plain English

Data was collected with an Audi Q8 e-tron carrying 6 RGB cameras, 1 LiDAR, and 1 GPS/IMU module. Four cameras have a 60° horizontal field of view and two have 100°, together giving full 360° coverage at 1920×1080 and 20 FPS. The LiDAR is a Robosense Ruby Plus with 128 rays, 360° horizontal FOV, −25° to 15° vertical FOV, 20 FPS, and up to 240 m range at 10% or higher reflectivity. Localization comes from a Genesys ADMA Pro+ at 100 FPS with RTK correction and 1 cm precise positioning. Sensor calibration and synchronization followed the procedures described in UrbanIng-V2X.

Human annotators labeled all objects in the LiDAR point clouds at 10 Hz with 3D bounding boxes, spatial coordinates (x, y, z), yaw orientation, and unique tracking IDs. Independent reviewers visually inspected both point clouds and images across multiple rounds. Faces and license plates in RGB images were anonymized with Gaussian blurring.

For the digital twin, the authors reconstructed the 6 × 4 km² collection area in CARLA, anchored by an HD map linking simulation to global coordinates, built from independent geo-referenced recordings. The twin supports two modes. In kinematic replay, the scene is reset every frame and each agent is spawned directly at its recorded pose using a surrogate model matched by class and dimensions, entirely bypassing CARLA's physics engine, with synchronous mode set to Δt = 100 ms. In interactive re-simulation, recorded trajectories serve as global reference paths that CARLA's built-in autopilot can follow while managing local interactions, enabling physics-based simulation for planning and control evaluation.

For benchmarking, the data was converted to the nuScenes format and run through the MMDetection3D pipeline. The 12 original categories were merged into 9 nuScenes classes; Animal was then excluded due to underrepresentation, leaving 8 classes for detection. Each full sequence was split into 50 sub-sequences, with 80% used for training, 10% for validation, and 10% for testing, so every partition covers highway, suburban, and urban scenes. PETR (camera-only) uses a pre-trained FCOS3D (V-99-eSE) backbone with images resized to 384 × 960 pixels and is trained for 75 epochs. CenterPoint (LiDAR-only) operates on a single sweep projected onto a 100 m × 100 m BEV grid with voxel size (0.1, 0.1, 0.2) m and is trained for 25 epochs. Evaluation follows the nuScenes protocol, reporting ATE, ASE, AOE, AVE, distance-aware mAP over 0.5 m, 1 m, 2 m, and 4 m thresholds, and NDS; AAE was excluded because the dataset's object attributes differ substantially from nuScenes. Evaluation ranges were set to [−54 m, 54 m] along both axes. Training ran on six NVIDIA L40S GPUs with an Intel Xeon Platinum 8480+ processor with 224 cores; for reference, training CenterPoint for 25 epochs on the Day sequence with a batch size of 4 per GPU requires approximately 18 hours.

Why This Matters

Impact on research: DrivIng addresses a structural gap in the autonomous driving literature. Most established datasets (KITTI, nuScenes, Waymo Open Dataset) provide many short independent sequences and no digital twin, while datasets that do offer digital twins — TWICE, CitySim, OPV2V, UrbanIng-V2X — are limited to closed test tracks, drone viewpoints, or compact urban areas (0.64 km² to 4 km²), and often lack ego-vehicle multi-sensor recordings. DrivIng reports 24 km² of total area, an approximately 18 km drivable track, and more than 31k assets, with annotated and sign geo-referencing both present. This combination enables reproducible, systematic benchmarking that existing datasets cannot support.

Real-world applications:

• Sim-to-real evaluation: Models can be trained or tested in the digital twin and validated against the paired real-world recordings, with kinematic replay allowing a direct comparison between real and simulated sensor data.
• Edge-case and safety-critical testing: Environmental conditions and scenario parameters can be modified in the twin to systematically probe perception failures that are rare or dangerous to reproduce on public roads.
• Cooperative perception research: Multiple agents can share and fuse sensor data in simulation to mitigate occlusions, a scenario the authors note is often prohibitively expensive or logistically challenging to reproduce in reality.
• Perception model validation and robustness studies: The three Day, Dusk, and Night sequences provide a controlled axis for measuring illumination-related degradation, as demonstrated by the reported performance drops.

Industry relevance: The nuScenes-format converter, MMDetection3D integration, and publicly released codebase and HD map lower the barrier for teams already using standard detection pipelines to adopt DrivIng. The dataset provides a ready testbed for evaluating whether perception stacks hold up under night driving and long-range conditions — precisely the regimes where the reported baselines show the largest drops.

Future Directions

• Improving agent visual fidelity in the twin: The authors explicitly note that agent appearance is constrained by the simulator's finite vehicle model library, leaving open the question of how much this limits realistic sensor-level simulation.
• Extending beyond 3D object detection: The paper states that the annotations support object tracking, trajectory prediction, and localization, but only 3D detection is benchmarked here — the other tasks remain to be evaluated.
• Expanding benchmarks to more model families and sensors: Only two models (PETR for camera, CenterPoint for LiDAR) are benchmarked, leaving fusion-based and other architectures untested on this data.
• Scaling the digital twin geographically: DrivIng covers one approximately 18 km route in three lighting conditions; whether the same reconstruction methodology generalizes to other cities, weather, or longer routes is not established.

Target Audience

Researchers and engineers working on autonomous driving perception, 3D object detection, and sensor fusion who need a dataset with a matched simulation environment. It is also well suited to groups working on sim-to-real transfer, scenario-based safety validation, and cooperative or multi-agent perception, as well as practitioners who want a nuScenes-compatible benchmark with paired real and simulated data. Readers primarily interested in end-to-end planning or control will find the dataset relevant, though only detection is benchmarked in this paper.

Authors’ abstract

Perception is a cornerstone of autonomous driving, enabling vehicles to understand their surroundings and make safe, reliable decisions. Developing robust perception algorithms requires large-scale, high-quality datasets that cover diverse driving conditions and support thorough evaluation. Existing datasets often lack a high-fidelity digital twin, limiting systematic testing, edge-case simulation, sensor modification, and sim-to-real evaluations. To address this gap, we present DrivIng, a large-scale multimodal dataset with a complete geo-referenced digital twin of a ~18 km route spanning urban, suburban, and highway segments. Our dataset provides continuous recordings from six RGB cameras, one LiDAR, and high-precision ADMA-based localization, captured across day, dusk, and night. All sequences are annotated at 10 Hz with 3D bounding boxes and track IDs across 12 classes, yielding ~1.2 million annotated instances. Alongside the benefits of a digital twin, DrivIng enables a 1-to-1 transfer of real traffic into simulation, preserving agent interactions while enabling realistic and flexible scenario testing. To support reproducible research and robust validation, we benchmark DrivIng with state-of-the-art perception models and publicly release the dataset, digital twin, HD map, and codebase.