A Multi-Vehicle Dataset with Camera, LiDAR, and Radar Sensors and Scanned 3D Models for Custom Auto-Annotation using RTK-GNSS

AI summary

Overview

Research area: Autonomous driving perception, sensor data collection, and benchmarking — specifically dataset construction for camera, LiDAR, and radar.

Technical level: Advanced. The paper assumes familiarity with sensor calibration, GNSS/INS reference frames, rigid-body transformations, radar Doppler measurements, and ROS-based data pipelines.

Scope in one sentence: The authors present 7V-Scanario, a multi-vehicle recording dataset in which every participating vehicle carries an RTK-GNSS/INS unit and has been scanned as a textured 3D model, so that ground truth for any sensor measurement can be generated automatically at user-chosen granularity.

What This Paper Is About

Existing autonomous driving datasets typically label objects with 2D/3D bounding boxes, semantic masks, or point-wise instance labels — representations that lose the precise shape, surface orientation, and exact motion of the observed vehicles. The authors instead record seven vehicles with a full camera/LiDAR/radar sensor suite while tracking each vehicle with centimeter-accurate RTK-GNSS, and additionally produce high-resolution 3D scans of each vehicle. The goal is a dataset where the complete dynamic state and geometry of every object is known at any instant, so that annotations of any format and detail level can be computed on demand rather than shipped as a fixed label set.

Key Contributions

1. A multi-sensor recording setup with seven instrumented target vehicles. One sensor vehicle (MuCAR-3, a VW Touareg) carries a 128-beam LiDAR, four RGB cameras, five medium-range 79 GHz radars, and one far-range 77 GHz radar. Every target vehicle carries its own RTK-GNSS/INS unit, so relative poses are measured directly rather than inferred from sensor data.

2. Textured 3D scans of every vehicle, aligned to its GNSS/INS coordinate frame. Each vehicle was scanned with an Artec Leo handheld scanner and post-processed into a globally registered model. Known features (screws and marker on the INS unit, axles, emblems) are used to solve for the rigid transformation between the scan frame and the INS frame, producing a model that projects accurately into all sensor coordinate systems.

3. A continuous, high-frequency kinematic reference for all objects, not just the ego vehicle. With RTK-GNSS/INS units reporting at 100 Hz with roughly 1 cm positional and 0.03° angular accuracy, ground truth for any sensor timestamp can be obtained by interpolation. This makes the dataset usable as reference for object tracking, odometry, and SLAM evaluation, not only for detection.

4. An "auto-annotation" paradigm instead of fixed labels. Because shape, texture, and pose of every object are known, ray casting against the 3D models can generate pixel labels, point labels, or range images in any desired granularity — for example distinguishing a wheel from a window, rather than just labeling "car." The models additionally support data augmentation such as texture swapping and synthetic relighting, and can be reused directly as simulation assets.

Main Findings

• Projection accuracy is high enough to be visually indistinguishable from reality. Rendering the aligned 3D models into camera images reproduces the original vehicles with barely recognizable projection error; overall accuracy after accounting for scan error, model alignment, sensor calibration, and relative GNSS/INS pose is on the order of a few centimeters.

• Radar measurement behavior can be analyzed spatially and quantitatively. Accumulating radar detections in the target's coordinate frame produces heatmaps that reveal which parts of a vehicle are strongly reflective. Without normalization, these heatmaps are biased by how often a given viewing geometry occurred, so the authors apply an inverse-frequency weighting scheme.

• View-dependent reflection patterns become visible. By filtering heatmaps to a specific sensor position (for example, rear-left), hotspots can be attributed to specific real structures such as corners, the roof-mounted sensor rig, and wheels — and the angular measurement noise of the radar becomes directly apparent.

• Known geometry exposes otherwise hidden sensor effects. LiDAR points were measured on the far side of a vehicle's B-pillar, showing laser penetration through the window glass. The StreetScooter's plastic transport box produces clutter detections from internal multiple reflections far beyond the box itself.

• Occlusion can be studied exactly, not approximately. Because the true geometry of the occluder is known, the authors can identify precisely which measurements should be missing and how partial occlusion affects Doppler and position estimates.

• Instrumentation has a measurable but bounded cost. Roof racks carrying GNSS antennas and INS units perturb the measurements, but the scans were taken with the racks mounted, so the models match what the sensors actually see.

Methodology in Plain English

The authors started by giving every target vehicle a way to know exactly where it is. Prototype vehicles already had GNSS/INS units; ordinary passenger cars received custom aluminum roof racks holding an OxTS RT3000v3-class INS, two multi-band GNSS antennas, a radio modem for RTK corrections from a nearby base station, a WiFi link, and power electronics. A local reference station within roughly one kilometer provides the correction data, giving centimeter-level positioning under open-sky conditions, and long warm-up maneuvers were performed before recording.

All vehicle states were synchronized and shared: the sensor vehicle acts as a WiFi access point and receives the INS streams from all targets, while each unit also dumps its raw data internally as a fallback. Hardware timestamping, hardware triggering, and bus-latency compensation put every sensor on a common clock. Sensor mounting poses were determined by a probabilistic joint multi-sensor calibration.

Separately, each vehicle was scanned with a handheld 3D scanner, using minimal texture spray on large homogeneous surfaces. The raw scan has an arbitrary coordinate orientation, so the authors manually selected identifiable features whose positions are documented — screws and markers on the INS unit, or axles and emblems on the vehicle — and formulated a least-squares problem over three translation and three rotation parameters to minimize residuals between measured and expected feature coordinates. The resulting transformation maps any scan point into the vehicle's INS frame, from which it can be projected into any sensor frame using the calibrated mounting pose and the interpolated relative vehicle pose at that timestamp.

Rather than publishing ready-made annotations, the authors leave annotation generation to the user, suggesting ray-casting libraries that can render the 3D models into whatever label format is needed.

Why This Matters

This dataset shifts the benchmark question from "does the algorithm detect the object?" to "does the algorithm respond correctly to the underlying physics?" By supplying exact geometry, surface normals, texture, and continuous kinematics for every object, it lets researchers attribute measurement behavior to specific causes instead of treating sensor output as an opaque label distribution. It also narrows the sim-to-real gap by providing real measurement data alongside the corresponding real 3D assets, which can be dropped into a simulator.

Real-world applications:

• Perception model training and validation. Generating pixel- and point-level labels at custom granularity without manual annotation labor, including fine classes such as wheels, windows, and mirrors.
• Radar and LiDAR sensor development. Characterizing reflectivity, elevation uncertainty, micro-Doppler from rotating wheels, glass penetration, and multi-path clutter against known ground truth geometry.
• Multi-object tracking and SLAM evaluation. Using the continuous GNSS/INS reference for all vehicles as a kinematic ground truth that is otherwise absent from camera and LiDAR datasets.
• Simulation and scenario generation. Reusing the scanned, textured vehicle models to build realistic synthetic scenes and to test occlusion or sensor-degradation effects.

Industry relevance: Companies developing ADAS and automated driving functions, radar and LiDAR suppliers tuning detection and tracking algorithms, and simulation platform vendors all depend on precisely labeled data and validated sensor models. A dataset that couples real measurements with exact object models and kinematics provides a stronger basis for safety-relevant validation than bounding-box annotations alone.

Future Directions

• Reduce the measurement footprint of the instrumentation. The authors note that hidden GNSS antennas and internally mounted IMUs would avoid the roof-rack structures that currently perturb sensor data, at the cost of harder alignment.
• Add articulated geometry. The current models are rigid, so wheels cannot rotate; adding joints would allow micro-Doppler effects from spinning wheels to be simulated and matched against real measurements.
• Build out the annotation tooling and formats. Since no ready-to-use labels are shipped, practical adoption depends on widely available ray-casting pipelines and agreed-upon output formats for camera, LiDAR, and radar.
• Expand beyond the seven vehicles. The current set covers seven specific cars and vans; extending to more vehicle types, and to vulnerable road users such as pedestrians and cyclists, would broaden the applicability of the auto-annotation concept.

Target Audience

This paper is most valuable to perception and sensor researchers working on radar, LiDAR, or camera measurement modeling; engineers building or validating object tracking and sensor fusion systems; dataset and benchmark designers interested in auto-annotation pipelines; and simulation engineers who need realistic, textured vehicle models paired with real measurement data. Readers without a background in sensor calibration or GNSS/INS reference frames will find the technical sections demanding but the dataset description and evaluation examples accessible.

Authors’ abstract

Datasets are a crucial element in the development of perception algorithms. They relate sensor measurement data to annotated reference information and allow for the deduction of sensor and object characteristics. In autonomous driving, the reference data commonly consist of semantic image segmentation, point-wise associations, or bounding box annotations. The dataset proposed in this work, however, aims to dig deeper into the evaluation of measurement principles and provides scanned 3D models of all vehicles together with a pose and continuous kinematics reference obtained by RTK-GNSS. Combined, the state of the complete dynamic surrounding of the sensor vehicle is known for any point in time. Subsequent reference formats can be easily computed in user-defined granularity. This dataset involves single-object and multi-object recordings with seven target vehicles. In particular, measurement effects such as occlusion, as well as reflections, can be evaluated, as the normals of the shape of the target vehicles are known. We describe the dataset, discuss the technical background of its development, and briefly present exemplary evaluations.