A Multi-Sensor Dataset for Monitoring the Operational Environment of Rail Vehicles

AI summary

Authors: Claudio Diotallevi, Rodrigo Gudiño, Zaharia Pachalieva, Philipp Neumaier, Patrick Naumann, Erik Bochinski, Volker Eiselein, Martin Köppel Affiliations: understandAI GmbH (Karlsruhe, Germany) and DB InfraGO AG (Berlin, Germany) arXiv: 2608.04704v1 [cs.CV], 05 Aug 2026

Overview

• Research area: Computer vision and multi-sensor perception for railway automation (autonomous train environment monitoring).
• Technical level: Intermediate. The paper is largely a dataset description paper; it assumes familiarity with annotation types (2D/3D bounding boxes, polygons, polylines, segmentation) and sensor modalities (RGB, infrared, LiDAR, radar), but does not require deep machine-learning background.
• Scope (one sentence): The paper describes a multi-sensor railway perception dataset containing 7,052,055 annotations across 21 object classes, collected with two instrumented rail vehicles and released on request by DB InfraGO AG.

What This Paper Is About

AI-based perception systems for automated trains need large amounts of accurately annotated sensor data to learn how to detect, classify and react to hazards in real time. Existing public railway computer-vision datasets are mostly camera-only and relatively small, so the authors set out to build a large, multi-sensor, railway-specific annotated dataset covering both railway infrastructure and general perception objects. The result is a dataset produced jointly by DB InfraGO AG and understandAI GmbH within the Digitale Schiene Deutschland (DSD) program, now available to industry on request.

Key Contributions

1. A large multi-sensor railway dataset: 88.2 minutes (5292 seconds) of annotated sensor data with a total of 7,052,055 annotations, recorded with RGB cameras, infrared cameras, LiDAR and radar.
2. Two distinct instrumented recording platforms: a GAF track maintenance vehicle (1981 seconds, 69 sequences) and a BR472 commuter train (3311 seconds, 194 sequences), giving complementary vehicle and route perspectives.
3. Annotation coverage over 21 object classes created jointly by understandAI and DB InfraGO AG since 2021, spanning railway-specific elements (tracks, switches, signals, catenary poles, buffer stops, wagons) and generic perception objects (persons, crowds, bicycles, road vehicles, animals, prams, wheelchairs).
4. A documented annotation and quality-assurance pipeline, including 3D-first annotation with four projection functions, five levels of quality control, and delivery in RailLabel JSON format (a subschema of ASAM OpenLABEL).

Main Findings

• Scale of the dataset: The dataset contains 7,052,055 annotations in total, described in the abstract as "over 7 million high-quality annotations."
• Duration and structure: 88.2 min (5292 s) of annotated sensor data, split into 69 sequences from the GAF vehicle (1981 s) and 194 sequences from the commuter train BR472 (3311 s).
• Annotation throughput: During the annotation projects, understandAI achieved a peak delivery throughput of 140,000 annotations/week.
• Quality control coverage: About 5% of the data was randomly reviewed by DB InfraGO AG as the final quality step.
• Class balance: The pie charts in Figures 5 and 6 show most annotations concentrated on typical rail vehicles, rail infrastructure elements and persons — which, the authors state, distinguishes this dataset from typical automotive datasets. Classes below 2% representation were grouped into "other classes."
• Largest single class: Catenary Pole is the most annotated class by a wide margin: 1,494,205 annotations in the GAF data and 516,902 in the BR472 data.
• Other high-count classes (GAF / BR472): Person 984,818 / 467,063; Track 738,833 / 522,621; Signal Pole 326,273 / 312,362; Signal 148,189 / 447,573; Road Vehicle 175,788 / 127,544; Switch 79,811 / 112,093; Train 98,057 / 102,878.
• Rare classes with zero instances: Motorcycle (2620 in GAF, none in BR472) and Wheelchair (225 in GAF, none in BR472) are the only classes listed with no BR472 annotations.
• Comparison with prior work: Table 1 lists nine public railway CV datasets — RailSem19 (2019, 8500 frames, camera), FRSign (2020, 105352, camera), RAWPED (2020, 26000, camera), Rail-DB (2022, 7432, camera), RailSet (2022, 6600, camera), GERALD (2023, 5000, camera), OSDaR23 (2023, 1534, camera/lidar/radar), RailGoerl24 (2024, 12205, camera/lidar) and SynDRA (2025, 6572, simulated camera). OSDaR23 is described as the first publicly available multi-sensor railway dataset; the authors note their own dataset is not public but requestable.
• Sensor configuration of the GAF vehicle: Six RGB cameras, three infrared cameras, six LiDAR sensors with varying ranges, one radar sensor, plus position and acceleration sensors and a gas-particle sensor. Data was collected in Hamburg and Berlin, Germany.
• Sensor configuration of the BR472 commuter train: Three RGB cameras, one infrared camera, six LiDAR sensors and four radar units. Recording took place along a 23-kilometer section of the S21 commuter rail line between Hamburg-Berliner Tor and Bergedorf, Germany.
• Annotation format: Annotations are delivered as JSON files following the RailLabel JSON schema, a subschema of the ASAM OpenLABEL standard.
• Intended applications: GoA2 through GoA4 automation, infrastructure monitoring and environment observation.

Methodology in Plain English

The team took two real rail vehicles and fitted them with overlapping sensor suites. The first was a GAF track maintenance vehicle whose front section was equipped with cameras, infrared cameras, LiDAR, radar, position and acceleration sensors, and a gas-particle sensor; it ran in Hamburg and Berlin across both regular and non-regular operating scenarios. The second was a BR472 commuter train used in the Sensors4Rail project with partners including Deutsche Bahn, Bosch Engineering, HERE Technologies, Ibeo Automotive Systems (now MicroVision) and Siemens Mobility; it recorded a 23-kilometer urban-to-suburban stretch of the S21 line.

Annotation followed a sensor-fusion-first strategy. Experts annotated objects directly in the 3D LiDAR point clouds, and those 3D boxes were then projected onto the other sensors using four projection functions: 3D bounding boxes to 2D bounding boxes, 3D bounding boxes to 2D polygons, RGB images to infrared images, and 3D point clouds to radar images. All projected annotations were then checked and reworked where needed. Software tools were used at every stage — data import, quality checks, annotation and export — to reduce manual effort.

Quality was managed through five sequential levels: assessment of raw data quality (calibration, odometry), prototype development loops on sample datasets before production-scale annotation, automated real-time QA validators during annotation, pre-delivery final checks, and a random check by DB InfraGO AG covering about 5% of the data.

Why This Matters

Impact on research: Most prior public railway computer-vision datasets are camera-only and small (the largest frame count in Table 1 is 105,352 for FRSign, and the largest multi-sensor dataset, OSDaR23, has 1534 frames). This dataset combines over 7 million annotations with camera, infrared, LiDAR and radar data, providing far denser supervision and cross-modal consistency for railway perception research.

Real-world applications:

• Driver assistance for route monitoring at automation level GoA2, where a train driver is still on board.
• Fully automated, driverless operation at GoA4, where the perception system takes over environment monitoring entirely.
• Infrastructure monitoring, using repeated recordings of tracks, switches and signals.
• Detection of people and obstacles in the track environment, including persons, crowds, workers, pedestrians and animals.

Industry relevance: The dataset was produced within the Digitale Schiene Deutschland (DSD) program and its "Data Factory" data ecosystem, a collaboration between DB InfraGO AG and understandAI GmbH, with the Sensors4Rail project involving Bosch Engineering, HERE Technologies, Ibeo Automotive Systems (now MicroVision) and Siemens Mobility. It is positioned as a resource for the wider rail industry rather than a purely academic release, and is available upon request from DB InfraGO AG.

Future Directions

• Expanding the dataset to cover future sensor configurations, since the authors emphasize the flexibility to adapt data formats and specifications to new project needs.
• Broadening scenario and geographical coverage: the current data comes from Hamburg and Berlin and a 23-kilometer section of the S21 line, so generalization to other routes and conditions is an open question.
• Growing representation of the classes that are sparse or absent in the commuter train data (Motorcycle and Wheelchair appear only in the GAF data), and of the classes grouped under "other classes" because they fall below 2% of annotations.
• Extending the annotation pipeline, including the automated QA validators and projection functions, to additional modalities and standards as railway perception requirements evolve.

Target Audience

Researchers and engineers working on railway perception, autonomous train operation and multi-sensor fusion; AI/ML practitioners looking for annotated railway data spanning 2D and 3D, camera, infrared, LiDAR and radar; and rail industry teams evaluating or building environment monitoring systems across Grades of Automation. Anyone needing the data itself should contact DB InfraGO AG, as the paper reports no public download link, no dataset license terms for the data itself, and no train/validation/test splits.

Authors’ abstract

Reliable environment monitoring is essential for the safe and efficient operation of automated railway systems, covering all Grades of Automation (GoA), from partially automated (GoA2) to fully automated operation (GoA4). Artificial Intelligence (AI) plays a central role in enabling these systems to detect, classify, and react to potential hazards in real time. The development of such AI-based perception systems requires large volumes of accurately annotated data for training and validation. Within the Digitale Schiene Deutschland (DSD) program, DB InfraGO AG and understandAI GmbH have developed a comprehensive multi- sensor dataset tailored to the needs of railway environment perception. This dataset contains over 7 million high-quality annotations of both railway-specific and general perception objects, captured under varying operational scenarios. The finalized dataset can now be requested at the DB InfraGO AG and serve as a valuable resource for advancing AI-driven environment monitoring in the railway domain.