Synset Signset Germany: a Synthetic Dataset for German Traffic Sign Recognition

AI summary

Overview

Research area: Computer vision — synthetic dataset generation for traffic sign recognition, with applications to explainable AI (XAI) and robustness testing.

Technical level: Intermediate. The paper assumes familiarity with GANs, physically-based rendering, and standard classification benchmarks, but the pipeline is described in accessible engineering terms.

Scope: The paper presents a synthesis pipeline and a 105,500-image synthetic German traffic sign dataset, evaluates its realism against real and synthetic benchmarks, and demonstrates its use for XAI and robustness analysis.

What This Paper Is About

Training and testing traffic sign recognition systems requires large amounts of labeled image data, but real-world collection is expensive — the paper cites an average of 90 minutes for annotation and quality control of a single pixel-level segmentation image in the Cityscapes dataset — and rare or newly released signs are hard to capture. The authors build a hybrid pipeline that combines data-driven GAN texture generation of dirt and wear with analytically controlled scene, lighting, and camera simulation, and release the resulting dataset as Synset Signset Germany with 211 classes and full parameter metadata. The goal is a synthetic dataset realistic enough to train on and parameterizable enough to test models against.

Key Contributions

1. A hybrid synthesis pipeline in the Fraunhofer OCTANE platform that separates data-driven factors (GAN-generated sign surface defects) from analytically modeled factors (lighting, camera pose, poles, occlusion, imaging artifacts), with each stage tied to a self-contained, explicitly specified model.
2. The Synset Signset Germany dataset: 105,500 images covering 211 German traffic sign classes at 500 images per class, including newly published (2020) and otherwise rare signs, with a 43-class subset designed as a "synthetic twin" of GTSRB.
3. Rich per-image annotation: a binary mask, a segmentation image, and extensive metadata recording the stochastically selected environment and imaging effect parameters for every image.
4. A demonstration of the dataset for XAI and robustness analysis, showing that explanation quality and global robustness can be quantified as functions of a controlled perturbation intensity (motion blur).

Main Findings

• Training on Synset Signset generalizes well: A ConvNeXt-Small model trained on Synset Signset Cycles reached 99.5% top-1 accuracy in-domain, 98.3% on the real-world GTSRB, and 84.4% on CATERED. The OGRE-trained version reached 99.6%, 99.6% in-domain, 98.2% on GTSRB, and 84.6% on CATERED.
• Cross-domain penalty is small against real data: Across cross-dataset scenarios, scores were only 1.2 percentage points behind in-domain training when evaluating on the real-world GTSRB dataset.
• Synset Signset is a harder evaluation set: Training on GTSRB reached 99.9% on GTSRB but only 89.4% and 87.4% on the two Signset versions; training on CATERED reached only 50.0% and 48.6% on Signset. The authors read this as evidence of the dataset's usefulness for evaluation given saturation of possible improvements on GTSRB.
• Full 211-class model performance: A model trained on all classes still achieved 99.6% for both full Signset dataset versions, Cycles and OGRE.
• Clear advantage over a purely analytical synthetic dataset: Compared to CATERED, Synset Signset Germany achieves approximately 20% better top-1 accuracy, which the authors attribute to increased realism from GAN-generated textures.
• Explanations degrade with motion blur: Using KernelSHAP feature attribution maps and binary masks, the pixel ratio (proportion of positively attributed pixels belonging to the sign) fell from 0.74 with no motion blur to 0.63 at mid-level and 0.58 at high-level motion blur.
• Robustness degrades with motion blur: Global robustness for a ResNet-18 trained on GTSRB fell from 0.9 (no motion blur) to 0.88 (mid-level) to 0.84 (high-level).
• Rendering cost motivates the OGRE choice: For XAI and robustness analysis the rendering was performed by OGRE to test more configurations due to reduced render time; mask and segmentation images were also created with OGRE.

Methodology in Plain English

The authors split the generation problem into parts that must be learned from data and parts that can be calculated from physics.

Sign surface defects. Wear, tear, fading, and vandalism are hard to model analytically, so a Pix2Pix-based GAN is used. It is made fully convolutional by removing the central 1×1×512 bottleneck so it can accept input and output dimensions of arbitrary size. It was trained on 200+ worn traffic signs at a fixed spatial resolution of 8 px/cm, converting template patches into defective texture patches. Training pairs were formed by randomly cropping and rotating originals into 256²×3 patches and randomly shuffling RGB channels to boost underrepresented hues (yellow, green, cyan, purple). Template inputs are derived from the Wikipedia overview of German traffic signs: each sign is split into black, white, red, orange, yellow, green, and blue components, each color component is stochastically faded based on the distribution of real sign samples, and a procedurally generated gray dirt mask is overlaid, combining arbitrary shapes with rectangular shapes representing worn stickers. Gray is not supported as a sign color, and retroreflector patterns are excluded — retroreflection is not simulated. The generated textures are used only as the diffuse component of the PBR surfaces.

Scene variation. Each sign instance gets a unique texture. PBR roughness is uniformly varied between 0.2 and 0.4 and the specular component between 0.3 and 0.5. Signs are assigned to vertical poles or to rotating horizontal/vertical poles (horizontal chosen uniformly with probability 0.3); pole diameter varies between 8 cm and 12 cm vertically and 8 cm and 20 cm horizontally, with PBR roughness between 0.4 and 0.6 and diffuse color R=G=B in [0.25, 0.4]. Permissible upper and lower signs per pole were manually labeled using the German StVO and real-world examples, covering the 211 signs in the dataset plus 135 additional supplementary traffic signs; additional signs are added to vertical poles with probability 0.5 to increase difficulty. Camera orientation is sampled from normal distributions (vertical poles: roll ~ N(0.0°, 2.0°), pitch ~ N(5.0°, 10.0°), yaw ~ N(0.0°, 21.0°); horizontal poles: roll ~ N(0.0°, 2.0°), pitch ~ N(30.0°, 10.0°), yaw ~ N(0.0°, 16.0°)), with the camera centered on the sign. Lighting uses image-based lighting over 327 uniformly sampled environment maps from Polyhaven, with uniformly varied azimuth. A 3D tree object is randomly placed to cast shadows in 3/4 of the images.

Optical simulation. Rendering follows physically-based rendering and separates idealized geometric ray/surface tracing from convolutional effects and degradations computed on the resulting HDR raster images. OCTANE supports both the rasterization-based OGRE engine and the Cycles path tracer from Blender; all 105,500 images are provided in both renderings. Post-processing includes stochastic automatic exposure control and white balance errors, a point spread function modeled on a Tamron M112FM35 35 mm lens via a mixture-of-Gaussian model, lens flares and lighting-dependent noise, motion blur and chromatic aberration via linear convolution kernels of uniformly distributed length ~ U(0 px, 10 px), digital sharpening via unsharp masking, and Bayer BGGR bilinear demosaicing artifacts. All stochastically selected parameter values are recorded per image.

Evaluation. The 43-class overlap subset with GTSRB was used, plus CATERED as a synthetic reference. A ConvNeXt-Small network was trained with learning rates of 10⁻⁴, 10⁻³, and 10⁻², selecting the best in-domain learning rate per dataset. Random flip augmentation was omitted because sign orientation is a distinguishing feature, and for CATERED the center crop was removed in favor of direct resizing to 224×224 since CATERED images are already tightly cropped. An 80–20 split was applied to Signset; official train–validation splits were used for the other datasets.

XAI and robustness experiments. Explanation quality was quantified as the pixel ratio — the proportion of positively attributed pixels, weighted by attribution value, that constitute the traffic sign relative to the entire image — computed with KernelSHAP from the Captum library. Global robustness was evaluated with a method based on a sequence of hypothesis tests certifying a level of global robustness at a required confidence level, comparing the prediction for the original input against the ground-truth label across perturbation intensities. The demonstration used a ResNet-18 trained on GTSRB, perturbed with three motion blur intensities (none, mid-level, high-level).

Why This Matters

The paper addresses a practical bottleneck in safety-relevant perception: obtaining sufficient, current, and well-annotated data for traffic sign recognition, a task explicitly named in the European AI Act's high-risk category of "AI systems intended to be used as safety components in the management and operation of road traffic." The AI Act's March 2024 text requires training, validation, and testing datasets to be relevant, sufficiently representative, and, to the best extent possible, free of errors and complete for the intended purpose. Because the pipeline records the parameters behind each image, effects observed on synthetic data can be traced back to known causes rather than inferred.

Real-world applications:

• Driver assistance and automated driving: Training and testing classifiers for sign detection, including rare classes such as newly published (2020) signs that are poorly represented in real datasets.
• Safety validation and robustness testing: Sweeping controlled perturbation intensities (motion blur, exposure, noise, shadow) to certify a model's stability under specified conditions.
• Explainable AI auditing: Using masks and parameterized scenes to measure whether saliency explanations focus on the sign itself rather than background.
• Mapping and inventory systems: Extending sign coverage to classes underrepresented in existing benchmarks, complemented by the 135 additional supplementary signs modeled in scene composition.

Industry relevance: Any organization developing road-traffic perception systems faces high annotation cost — the 90-minute figure per pixel-level segmentation image cited for Cityscapes — and difficulty covering rare signs. A downloadable dataset with ground-truth masks, segmentation images, and per-image parameter metadata lowers data acquisition cost and supports the documentation and traceability that regulatory compliance increasingly requires. The authors also note that the pipeline is scalable toward further applications, with international traffic signs identified as a next step.

Future Directions

• International extension: The authors state that an important next step is to abandon the current limitation to German traffic signs and provide extensions toward international traffic signs.
• Richer defect synthesis: The GAN-based defect synthesis currently uses a very simple concept that does not distinguish between types of dirt versus damage/wear, and lacks representation of gray sign areas and retroreflectors.
• More scene diversity: The number of occlusion objects should be increased for more complex and diverse shadow casts and occlusions; if the dataset is to be significantly enlarged, more environment maps should be collected for greater variance.
• Broader model evaluation: The choice of recognition models is limited to a few state-of-the-art deep learning models; a more extensive analysis including more diverse models and potentially classical, non-deep-learning approaches would substantiate the findings.
• Open question — synthetic data as testing data: The paper notes that while the experiments indicate the domain gap is low enough for practical use as training data, the requirements for use as testing data are considerably higher, since the dataset must enable quantitative performance estimation of trained models by relating effects on synthetic data to those on yet unseen real-world data.

Target Audience

Researchers and engineers working on traffic sign recognition, synthetic data generation, and simulation-based perception testing; practitioners needing parameterized datasets for robustness certification and XAI auditing of safety-relevant models; and teams addressing dataset documentation requirements under the European AI Act. The paper is also relevant to those studying sim-to-real transfer, since it reports explicit cross-dataset accuracy comparisons against GTSRB and CATERED.

Authors’ abstract

In this paper, we present a synthesis pipeline and dataset for training / testing data in the task of traffic sign recognition that combines the advantages of data-driven and analytical modeling: GAN-based texture generation enables data-driven dirt and wear artifacts, rendering unique and realistic traffic sign surfaces, while the analytical scene modulation achieves physically correct lighting and allows detailed parameterization. In particular, the latter opens up applications in the context of explainable AI (XAI) and robustness tests due to the possibility of evaluating the sensitivity to parameter changes, which we demonstrate with experiments. Our resulting synthetic traffic sign recognition dataset Synset Signset Germany contains a total of 105500 images of 211 different German traffic sign classes, including newly published (2020) and thus comparatively rare traffic signs. In addition to a mask and a segmentation image, we also provide extensive metadata including the stochastically selected environment and imaging effect parameters for each image. We evaluate the degree of realism of Synset Signset Germany on the real-world German Traffic Sign Recognition Benchmark (GTSRB) and in comparison to CATERED, a state-of-the-art synthetic traffic sign recognition dataset.