ObjectTransforms for Uncertainty Quantification and Reduction in Vision-Based Perception for Autonomous Vehicles

AI summary

Overview

Research area: Computer vision for autonomous driving — specifically uncertainty quantification and robustness of 2D object detectors used in safety-critical perception.

Technical level: Advanced. The paper combines a formal probabilistic framework (transformation invariance, law of total variance) with a practical pipeline involving HSV object perturbations and diffusion-model-based pedestrian synthesis.

Scope: The paper proposes ObjectTransforms, an object-level augmentation technique applied at both training and inference time, and evaluates it with YOLOv8 on the nuImages 10K dataset for reducing and quantifying uncertainty in vision-based object detection.

What This Paper Is About

Vision-based object detectors in autonomous vehicles still make mistakes: they miss pedestrians in low light and they hallucinate objects that are not there, which can cause unsafe behavior such as phantom braking. The root problem is that a detector's confidence score does not tell you how reliable that detection actually is, so the system cannot tell a trustworthy detection from an untrustworthy one. This paper introduces ObjectTransforms, which perturbs individual detected objects (rather than the whole image) to both make the detector more robust during training and to produce a real-time uncertainty signal during inference that can be used to discard false positives and recover false negatives.

Key Contributions

1. A theoretical formulation of uncertainty as a violation of transformation invariance. The authors formalize the intuition that a reliable detector should not change its detection decision when only an object's appearance is transformed, as long as the surrounding scene context stays the same. Using the law of total variance, they decompose the variance of the detection event into a "Noise" term (internal detector randomness) and an "Effect of transformations" term, showing that the second term vanishes for a perfectly invariant detector and therefore serves as a direct uncertainty measure.

2. Training-time use of ObjectTransforms to increase accuracy. Object-specific HSV perturbations are applied to individual object masks and reinserted at their original positions, and a diffusion-model pipeline generates additional realistic pedestrian instances. Training on this augmented data improves mAP50-95 relative to both the base dataset and standard image-level HSV augmentation.

3. Inference-time use of ObjectTransforms to improve the precision-recall curve. A controlled set of HSV perturbations is applied to each detected object, the detector is re-run, and the variance of confidence scores across perturbations becomes an uncertainty score. This score is used to filter unstable false positives and, by lowering the detection confidence threshold, to recover false negatives.

4. Practical feasibility evidence. The authors report the runtime cost of inference-time uncertainty estimation on real hardware.

Main Findings

• The theoretical claim is that transformation invariance implies zero variance. If the probability of detection is independent of the object-level transformation (given context), the variance across transformations is zero; any measured variance therefore indicates uncertainty. Uncertainty is defined in two parts: U_class(C) = variance of the classification score across transformations, and U_bbox(C) = one quarter of the sum of the variances of the box parameters x, y, w, and h. These are combined as U(C) = ω₁·U_bbox(C) + ω₂·U_class(C), with ω₁ + ω₂ = 1.

• Training with ObjectTransforms improves mAP50-95 on nuImages 10K. Overall mAP50-95 rose from 0.384 on the base dataset to 0.407 with ObjectTransforms, versus 0.383 with image-level HSV augmentation. Per class: pedestrians 0.298 → 0.314 (base → ObjectTransforms), barriers 0.359 → 0.397, cones 0.35 → 0.382, vehicles 0.528 → 0.533. The paper notes that YOLOv8 trained with ObjectTransforms data achieves an mAP50 of over 0.6, and that full-scale training on the entire nuImages dataset (over 90K data points) would likely push these metrics higher.

• Training with ObjectTransforms lowers measured uncertainty. Using Monte Carlo dropout (10 passes, conf = 0.5) on the nuImages test dataset, the model trained with ObjectTransforms yields lower bounding-box uncertainty and lower confidence uncertainty, with a consistent reduction of up to 20%, and the relative improvement grows with higher dropout rates.

• Inference-time uncertainty separates true positives from false positives. On the 1,484 test images, mean uncertainty scores were far lower for true positives than false positives: x-uncertainty 4.02×10⁻⁶ (TP) vs 2.34×10⁻⁵ (FP), separation 5.82; y-uncertainty 2.98×10⁻⁶ vs 2.74×10⁻⁵, separation 9.20; w-uncertainty 7.25×10⁻⁶ vs 6.75×10⁻⁵, separation 9.31; h-uncertainty 9.44×10⁻⁶ vs 1.07×10⁻⁴, separation 11.36; confidence uncertainty 6.26×10⁻³ vs 2.60×10⁻², separation 4.16.

• Filtering preserves true positives while cutting false positives. With weights ω₁ = 0.25 and ω₂ = 0.75 (found by grid search) and a threshold U_th = 0.146, at a detection confidence threshold of 0.25 the framework preserves 95% of true positives and eliminates about 32% of false positives, improving the TP/FP ratio from 3.57 to 4.98 (TP 3349 → 3186, FP 938 → 640).

• Lowering the confidence threshold with uncertainty filtering recovers false negatives. At conf = 0.01, without ObjectTransforms the detector produces 4818 TPs and 1864 FPs (ratio 2.59); with ObjectTransforms at U_th = 0.146 it produces 3719 TPs and 685 FPs (ratio 5.43); at U_th = 0.19 it produces 4156 TPs and 908 FPs (ratio 4.58). The paper states that with U_th = 0.19 the increase in TPs is more significant with a slight reduction in FPs compared with the conf = 0.25 baseline.

• Diffusion-based pedestrian augmentation gives comparable results. In a second experiment targeting only pedestrians, the ObjectTransforms dataset was generated using diffusion-based inpainting on pedestrian instances plus the Object Consistency Filter, producing 2,072 images. The paper reports similar results but limits extended discussion due to page length.

• Real-time feasibility is reported but modest. Inference-time uncertainty calculation with ObjectTransforms achieves 5 fps with the YOLOv8 Extra Large model, using 2.5 GB of GPU memory on an Nvidia L4 GPU. The authors expect a significant frame-rate improvement with lighter variants such as nano.

Methodology in Plain English

The central idea is that you can test how trustworthy a detection is by changing how the object looks and seeing whether the detector's answer stays the same. A confident, correct detection should keep roughly the same confidence score whether the detected object's colors are shifted slightly; an unstable, likely-wrong detection will see its score swing around.

During training, instead of applying one global color change to the whole image (as conventional HSV jittering does), the authors mask out each individual object, randomly perturb its hue, saturation, or value, and paste it back into its original location. This preserves scene context while greatly expanding object appearance variability — useful for handling lighting and camouflage conditions. For pedestrians, HSV shifts are considered unrealistic because they can distort skin and hair colors, so instead the authors use a diffusion model to inpaint pedestrians inside their masks. Because diffusion outputs can be wrong or inappropriate, they add an Object Consistency Filter: a CLIP encoder checks similarity between generated pedestrian images and reference text embeddings, keeping only high-similarity outputs, and an additional filter removes socially or ethically unacceptable images.

During inference, the detector runs on an image, and then a controlled set of HSV perturbations is applied to each detected object and the detector is re-run. The variance of the confidence scores across these perturbations becomes the uncertainty estimate, computed alongside the variance of the box coordinates. A weighted combination of the two is compared against a threshold. Detections above the uncertainty threshold are discarded as likely false positives. Because this filtering removes false positives, the detection confidence threshold can be lowered to catch more objects, recovering false negatives that would otherwise be missed.

The experiments use the YOLOv8x network on the nuImages 10K dataset, which contains 6,999 training, 1,515 validation, and 1,484 test images; the authors keep this partition. In the first setup they detect four classes (pedestrians, vehicles, barriers, cones) and generate 97,778 images — about 14 transformations of each image in the base dataset — using randomized HSV transformations. Three models are trained for 100 epochs each: on the base dataset, on the base dataset with image-level HSV augmentation, and on the ObjectTransforms dataset. The second setup targets pedestrians only.

Why This Matters

Impact on research. Most augmentation and test-time methods in this space operate at the image level. This paper argues that object-level transformations are a distinct and underexplored lever, and it claims to be the first work to explore object-specific test-time augmentations in the AV context for quantifying and reducing uncertainty. It also reframes uncertainty not as a Bayesian posterior quantity (as in Monte Carlo dropout or deep ensembles) but as a measurable violation of an invariance property, which gives a lightweight, architecture-agnostic signal. The paper explicitly links low uncertainty estimates to the possibility of reliable sensor fusion with lidar, radar, and V2X communication.

Real-world applications:

• Autonomous vehicle perception stacks that need to suppress phantom-braking-inducing false positives while still catching pedestrians in low-light, out-of-distribution conditions.
• Sensor fusion and downstream planning modules that need a per-detection reliability signal to weight or gate inputs from cameras against lidar and radar.
• Robotic and industrial perception systems that must operate under changing illumination or camouflage-like appearances.
• Datasets and data pipelines for safety-critical perception, where diffusion-generated, filtered synthetic instances can address class imbalance or rare appearance conditions.

Industry relevance. The results speak directly to the precision-recall tradeoff that AV perception teams manage daily: the demonstrated ability to lower the confidence threshold while holding false positives down is exactly the operating-point problem in deployment. The reported real-time numbers (5 fps, 2.5 GB of GPU memory on an Nvidia L4) define the current practical limit, but the authors point to lighter model variants as a path to higher frame rates, which is where industry deployment would need to land. The proposed use of a calibration set — analogous to conformal learning — to tune the weights and uncertainty threshold also maps onto the calibration and validation practices used in safety engineering.

Future Directions

• Improving low-visibility detection. The authors state directly that they plan to study how object detection in low-visibility conditions can be improved, which is the exact scenario motivating the night-time crosswalk example in the paper.
• Scaling beyond the 10K subset. The paper notes that full-scale training on the entire nuImages dataset (over 90K data points) is likely to yield much higher metrics, so verifying the gains at full scale is an open question.
• Broadening the transformation set. The framework is written for any object-specific transformation; the paper instantiates only HSV perturbations and diffusion-based pedestrian augmentation, leaving noise, crop, flip, and rotation unexplored, as well as object-level transforms for classes other than pedestrians, vehicles, barriers, and cones.
• Making inference-time uncertainty faster. At 5 fps with YOLOv8x, the method is not yet at automotive frame rates; the authors expect lighter variants such as nano to improve this significantly, and that expectation has not yet been demonstrated.
• Extending the reported results for the diffusion pipeline. The second experiment's comparable results are described only briefly because of page length, so a fuller quantitative treatment of the diffusion-based augmentation remains to be published.

Target Audience

Perception and autonomy engineers working on object detection and uncertainty estimation; machine learning researchers interested in invariance-based uncertainty quantification, object-level augmentation, or diffusion-based data synthesis; safety and validation engineers who need to reason about false positive and false negative tradeoffs in deployment; and graduate students or advanced practitioners with background in object detection, probability, and test-time augmentation. Beginners will find the theoretical section (Section 3) dense, but the methodological and experimental sections are accessible without deep prior exposure.

Authors’ abstract

Reliable perception is fundamental for safety critical decision making in autonomous driving. Yet, vision based object detector neural networks remain vulnerable to uncertainty arising from issues such as data bias and distributional shifts. In this paper, we introduce ObjectTransforms, a technique for quantifying and reducing uncertainty in vision based object detection through object specific transformations at both training and inference times. At training time, ObjectTransforms perform color space perturbations on individual objects, improving robustness to lighting and color variations. ObjectTransforms also uses diffusion models to generate realistic, diverse pedestrian instances. At inference time, object perturbations are applied to detected objects and the variance of detection scores are used to quantify predictive uncertainty in real time. This uncertainty signal is then used to filter out false positives and also recover false negatives, improving the overall precision recall curve. Experiments with YOLOv8 on the NuImages 10K dataset demonstrate that our method yields notable accuracy improvements and uncertainty reduction across all object classes during training, while predicting desirably higher uncertainty values for false positives as compared to true positives during inference. Our results highlight the potential of ObjectTransforms as a lightweight yet effective mechanism for reducing and quantifying uncertainty in vision-based perception during training and inference respectively.