A Unified Detection Pipeline for Robust Object Detection in Fisheye-Based Traffic Surveillance

AI summary

Overview

Research area: Computer vision for intelligent transportation systems — specifically object detection on fisheye (wide-angle) traffic surveillance imagery.

Technical level: Intermediate. The paper assumes familiarity with standard detector families (YOLO, DETR variants) and pipeline stages such as preprocessing, ensembling, and non-maximum suppression, but it explains each design choice in plain terms.

Scope: The paper presents and evaluates a unified pre-processing, detection, and ensemble pipeline for detecting traffic objects (Bus, Bike, Car, Pedestrian, Truck) in heavily distorted fisheye video, submitted to the 2025 AI City Challenge Track 4.

What This Paper Is About

Fisheye cameras capture panoramic views of up to 180 degrees from a single vantage point, which makes them attractive for traffic surveillance, but the strong radial distortion they introduce warps, compresses, and blurs objects — especially near the image edges — causing standard object detectors to miss them. The paper's goal is to build a detection pipeline that stays robust under this distortion without explicitly correcting the geometry of the image. Rather than redesigning detector architectures, the authors combine targeted image enhancement, a multi-model ensemble, and output refinement to improve detection consistency across the whole frame.

Key Contributions

1. A unified detection framework that operates robustly on fisheye images without explicit distortion modeling or geometric rectification.
2. A set of pre- and post-processing steps aimed at consistency under difficult conditions. The key preprocessing contribution is extending the training distribution to underrepresented nighttime scenes by adding synthetic nighttime images and generating pseudo-labels for them with YOLO-World.
3. A novel ensemble that combines four state-of-the-art detectors — YOLOR, YOLOv12, Salience-DETR, and Co-DETR — chosen for complementary strengths.
4. Extensive experiments validating the framework, including an ablation study and a final evaluation on the 2025 AI City Challenge Track 4.

Main Findings

• Final competition result: The pipeline achieved an F1 score of 0.6366 on the 2025 AI City Challenge Track 4, placing 8th out of 62 teams.
• Margin to the leader: The top team (UIT-OpenCubee) scored 0.6493, a difference of just 0.0127 from the authors' score.
• Best lightweight model: Among lightweight detectors, YOLOv12s at 1280 input resolution performed best overall, particularly on cars (0.597 recall, 0.810 precision) and bikes (0.472 recall, 0.766 precision).
• Best individual detector: YOLOR achieved the strongest recall and precision across nearly all categories, including bikes (0.898 recall, 0.973 precision), cars (0.937 recall, 0.987 precision), and trucks (0.970 recall, 0.985 precision), which led the authors to adopt it as the base model for the final ensemble.
• Ablation progression: A YOLOR-only ensemble at resolutions 1280, 1536, and 1920 reached an F1 of 0.6081; adding YOLOv12, Salience-DETR, and Co-DETR raised it to 0.6144; adding super-resolved YOLOR inputs raised it to 0.6225; and adding SAHI with Co-DETR produced the final 0.6366.
• No single component dominates: The ablation results indicate the gains emerge from the interaction of complementary modules rather than from any one step.
• Fusion method: Weighted Box Fusion (WBF) consistently outperformed Non-Maximum Suppression (NMS), Soft-NMS, and Non-Maximum Weighted (NMW) fusion in the authors' experiments, and was adopted as the final ensemble step.
• Automatic thresholding: Modeling prediction scores as a unimodal distribution and applying Otsu's method produced a confidence cutoff of 0.571, which retained the high-density region of confident scores.

Methodology in Plain English

The pipeline has three stages: clean the input, detect with several models, then clean up the outputs.

Preprocessing. Traffic frames from fisheye cameras suffer from motion blur, so the authors ran every training and test image through NAFNet, a deblurring network, and trained all detectors exclusively on the deblurred images. Because nighttime scenes were scarce in the base dataset, they added synthetically generated night images from a publicly available dataset used in the previous year's competition, and used YOLO-World to auto-generate pseudo-labels for those synthetic images.

Detection. They fine-tuned both lightweight and heavyweight detectors from COCO-pretrained weights on the Fisheye8K training set: YOLOR-D6 (300 epochs, Adam, learning rate 0.01, decay 0.001, 1280×1280 input); YOLOv11n and YOLOv11s at 640 and 1280 (SGD, learning rate 0.01); YOLOv12n and YOLOv12s at 640 and 1280 (300 epochs, SGD with cosine annealing from 0.01); five Salience-DETR backbones (Swin-L, ConvNeXt, ResNet-50, FocalNet, ResNet-5scale; AdamW, learning rate 1e-4); and two Co-DETR variants (ViT-L and Co-Deformable ResNet-50; AdamW, learning rate 1e-5). All training ran on NVIDIA L40 GPUs with 48 GB of memory.

Post-processing. Small objects near the frame edges were addressed with the Dual Aggregation Transformer (DAT) for super-resolution at a factor of four — for example, original 1755×1760 images were upscaled to 7020×7040, then resized to standard inference dimensions such as 2520 or 1280. Slicing Aided Hyper Inference (SAHI) split large images into overlapping slices (half the original size, 25 percent overlap) for localized detection. Low-confidence noise was removed with an Otsu-derived confidence threshold of 0.571, and the remaining boxes from the four ensemble detectors were merged using Weighted Box Fusion, which computes a confidence-weighted average of box coordinates rather than suppressing overlapping boxes.

Why This Matters

The paper shows that a carefully assembled pipeline can make conventional detectors competitive on strongly distorted wide-angle imagery without paying the cost of spherical convolutions, curved bounding boxes, or explicit image rectification. That matters because rectification and distortion-aware architectures add complexity, latency, and calibration burden to real deployments.

Real-world applications:

• Intersection and roundabout monitoring, where a single fisheye camera can cover the whole junction with better spatial continuity than several narrow-field cameras.
• Incident and congestion detection, since the framework targets consistent detection across the entire frame, including the periphery where distorted objects are usually missed.
• Low-light and nighttime traffic surveillance, the specific gap the synthetic night-image augmentation was designed to close.
• Infrastructure-cost reduction, replacing multi-camera rigs with a single wide-angle device while avoiding added calibration and alignment work.

Industry relevance: Intelligent transportation systems deployments often trade detection robustness for hardware simplification. This framework suggests that competitive accuracy on panoramic cameras is reachable with off-the-shelf models plus a disciplined pre- and post-processing pipeline, which lowers the barrier for municipalities and vendors operating large camera fleets.

Future Directions

• How much of the remaining gap to the top-ranked team can be closed by better handling of the hardest categories, given that pedestrians and small peripheral objects show the lowest precision in the reported tables?
• Can the super-resolution and slicing steps be made cheap enough for real-time streaming inference, since the reported pipeline upscales images by a factor of four before resizing them down again?
• How reliable are YOLO-World pseudo-labels for synthetic nighttime imagery, and would higher-quality labels or real nighttime data change the low-light gains?
• Does the ensemble's benefit persist when the number of contributing detectors is reduced, or when the base model is swapped away from YOLOR?

Target Audience

This paper is most useful to computer vision engineers and researchers working on traffic surveillance, smart-city sensing, or any application involving wide-angle and fisheye cameras, as well as practitioners who need a practical recipe for combining detectors and post-processing under domain-specific image degradation. Competition participants in benchmarks such as the AI City Challenge will also find the ablation breakdown and the specific component choices directly actionable.

Authors’ abstract

Fisheye cameras offer an efficient solution for wide-area traffic surveillance by capturing large fields of view from a single vantage point. However, the strong radial distortion and nonuniform resolution inherent in fisheye imagery introduce substantial challenges for standard object detectors, particularly near image boundaries where object appearance is severely degraded. In this work, we present a detection framework designed to operate robustly under these conditions. Our approach employs a simple yet effective pre and post processing pipeline that enhances detection consistency across the image, especially in regions affected by severe distortion. We train several state-of-the-art detection models on the fisheye traffic imagery and combine their outputs through an ensemble strategy to improve overall detection accuracy. Our method achieves an F1 score of0.6366 on the 2025 AI City Challenge Track 4, placing 8thoverall out of 62 teams. These results demonstrate the effectiveness of our framework in addressing issues inherent to fisheye imagery.