Detector-Augmented SAMURAI for Long-Duration Drone Tracking

AI summary

Overview

Research area: Computer vision — visual object tracking (VOT), specifically RGB-based drone (UAV) tracking for surveillance, combining a foundation-model tracker with a task-specific drone detector.

Technical level: Intermediate. The paper is written for readers familiar with tracking benchmarks and metrics, but its core ideas (detector-guided correction of a tracker) are conceptually straightforward.

Scope: The paper reports the first systematic evaluation of the zero-shot tracker SAMURAI on drone tracking, proposes a detector-augmented variant that fuses YOLO-FEDER FusionNet detections into the tracking loop, and releases four long-duration drone sequences.

What This Paper Is About

Frame-by-frame drone detectors can be accurate on individual frames but produce temporally inconsistent output, with frequent detection dropouts that break continuous monitoring. Meanwhile, SAMURAI — a transformer-based foundation-model tracker with motion-aware memory — has shown strong zero-shot performance on general tracking benchmarks, but had never been evaluated on drone-specific scenarios. The authors ask whether SAMURAI can track drones robustly in urban surveillance footage, how sensitive it is to how the first bounding box is initialized, and whether injecting detector cues can fix its weaknesses over long sequences.

Key Contributions

1. A comprehensive evaluation of SAMURAI for RGB-based drone tracking across diverse environmental conditions, target scales, and sequence lengths, using both ground-truth and detector-based first-frame initialization.
2. A detector-augmented variant of SAMURAI, in which YOLO-FEDER FusionNet detections are fused with the tracker's mask-derived boxes through a Prediction Fusion Module for periodic re-prompting, re-localization, and correction.
3. A new dataset of four long-duration drone sequences (recorded at 2040 × 1086 pixels, 60 fps, lengths from 1484 to 6327 frames with increased scale variation), released publicly via Zenodo to complement DUT Anti-UAV.
4. The first systematic study positioning SAMURAI against previously reported drone trackers and detection-plus-tracking combinations on DUT Anti-UAV.

Main Findings

• Strong zero-shot performance with ground-truth initialization. On DUT Anti-UAV, SAMURAI with first-frame ground-truth boxes reaches a success rate (S) of 0.663, precision (P) of 0.888, and normalized precision (P_norm) of 0.973, with FNR 0.023 and FDR 0.031. This exceeds the best-performing tracker reported in the DUT Anti-UAV benchmark by 0.055 (S), 0.056 (P), and 0.115 (P_norm); the best such baseline is LTMU at 0.608 (S), 0.783 (P), 0.858 (P_norm).

• Sensitivity to initialization. Switching to detector-based initialization (the first YOLO-FEDER FusionNet box) degrades DUT Anti-UAV metrics by 0.044 to 0.079 on average (S drops from 0.663 to 0.614; FNR rises from 0.023 to 0.072). On R1, detector initialization instead improved every metric, cutting average FNR from 0.591 to 0.384; on R2 it made negligible difference.

• Large gains from detector augmentation on long custom sequences. Adding detector cues raises success rate from 0.343 to 0.635 on R1 (a 0.292 absolute gain) and from 0.374 to 0.569 on R2 (a 0.195 gain, both relative to detector-only initialization). FNR falls by 0.475 on R1 and 0.246 on R2. Across datasets and metrics the authors report success-rate improvements up to +0.393 and FNR reductions up to −0.475.

• Augmentation also helps on the public benchmark. On DUT Anti-UAV, detector-augmented SAMURAI reaches S 0.725, P 0.924, P_norm 0.971, mAP@0.25 0.976, mAP@0.5 0.865, mAP@0.5-0.95 0.490, FNR 0.012, FDR 0.017, compared with 0.614 / 0.842 / 0.925 / 0.909 / 0.641 / 0.354 / 0.072 / 0.080 for detector-initialized SAMURAI alone. The best earlier tracking-with-detection combination in the comparison table is LTMU with FRCNN-ResNet50 at 0.664 (S), 0.961 (P), 0.865 (P_norm).

• Recovery from bad initialization. In DUT Anti-UAV video05, the initial detection erroneously marks a car mirror as the drone; first-frame-only SAMURAI collapses to S 0.010 and FNR 1.000, while the detector-augmented variant recovers to S 0.699. In video04, an unfolding attached parachute causes appearance change; SAMURAI alone scores S 0.412, while the augmented variant reaches S 0.872.

• Apparent false positives traced to annotation gaps. Elevated FDRs in some sequences occur when the drone leaves and re-enters the field of view: the tracker follows the target until it disappears and resumes on re-entry, whereas ground-truth labels often end earlier or start later.

• Small-scale and high-complexity tracking remains difficult. On the custom datasets, even augmented SAMURAI peaks at S 0.635 (R1) and 0.569 (R2), and the authors state it occasionally still struggles with very small drones in complex scenes.

• Notable failure cases. Per-sequence R2 results vary widely: FNR is 0.009 on POS7 but 0.614 on POS3, where the tracker drifts to background structures. On DUT Anti-UAV video04, appearance variation from external attachments produces erroneous segmentation masks and inaccurate mask-derived boxes.

Methodology in Plain English

The authors take an existing zero-shot tracker, SAMURAI, which is built on SAM 2 and uses a Kalman filter plus motion-aware memory selection to pick masks, and leave it unmodified. They evaluate it two ways: starting it from the ground-truth box in frame one (the standard benchmark protocol), and starting it from the first frame's detection produced by YOLO-FEDER FusionNet (a drone detector that fuses a YOLOv8l backbone with the camouflage-detection model FEDER, trained on SynDroneVision and the DUT Anti-UAV detection subset, running at 12.4 ms per frame on an NVIDIA A100).

They then add a Prediction Fusion Module. The tracker is periodically re-prompted with a new bounding box whenever a detector output is deemed reliable: confidence above 0.75, strong spatial alignment with the tracker's own estimate (CIoU above 0.7), or proximity to recent trajectory history (IoU above 0.8 with the most recent prediction, or normalized center distance below 0.05, i.e. 5% of frame dimensions, within a 10-frame window). A prompt is forced at least once every 30 frames, roughly one second at 30 fps, provided a reliable detection exists. If the tracker's box is fully enclosed by the detector's box, the final estimate is the arithmetic mean of their coordinates, because the detector is trained specifically for drones while SAMURAI is zero-shot. When no detector output is available, the tracker propagates on its own.

Evaluation uses tracking metrics (Success Rate as AUC over IoU thresholds, Precision as AUC over center-distance thresholds from 0 to 50 pixels, Normalized Precision over thresholds from 0 to 0.5 normalized by image diagonal) plus detection metrics (mAP@0.25, mAP@0.5, mAP@0.5-0.95, FNR, FDR), aggregating by averaging per-sequence results. Test data are the 20-sequence DUT Anti-UAV tracking subset (83 to 2635 frames, object area ratios 2.7e-04 to 0.045, mean 0.0031) and the four custom sequences R1/R2 (2040 × 1086, 60 fps, up to 100+ consecutive frames out of frame, lengths 1484 to 6327 frames), captured with a Basler acA200-165c camera using an 8 mm lens for R1 and a 25 mm lens for R2, all showing a single Yuneec Mantis G.

Things the paper does not report: SAMURAI's own running speed or parameter count, training details for the augmented pipeline beyond the fixed fusion thresholds, and quantitative comparison against the drone-specific trackers SiamAD or the LSTM-based approach discussed in related work.

Why This Matters

Impact on research. The paper establishes that a general-purpose foundation-model tracker can beat conventional drone-tracking baselines out of the box, and that its main weaknesses — initialization sensitivity, appearance-change instability, and degradation over long sequences — are addressable with a modular, detector-agnostic fusion layer rather than task-specific retraining. It also documents that existing VOT benchmarks barely contain drones (drones are 1/80 of LaSOT), motivating drone-specific evaluation.

Real-world applications:

• Urban perimeter surveillance with fixed passive RGB cameras, where continuous tracking rather than isolated frame detections is needed.
• Airspace monitoring around sensitive infrastructure, given the increasing threat potential of off-the-shelf multi-rotor drones cited in the introduction.
• Counter-UAV systems that need to maintain a target lock through occlusion, partial field-of-view exits, and re-entries.
• Detection-plus-tracking pipelines that can swap in any object detector, since the Prediction Fusion Module is detector-agnostic.

Industry relevance. The method reuses an off-the-shelf drone detector and an off-the-shelf foundation-model tracker, so the engineering cost is integration rather than model training. The reported behavior under drone exit and re-entry — remaining locked when ground-truth annotations stop — is directly relevant to operators whose monitoring cannot tolerate gaps, though the paper also shows the reliability depends heavily on detector quality and on fusion parameters that can be affected by frame rate and drone velocity.

Future Directions

1. Reducing the remaining sensitivity to detector quality, which the conclusion names as a key bottleneck alongside reliable tracking of small targets in complex scenes.
2. Adapting the fusion logic to detectors with different reliability profiles, since the coordinate-averaging adjustment assumes the detector is generally more trustworthy than the zero-shot tracker.
3. Investigating the failure mode observed after temporary target disappearance, where the tracker struggles to re-identify the drone once it returns.
4. Tuning or adapting the fusion parameters, which the paper notes can be influenced by frame rate and drone velocity and were fixed at 0.75 confidence, 0.7 CIoU, 0.8 IoU, 0.05 center distance, a 10-frame window, and a 30-frame forced-prompt interval.

Target Audience

Researchers and engineers working on drone detection and tracking, visual object tracking, or foundation-model adaptation for surveillance; developers building counter-UAV or airspace-monitoring systems who need to decide between detector-only and detector-plus-tracker pipelines; and benchmark curators interested in the long-duration, high-scale-variation sequences the authors released. Readers with no background in tracking metrics will find the metric definitions manageable but will need the related-work section to place the baselines in context.

Authors’ abstract

Robust long-term tracking of drone is a critical requirement for modern surveillance systems, given their increasing threat potential. While detector-based approaches typically achieve strong frame-level accuracy, they often suffer from temporal inconsistencies caused by frequent detection dropouts. Despite its practical relevance, research on RGB-based drone tracking is still limited and largely reliant on conventional motion models. Meanwhile, foundation models like SAMURAI have established their effectiveness across other domains, exhibiting strong category-agnostic tracking performance. However, their applicability in drone-specific scenarios has not been investigated yet. Motivated by this gap, we present the first systematic evaluation of SAMURAI's potential for robust drone tracking in urban surveillance settings. Furthermore, we introduce a detector-augmented extension of SAMURAI to mitigate sensitivity to bounding-box initialization and sequence length. Our findings demonstrate that the proposed extension significantly improves robustness in complex urban environments, with pronounced benefits in long-duration sequences - especially under drone exit-re-entry events. The incorporation of detector cues yields consistent gains over SAMURAI's zero-shot performance across datasets and metrics, with success rate improvements of up to +0.393 and FNR reductions of up to -0.475.