Benchmarking SAM2-based Trackers on FMOX

AI summary

Overview

Research area: Computer vision — video object tracking (VOT) and video object segmentation (VOS), specifically benchmarking Segment Anything Model 2 (SAM2)-based trackers on fast-moving object (FMO) sequences.

Technical level: Intermediate. The paper assumes familiarity with tracking metrics (IoU, Dice) and segmentation backbones, but the evaluation protocol is straightforward.

Scope: A benchmarking study that evaluates four SAM2-based trackers (SAM2, EfficientTAM, DAM4SAM, SAMURAI) on the 46-sequence FMOX dataset using mIoU and mDice, plus execution-time measurements.

What This Paper Is About

Several trackers built on SAM2 have appeared in the past year, each claiming advantages for different tracking difficulties (distractors, fast motion, efficiency). The authors ask how these trackers actually behave on datasets specifically built around fast-moving objects — a case that standard benchmarks under-represent because of high speed and motion-induced blur. Their goal is to give a more detailed picture of current limitations by scoring all four trackers on the same challenging sequences with standard metrics.

Key Contributions

1. A unified benchmark of four SAM2-based trackers (SAM2, EfficientTAM, DAM4SAM, SAMURAI) on FMOX, initialized identically with ground-truth bounding boxes under each tracker's default configuration.
2. A shift in evaluation metric from Trajectory Intersection over Union (TIoU), used in prior FMOX work (Aktas et al.), to the more widely comparable Mean Intersection over Union (mIoU) and Dice Score (mDice).
3. Per-dataset performance rankings across the four FMO datasets in FMOX (Falling Object, TbD, TbD-3D, FMOv2), showing that the best tracker changes depending on the dataset.
4. Frame-level failure analysis and compute-cost profiling, identifying motion blur as a recurring trigger for missed detections and quantifying execution time per dataset for each tracker.

Main Findings

• DAM4SAM is the strongest overall tracker. On the full 46-sequence FMOX set, DAM4SAM achieves the best mean and median for both metrics: mIoU median 0.605 and mean 0.505; mDice median 0.744 and mean 0.600 (marked best in the paper's Table 1).
• EfficientTAM is the weakest performer. It records the lowest mean and median on both metrics: mIoU median 0.548, mean 0.438; mDice median 0.684, mean 0.520.
• SAMURAI is second best overall. mIoU median 0.596, mean 0.488; mDice median 0.736, mean 0.579.
• SAM2 shows the highest single-sequence scores in some cases. Its maximums are mIoU 0.928 and mDice 0.962, above DAM4SAM's maxima (mIoU 0.819, mDice 0.899), even though its means and medians are lower.
• Complete failures occur. Minimum mIoU is 0.000 for SAM2, EfficientTAM and DAM4SAM, and 0.001 for SAMURAI; minimum mDice is 0.000 for the first three and 0.002 for SAMURAI.
• Rankings differ by dataset. Falling Object: DAM4SAM > SAM2 > SAMURAI > EfficientTAM. TbD-3D: SAM2 > DAM4SAM > SAMURAI > EfficientTAM. FMOv2: SAMURAI > DAM4SAM > SAM2 > EfficientTAM. TbD: DAM4SAM > SAMURAI > EfficientTAM > SAM2.
• Smaller objects and non-overlapping boxes are harder. Only FMOv2 and TbD contain sequences with object sizes from "extremely tiny" to "small", and in those datasets ground-truth bounding boxes rarely overlap between frames n and n+1; these are the datasets where tracker performance is most affected.
• EfficientTAM is the fastest tracker on every dataset. Execution times in seconds: Falling Object 24.64, FMOv2 410.07, TbD 67.31, TbD-3D 39.44. DAM4SAM is the slowest on every dataset (67.16, 1316.92, 261.42, 168.44 respectively).
• Motion blur triggers prolonged EfficientTAM failures. On sequence v_rubber_GTgamma (Falling Object), all trackers fail to produce a bounding box for frame 38, and EfficientTAM additionally misses frames 39, 40 and 41. On v_key_GTgamma, all trackers fail at frame 36, while DAM4SAM is the only successful tracker at frame 35.
• On throw_tennis (TbD), all trackers performed poorly except DAM4SAM, and EfficientTAM failed to initialize because of strong motion blur on the object, producing no performance curve at all for that sequence.
• Tracker behaviour correlates on some sequences and diverges on others. On HighFPS_GT_depth2 (TbD-3D) all trackers performed well and gave similar results; on throw_tennis one tracker succeeded while others were near zero.

Methodology in Plain English

The authors took four tracking models that extend SAM2 and gave each one the same starting information: the first ground-truth bounding box of the target object in each sequence (which is sometimes not in the very first frame). They used each tracker's default model configuration — SAM2 and DAM4SAM with SAM2.1 Hiera Large, SAMURAI with SAM2.1 Hiera Base Plus, and EfficientTAM with efficienttam_s — and did not retrain or fine-tune anything, so the comparison is zero-shot.

Scoring works frame by frame. For every frame where FMOX provides a ground-truth box, the predicted box is compared to it using IoU and Dice. Frames with no ground truth (for example when the object has left the view) are skipped. The initialization frame is omitted because its location was handed to the tracker. If a tracker produces no box at all, that frame is scored as zero, which is the worst possible value. These per-frame values are then averaged per sequence to give mIoU and mDice, and summarized across the 46 sequences using minimum, maximum, mean and median. None of the 46 sequences were used during training, so the authors state there is no data leakage. Execution times were measured on an NVIDIA GeForce RTX 4090 GPU with a 13th Gen Intel Core i9 processor, 64 GB of RAM, CUDA 12.4.1, and Ubuntu 20.04.6 on Windows Subsystem for Linux.

Why This Matters

Impact on research. The paper shows that headline benchmark scores can hide real weaknesses: a tracker can score near zero on some sequences while excelling on others, and the top-ranked tracker changes with the dataset. By reporting mIoU and mDice instead of TIoU, the results are directly comparable with the broader VOS/VOT literature, and the alignment with observations from MOSEv2 (DAM4SAM performing best) adds cross-benchmark evidence.

Real-world applications:

• Sports analysis, where balls, pucks or other fast projectiles are tracked across frames with heavy motion blur.
• Autonomous driving, where fast-moving pedestrians or vehicles must be segmented reliably.
• Robotics, where the system must follow rapidly moving objects in cluttered scenes.
• Any deployment needing efficient on-device segmentation, where EfficientTAM's speed advantage (lowest execution time on all four datasets) trades off against its lowest accuracy.

Industry relevance. The paper reports both accuracy and wall-clock cost, which is the practical trade-off for deployment: DAM4SAM gives the best accuracy but the highest execution time on every dataset, while EfficientTAM is fastest but least accurate. SAMURAI is described as the second fastest tracker while retaining good accuracy relative to EfficientTAM.

Future Directions

1. Address motion-blur-driven initialization failure. EfficientTAM failed to initialize on throw_tennis because of strong motion blur; how trackers should be initialized when the first available box is blurred remains open.
2. Improve memory handling for fast motion. DAM4SAM already replaces SAM2's first-in-first-out memory with Recent Appearance Memory and Distractor Resolving Memory; further memory designs could target the prolonged failures EfficientTAM shows after blurred frames.
3. Investigate the small-object, non-overlapping-box regime. FMOv2 and TbD are identified as the most challenging settings, and the low minima of zero across datasets indicate sequences where trackers fail entirely.
4. Extend beyond bounding-box initialization and box-based scoring. The paper initializes all trackers with bounding boxes and scores predicted boxes; mask-level evaluation and point initialization are noted as prior alternatives but are not explored here.

Target Audience

Researchers and engineers working on video object tracking and segmentation, particularly those evaluating or deploying SAM2-family trackers; practitioners who need to choose between accuracy and compute cost for fast-moving-object applications; and benchmark designers interested in how metric choice (TIoU versus mIoU/mDice) and dataset difficulty affect conclusions about state-of-the-art trackers.

Authors’ abstract

Several object tracking pipelines extending Segment Anything Model 2 (SAM2) have been proposed in the past year, where the approach is to follow and segment the object from a single exemplar template provided by the user on a initialization frame. We propose to benchmark these high performing trackers (SAM2, EfficientTAM, DAM4SAM and SAMURAI) on datasets containing fast moving objects (FMO) specifically designed to be challenging for tracking approaches. The goal is to understand better current limitations in state-of-the-art trackers by providing more detailed insights on the behavior of these trackers. We show that overall the trackers DAM4SAM and SAMURAI perform well on more challenging sequences.