Tracking and Segmenting Anything in Any Modality

AI summary

Overview

Research area: Computer vision — unified video object tracking and segmentation across multiple input modalities (RGB, thermal, depth, event).

Technical level: Advanced. The paper assumes familiarity with Transformer encoders, Mixture-of-Experts layers, memory-based video segmentation (SAM2), and tracking-by-detection/tracking-by-query paradigms.

Scope: The paper proposes SATA, a single framework with fully shared architecture and weights that performs Single Object Tracking (SOT), Multiple Object Tracking (MOT), Video Object Segmentation (VOS), and Multi-Object Tracking and Segmentation (MOTS) on four modality combinations, evaluated on 18 benchmarks.

What This Paper Is About

Tracking and segmentation are usually solved by separate models built for one task and one input modality (e.g., an RGB-only tracker, an RGB-Thermal segmenter), which means redundant parameters, separate training runs, and no reuse of knowledge across tasks or sensors. Recent work has tried to unify either the tasks (one model, many tracking/segmentation subtasks) or the modalities (one model, any sensor input), but the authors argue that simply combining the two still leaves two gaps: a distributional gap between modalities and a feature representation gap between tasks. SATA is the authors' attempt to close both gaps within one model, one set of weights, and one inference pipeline.

Key Contributions

1. A first-of-its-kind unified framework (SATA). The authors state that SATA is the first unified framework capable of performing both tracking and segmentation tasks with arbitrary modality input and multi-task joint prediction.

2. Decoupled Mixture-of-Expert (DeMoE). A mechanism that replaces the feed-forward network in each Transformer encoder layer and splits unified representation learning into two parallel processes: a Common-prompt Mixture of Expert (CpMoE) for cross-modal shared knowledge and a Specific-activated Mixture of Expert (SaMoE) for modality-specific clues, plus a Decoupling Learning objective.

3. Task-aware MOT (TaMOT) pipeline. Instead of separate task-specific heads, all subtask outputs (SOT, VOS, MOT, MOTS) are unified as a set of instances with calibrated ID information, which the authors say alleviates degradation of task-specific knowledge during multi-task training.

4. Empirical validation on 18 benchmarks. Results are reported across 4 subtasks and 4 modality types using the same model architecture and parameter set.

Main Findings

• RGB SOT (Table 1a): SATA reaches 81.3% AO on GOT10K and 77.3% AUC on LaSOT, surpassing SAM2.1++ (75.1 AUC on LaSOT), SUTrack-L (75.2), and LMTrack (73.2) — reported as gains of 0.2%/2.2%, 0.3%/2.1%, and 1.2%/4.1% respectively. Against unified task models UTT, Unicorn, and OmniTracker-L, the paper reports LaSOT AUC gains of 12.7%, 8.8%, and 7.9%. SATA scores 91.4 SR0.5 and 83.7 SR0.75 on GOT10K, and 85.7 P Norm / 84.6 P on LaSOT.

• Multi-modal SOT (Table 1b–d): SATA reports 77.8 PR / 61.7 SR on LasHeR and 94.3 PR / 71.5 SR on RGBT234 (RGB-T); 67.9 PR / 67.6 Re on DepthTrack and 78.4 EAO / 84.1 Acc on VOTRGBD (RGB-D); 82.8 PR / 66.7 AUC on VisEvent and 80.4 PR / 71.6 SR on COESOT (RGB-E). The paper states it surpasses XTrack, SUTrack, and FlexTrack by 4.7%/2.5%/2.3%, 0.9%/1.4%/2.3%, and 0.5%/0.8%/1.4% in PR on LasHeR, DepthTrack, and VisEvent respectively.

• MOT (Table 2): On BDD, SATA reports 73.2 IDF1, 46.3 mMOTA, 67.8 MOTA; on DanceTrack, 83.7 IDF1, 76.1 HOTA, 90.5 MOTA. The BDD mMOTA of 67.8% is reported as surpassing Unicorn, UNINEXT-H, and SAM2MOT-Co by 1.2%, 0.7%, and 10.3%. On RGB-T MOT (UniRTL), SATA reports 59.7 HOTA, 63.6 DetA, 53.7 AssA, 75.1 MOTA, 65.4 IDF1, outperforming UnisMOT by 5.3% in HOTA.

• VOS (Table 3): SATA reports 93.4 J&F on DAVIS 2016 val (91.6 J, 95.2 F) and 89.7 J&F on DAVIS 2017 val (86.1 J, 93.0 F). On multi-modal VOS it reports 87.4 J&F on VisT300 and 88.5 on VTUAV (RGB-T), 85.1 on ARKitTrack (RGB-D), and 71.4 on LLE-VOS (RGB-E).

• MOTS (Table 4): On BDD MOTS, SATA reports 38.1 mMOTSA, 72.3 mMOTSP, 52.4 mIDF1, and 721 ID switches, which the paper describes as surpassing Unicorn and UNINEXT-H by 8.5% and 2.4% in mMOTSA.

• Ablation on DeMoE (Table 5): Removing CpMoE drops LasHeR PR from 77.8 to 75.8 and UniRTL HOTA from 59.7 to 56.2; removing SaMoE drops LLE-VOS to 69.3 and UniRTL to 55.3; removing both gives 80.8/74.7/67.9/56.1/37.7. Removing the MoE loss gives 80.7 on GOT10K, 75.2 on LasHeR, 70.2 on LLE-VOS, 58.4 on UniRTL, and 36.9 on BDD MOTS.

• Ablation on TaMOT (Table 5): Removing the Candidates Generation Module (CGM) drops GOT10K to 78.5 and LLE-VOS to 68.7 (MOT/MOTS not applicable). Removing fine-grained memory gives 34.1 mMOTSA; removing spatiotemporal memory gives 30.7; removing the whole Memory-enhanced Module (MEM) gives 29.1 on BDD MOTS versus 38.1 for the full model.

Methodology in Plain English

SATA turns every subtask into a version of multiple-object tracking. The model has two parts.

First, a weight-shared Transformer encoder (the main text specifies HiViT-L; the supplementary material states Hiera-L) turns RGB frames and auxiliary-modality frames (thermal, depth, or event, collectively abbreviated "TDE" in the paper) into tokens. At each encoder layer, the standard feed-forward block is replaced by DeMoE. One branch, CpMoE, keeps a frozen copy of the pre-trained feed-forward weights as a shared expert, adds several "modality-common" experts, and multiplies the resulting RGB and TDE prompts element-wise so the two modalities share one common prompt. The other branch, SaMoE, uses separate routers and expert pools for RGB and TDE plus a cross-modal router that activates the appropriate modality branch. The outputs are added back with residual connections. Two losses enforce this split: a cross-modal complementary loss that masks patches of one modality and requires the model to reconstruct them, and a cross-expert orthogonal loss that pushes common and specific expert outputs apart. The combined loss is L_MoE = μ·L_CM + λ·L_CE (the individual values of μ and λ are not reported in the provided content).

Second, the TaMOT pipeline handles all tasks the same way. A Candidates Generation Module built on a modified SAM2 produces candidate objects: for SOT/VOS the first-frame box or mask seeds a prompt and subsequent high-affinity masks become distractors and candidates; for MOT/MOTS a detection head predicts objects that are fed as multi-object prompts. RoI Align extracts instance features. The Memory-enhanced Module then refines these in two ways — masking background from edges for fine-grained instance embeddings, and modeling spatiotemporal relationships using learnable queries passed through a Q-Former-style architecture with self-attention across instances and self-attention across time. Matching is done with a bi-softmax nearest-neighbor search combining a softmax-based similarity and a cosine similarity, producing an assignment matrix. Training uses MAE, cross-entropy, and L1 losses on the mask decoder, KeepTrack's partial-supervision and self-supervised losses for SOT/VOS, and cross-entropy for MOT/MOTS. Experiments ran on 8 NVIDIA A100 GPUs with Python 3.8 and PyTorch 1.11; inference speed was measured on a single NVIDIA 3090TI.

Why This Matters

Impact on research. The paper argues that prior unified models inherit either modality bias or task bias, and that combining a unified-modality model with a unified-task model (the paper's "SU-Unicorn" and "Flex-UNINEXT" combinations) is not sufficient. SATA's claim is that decoupling shared from modality-specific knowledge, and unifying all task outputs under one instance set, is what makes genuine cross-modal and cross-task knowledge sharing possible. It also targets the practical cost problem — one network, one weight set, one inference pipeline instead of many specialized deployments.

Potential real-world applications (implied by the tasks, not enumerated in the paper):

• Autonomous driving and advanced driver assistance, where RGB, depth, and event sensors must all track and segment pedestrians, vehicles, and lanes.
• Surveillance and security monitoring using RGB-Thermal cameras for day/night operation.
• Robotics and drone perception, where a single onboard model needs to track and segment objects in whatever sensor suite is available.
• Video editing, content creation, and AR/VR, where per-object masks and persistent identities must be maintained across frames.

Industry relevance. A single shared-parameter model covering four subtasks and four modality combinations reduces the need to train, tune, and deploy a separate network per sensor-task pair — the exact fragmentation the paper identifies as the motivation for unified models. The use of SAM2 as a foundation model also means the approach builds on widely available pre-trained weights.

Future Directions

• Closing remaining multi-modal gaps. The paper reports the largest ablation drops for multi-modal tasks (LasHeR, LLE-VOS, UniRTL) when CpMoE or SaMoE is removed, suggesting modality-specific modeling is not yet fully solved.
• Balancing training data across subtasks. Figure 2(a) shows inconsistent quality and scale of training data across multi-modal subtasks; the authors identify this as a source of task and modality bias, and no rebalancing method is proposed.
• Expanding beyond the tested scope. The framework is demonstrated on 4 subtasks and 4 modality types (RGB, RGB-T, RGB-D, RGB-E); whether it extends to other sensors or other video understanding tasks is untested.
• Efficiency and deployment. Training used 8 NVIDIA A100 GPUs; inference was measured on a single NVIDIA 3090TI, but runtime/latency numbers and model size are not reported in the provided content, leaving practical deployment cost an open question.

Target Audience

Researchers and graduate students working on video object tracking, video object segmentation, multi-modal fusion, or Mixture-of-Experts architectures. It is also relevant to applied engineers building perception systems that must run on heterogeneous sensor inputs, and to readers following the broader push toward generalist video understanding models. Readers without a background in Transformer-based tracking and memory-based segmentation will find the method section demanding.

Authors’ abstract

Tracking and segmentation play essential roles in video understanding, providing basic positional information and temporal association of objects within video sequences. Despite their shared objective, existing approaches often tackle these tasks using specialized architectures or modality-specific parameters, limiting their generalization and scalability. Recent efforts have attempted to unify multiple tracking and segmentation subtasks from the perspectives of any modality input or multi-task inference. However, these approaches tend to overlook two critical challenges: the distributional gap across different modalities and the feature representation gap across tasks. These issues hinder effective cross-task and cross-modal knowledge sharing, ultimately constraining the development of a true generalist model. To address these limitations, we propose a universal tracking and segmentation framework named SATA, which unifies a broad spectrum of tracking and segmentation subtasks with any modality input. Specifically, a Decoupled Mixture-of-Expert (DeMoE) mechanism is presented to decouple the unified representation learning task into the modeling process of cross-modal shared knowledge and specific information, thus enabling the model to maintain flexibility while enhancing generalization. Additionally, we introduce a Task-aware Multi-object Tracking (TaMOT) pipeline to unify all the task outputs as a unified set of instances with calibrated ID information, thereby alleviating the degradation of task-specific knowledge during multi-task training. SATA demonstrates superior performance on 18 challenging tracking and segmentation benchmarks, offering a novel perspective for more generalizable video understanding.