AerialMind: Towards Referring Multi-Object Tracking in UAV Scenarios

AI summary

Overview

Research area: Referring Multi-Object Tracking (RMOT) — using natural language instructions to detect and track specified objects in video — extended from ground-level scenes to Unmanned Aerial Vehicle (UAV) scenarios. The work sits at the intersection of computer vision, robotics, and Embodied AI, and is published under the arXiv category Robotics.

Technical level: Advanced. The paper assumes familiarity with transformer-based trackers (DETR-style architectures), vision-language fusion, deformable attention, atrous convolution, and tracking metrics such as HOTA, DetA, AssA, and LocA.

Scope: This paper introduces AerialMind, the first large-scale referring multi-object tracking benchmark for UAV scenes, together with a semi-automated annotation framework (COALA), a tracking method (HawkEyeTrack), and attribute-based evaluation metrics.

What This Paper Is About

Referring multi-object tracking lets a system find and follow objects in a video based on a text description such as "black automobiles traveling leftward to rightward." Existing RMOT datasets and methods are almost entirely built around ground-level cameras, so they cannot capture the wide-area context, changing viewpoints, and small objects that aerial platforms produce. The paper's goal is to make language-guided tracking work in UAV scenarios by building a dedicated large-scale dataset, an efficient way to annotate it, and a method designed for the specific difficulties of aerial perception.

Key Contributions

1. AerialMind benchmark dataset: The first large-scale referring multi-object tracking benchmark for UAV scenarios, extended from VisDrone and UAVDT, containing 93 video sequences, 24.6K referring expressions, 293.1K object instances, and up to 46.14M bounding box annotations. It covers cross-domain scenarios, complex reasoning expressions, and fine-grained attribute annotations, plus the first frame-by-frame attribute annotations in the RMOT field.
2. COALA annotation framework: A semi-automated, multi-stage agent-collaborative labeling assistant (Collaborative Agent-based Labeling Assistant) that uses large language models to reduce manual labor costs while maintaining annotation quality, encompassing scene understanding, semi-automated object labeling, consistency checking, and expression expansion.
3. HawkEyeTrack (HETrack) method: A tracking framework combining a Co-evolutionary Fusion Encoder (CFE) for synergistic vision-language representation refinement and a Scale Adaptive Contextual Refinement (SACR) module for improved perception of UAV scenes.
4. Attribute-based evaluation metrics: New composite metrics, HOTA_S (Scene-Robustness) and HOTA_M (Motion-Resilience), that decompose tracking performance across scene-induced and motion-induced challenge attributes.

Main Findings

• Dataset scale and diversity: AerialMind contains 93 video sequences, 24.6K referring expressions, 293.1K instances, and 46.14M bounding box annotations, compared with 9.8K expressions for the larger-scale Refer-KITTI-V2 — less than half of AerialMind. It includes 752 no-target expressions and 458 reasoning expressions, 1.2K words in its vocabulary, and 11.9 instances per expression.
• Frame-level statistics: The dataset spans 48,485 frames, with 247.4 expressions per sequence (second among the compared benchmarks), 7,601 distinct expressions, 8,778 distinct instances, and a temporal ratio per expression of 0.707. It covers over 70 distinct aerial scenarios.
• In-domain tracking performance: HETrack achieves a state-of-the-art HOTA of 31.46%, DetA of 21.57%, AssA of 46.23%, LocA of 82.77%, HOTA_S of 34.37%, and HOTA_M of 31.12%, surpassing MOTR-V2 (19.51 HOTA), TransRMOT (23.54), TempRMOT (26.24), CDRMT (25.81), and MGLT (26.16).
• Cross-domain tracking performance: On the cross-domain test set drawn from UAVDT, HETrack reaches HOTA 31.60%, DetA 21.35%, AssA 47.10%, LocA 83.98%, HOTA_S 27.53%, and HOTA_M 31.93%, again outperforming all compared methods.
• Counterintuitive cross-domain result: Most methods, including HETrack, score higher HOTA on the cross-domain set than in-domain. The authors attribute this to the training domain (VisDrone) having ten distinct object categories and richer semantics, while the cross-domain test set (UAVDT) is predominantly limited to vehicle-only annotations, which simplifies language grounding.
• Attribute-level strengths: HETrack achieves the highest performance across all challenging attributes, with particularly large leads in Low Resolution (38.49%), Fast Motion (35.41%), and Night (35.4%) conditions. The gain in localizing small objects improves overall detection accuracy but leads to a marginal decrease in average localization score (LocA 82.77%).
• Component ablation: Removing both CFE and SACR drops HOTA to 26.41% (DetA 16.43, AssA 42.80). Removing CFE alone gives 28.27% HOTA (DetA 18.53, AssA 43.49); removing SACR alone gives 29.89% HOTA (DetA 19.86, AssA 45.34), versus 31.46% for the full model.
• Fusion strategy ablation: Compared with concatenation (28.88 HOTA), addition (30.39), and cross-attention with sentence-level features (30.52), the proposed fusion reaches 31.46 HOTA with DetA 21.57 and AssA 46.23.
• Ground-level generalization: On Refer-KITTI-V2, HETrack scores 35.40% HOTA, competitive with SKTrack (35.29) and HFF-Track (36.18), and above TransRMOT (31.00), CDRMT (31.99), and TempRMOT (35.04), indicating benefits beyond aerial scenes.
• Annotation cost: The paper states that COALA significantly reduces manual labor cost while maintaining annotation quality, but no quantitative reduction figure is reported.

Methodology in Plain English

Dataset construction. The researchers built AerialMind on top of two existing UAV datasets, VisDrone and UAVDT, covering multiple flight altitudes, environmental conditions, and target categories. Of the sequences, 63 VisDrone sequences form the training set, the remaining 17 VisDrone sequences form the in-domain test set, and 13 representative UAVDT sequences form the cross-domain test set. The dataset is designed around four stated challenges: drastic appearance differences from altitude and viewpoint changes, complex spatial relationships in aerial views, dynamic scene and illumination variation from UAV maneuverability, and richer referring expressions involving spatial, motion, and object descriptions.

COALA annotation pipeline. Four LLM-agent stages reduce human effort. The Scene Understanding Prompt Agent reads key frames and generates a high-level semantic digest of the scene. In the Semi-automated Object Labeling stage, a human annotator reads that digest, picks a templated language description, and clicks twice on a target instance to mark the start and end of the referring event; the SOL-Agent then propagates the bounding box trajectory frame by frame using existing detection boxes. An optional Consistency Check Agent performs cross-modal spatio-temporal logical reasoning over visual features, language descriptions, and motion patterns inferred from trajectories, and returns failing annotations for correction. Finally, the Expression Expansion Agent treats validated expressions as semantic seeds and generates syntactically and lexically different but semantically equivalent new expressions, which then pass a final human verification step.

Tracking method. HETrack follows the encoder-decoder design used by prior RMOT work. A visual backbone (ResNet50) extracts a multi-scale feature pyramid, and a language encoder (RoBERTa) produces word-level and sentence-level text features. The Co-evolutionary Fusion Encoder stacks blocks that each combine a Bidirectional Fusion Layer, where visual and text features mutually update each other, with a Deformable Encoder Layer for intra-modal spatial modeling; the sentence-level feature then modulates the co-evolved visual features. The Scale Adaptive Contextual Refinement module is inserted between encoder and decoder and applies parallel atrous convolutions with dilation rates of 6, 12, and 18 on the highest-resolution feature map to capture multi-scale context without losing spatial resolution, followed by channel-wise recalibration driven by global average pooling and a one-dimensional convolution whose kernel size is determined by the channel dimension. A Semantic Guidance Module performs target-aware modulation of detection queries through cross-attention with word features. Training uses a weighted sum of focal loss, L1 loss, GIoU loss, and a referring loss, deployed on 8 NVIDIA A100 GPUs with batch size 1 for 100 epochs, with a score threshold of 0.5 and a referring matching score threshold of 0.4.

Evaluation. Standard HOTA, defined as the geometric mean of DetA and AssA, is used for overall tracking. HOTA_S aggregates Night, Occlusion, and Low Resolution attributes, while HOTA_M aggregates Viewpoint Change, Scale Variation, Fast Motion, and Rotation, with each composite computed as the geometric mean of the per-attribute HOTA values. Eight challenge attributes are frame-by-frame annotated in the test set.

Why This Matters

Impact on research. AerialMind is presented as the first large-scale RMOT benchmark for UAV scenarios, and the authors note that existing RMOT work is almost entirely confined to ground-level settings. The dataset adds cross-domain coverage, complex reasoning expressions, no-target expressions, and the first frame-by-frame attribute annotations in the field, alongside new diagnostic metrics. The authors also present AerialMind as valuable for pre-training, noting that its distributional diversity may explain the cross-domain performance pattern they observe.

Real-world applications:

• Wide-area aerial surveillance and monitoring: Tracking specified vehicles or people from UAVs using plain-language queries such as "Autos positioned on right of straight road."
• Search and rescue and event response: Rapidly specifying targets in changing scenes, including night conditions and low-resolution imagery where HETrack shows its largest gains (Night 35.4%, Low Resolution 38.49%).
• Traffic monitoring and enforcement reasoning: Expressions such as "Cars invading other lanes" require logical inference rather than simple visual matching, matching the dataset's 458 reasoning expressions.
• Embodied aerial intelligence and natural language interaction: UAVs are described as critical platforms for Embodied AI, creating demand for aerial systems that can act on natural-language instructions and support comprehensive tracking and path planning.

Industry relevance. The work targets drone manufacturers, aerial surveillance and monitoring providers, and robotics companies building language-interfaced autonomous platforms. The COALA framework is also relevant to anyone producing large annotated video datasets, since it uses LLM agents to cut manual annotation effort while keeping a human in the loop for semantic judgment and final verification.

Future Directions

• Fully automated annotation: The Consistency Check Agent is described as optional and as the foundation for fully automated annotation in the future, suggesting a path toward pipelines that need less human correction.
• Generalizing across aerial domains and semantics: Because the cross-domain set (UAVDT) is predominantly limited to vehicle-only annotations, extending evaluation to richer multi-category aerial semantics is a natural next step.
• Long-term temporal modeling: The frame count distribution has long-tail characteristics, with many referring events spanning hundreds of frames, and the dataset's temporal ratio per expression of 0.707 reflects intermittent, event-driven targets — both pointing to a need for stronger temporal event localization.
• Broader language-guided aerial perception: The authors state they hope the work paves the way for future research in aerial language-guided perception, including more complex spatial reasoning and implicit descriptions such as those illustrated in their qualitative results. Specific performance targets, runtime, and model-size considerations are not reported.

Target Audience

Researchers and graduate students working on referring multi-object tracking, vision-language models, UAV and aerial perception, and Embodied AI; engineers building drone-based surveillance, monitoring, or autonomous flight systems that need to follow natural-language instructions; and dataset builders interested in semi-automated, LLM-agent-assisted annotation pipelines. Readers should be comfortable with transformer-based detection and tracking architectures and with tracking evaluation metrics.

Authors’ abstract

Referring Multi-Object Tracking (RMOT) aims to achieve precise object detection and tracking through natural language instructions, representing a fundamental capability for intelligent robotic systems. However, current RMOT research remains mostly confined to ground-level scenarios, which constrains their ability to capture broad-scale scene contexts and perform comprehensive tracking and path planning. In contrast, Unmanned Aerial Vehicles (UAVs) leverage their expansive aerial perspectives and superior maneuverability to enable wide-area surveillance. Moreover, UAVs have emerged as critical platforms for Embodied Intelligence, which has given rise to an unprecedented demand for intelligent aerial systems capable of natural language interaction. To this end, we introduce AerialMind, the first large-scale RMOT benchmark in UAV scenarios, which aims to bridge this research gap. To facilitate its construction, we develop an innovative semi-automated collaborative agent-based labeling assistant (COALA) framework that significantly reduces labor costs while maintaining annotation quality. Furthermore, we propose HawkEyeTrack (HETrack), a novel method that collaboratively enhances vision-language representation learning and improves the perception of UAV scenarios. Comprehensive experiments validated the challenging nature of our dataset and the effectiveness of our method.