OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedback

AI summary

Overview

Research area: Computer Vision — Multi-Object Tracking (MOT) for omnidirectional / 360° panoramic imagery, with a focus on embodied robotic perception.

Technical level: Advanced. The paper assumes familiarity with tracking paradigms (tracking-by-detection, end-to-end tracking), attention-based detection transformers, state space models, and mixture-of-experts architectures.

Scope: The paper introduces OmniTrack++, a unified, feedback-driven framework for panoramic multi-object tracking, alongside the EmboTrack benchmark for evaluating tracking on mobile robotic platforms under 360° fields of view.

What This Paper Is About

Standard multi-object tracking algorithms are built for narrow-field-of-view (pinhole) cameras and break down on panoramic 360° imagery, where equirectangular projection causes geometric distortion, resolution loss, lighting inconsistency, and an enormous search space for targets. OmniTrack++ tackles this by recycling trajectory information from previous frames back into the detector as a feedback signal, stabilizing both localization and identity association across time. The paper also introduces a new benchmark, EmboTrack, captured by quadrupedal and wheel-legged robots, to test tracking under realistic dynamic motion.

Key Contributions

1. ExpertTrack Memory with a shared Mixture-of-Experts design. A hierarchical memory that combines a Stable Identity Memory (long-term, confidence-selected keyframes) with a Dynamic Interaction Memory (short-term appearance and motion updates), routed through specialized experts that each compensate for a distinct type of panoramic variation such as illumination shift or geometric deformation.

2. Refined Tracklet Management with adaptive paradigm switching. A mechanism that dynamically routes object proposals to either an End-to-End (E2E) tracking branch or a classical Tracking-by-Detection (TBD) branch — or fuses both — depending on scene dynamics and target behavior, rather than committing to a single paradigm.

3. The EmboTrack benchmark. A new panoramic MOT dataset of 44 sequences and 26,400 annotated frames spanning five campuses, comprising QuadTrack (quadrupedal robot, 360°×70° camera) and the newly added BipTrack (bipedal wheel-legged robot with an Insta360 X5). It is 37.5% larger than the prior QuadTrack dataset and captures real gait-induced motion perturbations.

4. A unified feedback architecture built on four cooperative modules. The DynamicSSM Block for distortion and photometric stabilization, FlexiTrack Instances for trajectory-informed short-term association, ExpertTrack Memory for long-term identity, and Tracklet Management for adaptive association — all connected in a closed perception loop.

Main Findings

• Large HOTA gains over the prior OmniTrack. OmniTrack++ improves HOTA by +3.94 (21.56 → 25.50) on the JRDB dataset and by +15.03 (19.87 → 34.90) on the challenging QuadTrack test set.

• Strong performance in both tracking paradigms on QuadTrack. Under the E2E paradigm it reaches HOTA 34.90 and IDF1 41.21 (43% and 52% relative improvements over the original OmniTrack). Under the TBD paradigm it achieves HOTA 36.08 and IDF1 42.76 (35% and 38% relative gains).

• Long-term memory is the main driver on long-trajectory data. The largest improvements appear on QuadTrack, which contains long, continuous trajectories — evidence that the ExpertTrack Memory successfully injects long-range identity cues into the feedback loop and prevents drift during occlusion and re-entry.

• Feedback provably reduces tracking uncertainty. The authors formalize the feedback mechanism using entropy: the conditional entropy of detections given prior trajectory feedback, H_fb, is strictly less than the entropy of independent per-frame matching, H_ind, meaning trajectory conditioning monotonically constrains uncertainty over time.

• Panoramic distortion is handled implicitly, not geometrically. The DynamicSSM Block does not explicitly model panoramic geometry. Instead it predicts a distortion field and scale prior, applies distortion-aware dynamic convolution, and uses a multi-directional state space model to enforce long-range photometric consistency — a plug-in that integrates into a DAB Transformer encoder.

• Regularized trajectory instances generalize better. FlexiTrack Instances inject Gaussian noise into both the feature vector and anchor during training, preventing over-reliance on historical cues and improving robustness to unseen motion patterns.

Methodology in Plain English

The core idea is a closed feedback loop. At each frame, the network's backbone and a distortion-correcting module (DynamicSSM) produce features for the current frame. Simultaneously, the system retrieves "FlexiTrack Instances" — compact representations of each existing track's feature and spatial anchor — from the ExpertTrack Memory, which stores past appearance information. These two sets of tokens are concatenated and fed into a shared decoder, so the model simultaneously detects new objects and reasons about where existing tracks should be.

A Dual-Branch Adapter then decides how to update the tracks. If the scene is stable and objects are slow-moving, the E2E branch can be used, which simply thresholds the decoder's predictions by confidence. If objects are moving fast, occluded, or reappearing, the TBD branch computes distance-based associations between new detections and existing tracks. When both are triggered, an Ensemble Module fuses the outputs. The resulting track set is written back into memory, and the cycle repeats for the next frame.

The memory itself is split in two. Half the slots hold high-confidence keyframes for stable long-term identity; the other half hold recent frames for adapting to rapid appearance changes. A router selects the most relevant memory entries across both halves and passes them to a Mixture-of-Experts module, where different experts specialize in compensating for different distortions or lighting conditions. The aggregated expert output becomes the FlexiTrack Instance for the next frame.

Why This Matters

Impact on research: The paper is the first to define panoramic multi-object tracking as a distinct problem and to unify the E2E and TBD paradigms within a single adaptive framework. Its entropy-based justification for trajectory feedback provides a theoretical framing that other tracking systems could adopt, and the EmboTrack benchmark fills a genuine gap — existing MOT datasets (MOT17, KITTI, nuScenes) assume static or smoothly moving narrow-FoV sensors, not the pitch oscillations and gait perturbations of legged robots.

Real-world applications:

• Legged and wheel-legged robot navigation: A quadruped or bipedal robot navigating a crowd needs to track pedestrians and obstacles across its full surroundings without blind spots; OmniTrack++ is designed for exactly this motion profile.
• Autonomous driving and smart vehicles: 360° surround perception for detecting and maintaining identities of vehicles, cyclists, and pedestrians over long time spans, including during occlusion by other traffic.
• Security and surveillance: Continuous identity-preserving tracking across a full room or plaza with a single panoramic camera, eliminating the handoff problem between multiple narrow-FoV cameras.
• Human-robot interaction and service robotics: Maintaining consistent awareness of multiple people around a robot, enabling it to follow, avoid, or address individuals reliably even after temporary occlusion.

Industry relevance: The method targets small mobile platforms where sensor payload matters. A single panoramic camera replaces multiple sensors, which lowers hardware cost and weight — attractive for commercial robotics, warehouse automation, and consumer service robots. The adaptive E2E/TBD switching also offers a practical efficiency-versus-robustness tradeoff that deployment engineers can tune per scene.

Future Directions

• Scaling the benchmark further and toward pre-adaptation. The authors explicitly note that BipTrack's irregular gait trajectories enable research into domain generalization and transfer learning for robotic tracking — testing whether models trained on one robot's motion generalize to another's.

• Explicit geometric modeling versus learned implicit correction. DynamicSSM corrects distortion implicitly. Whether combining it with explicit spherical geometry priors (HEALPix sampling, spherical convolutions) would yield further gains remains an open question.

• Extending to other panoramic perception tasks. The feedback mechanism and ExpertTrack Memory are task-agnostic in principle; applying them to panoramic segmentation, depth estimation, or flow estimation could be productive.

• Real-time deployment and efficiency. The paper reports accuracy, not latency. Whether the dual-branch adapter, MoE routing, and multi-directional SSM scanning can run in real time on robot-embedded compute is an unresolved practical question.

• Multi-modal fusion. Related works cited (HumanoidPano, Humanoid Occupancy) fuse panoramic imagery with LiDAR. Incorporating depth or LiDAR into the OmniTrack++ feedback loop could resolve the inaccurate distance estimation that plagues small, distant targets in wide-FoV views.

Target Audience

This paper is most valuable to computer vision researchers working on multi-object tracking, omnidirectional or spherical perception, and transformer-based detection architectures. It is also directly relevant to robotics engineers building perception stacks for legged, wheeled, or mobile platforms, and to dataset builders interested in how EmboTrack was constructed and annotated. Practitioners evaluating tracking paradigms (E2E vs. TBD) for real deployment will find the adaptive switching analysis and the paradigm comparison in Figure 2 useful, though the entropy formulations and SSM/MoE details make the methodology sections best suited to readers with graduate-level familiarity with deep learning architectures.

Authors’ abstract

To address panoramic distortion, large search space, and identity ambiguity under a 360° FoV, OmniTrack++ adopts a feedback-driven framework that progressively refines perception with trajectory cues. A DynamicSSM block first stabilizes panoramic features, implicitly alleviating geometric distortion. On top of normalized representations, FlexiTrack Instances use trajectory-informed feedback for flexible localization and reliable short-term association. To ensure long-term robustness, an ExpertTrack Memory consolidates appearance cues via a Mixture-of-Experts design, enabling recovery from fragmented tracks and reducing identity drift. Finally, a Tracklet Management module adaptively switches between end-to-end and tracking-by-detection modes according to scene dynamics, offering a balanced and scalable solution for panoramic MOT. To support rigorous evaluation, we establish the EmboTrack benchmark, a comprehensive dataset for panoramic MOT that includes QuadTrack, captured with a quadruped robot, and BipTrack, collected with a bipedal wheel-legged robot. Together, these datasets span wide-angle environments and diverse motion patterns, providing a challenging testbed for real-world panoramic perception. Extensive experiments on JRDB and EmboTrack demonstrate that OmniTrack++ achieves state-of-the-art performance, yielding substantial HOTA improvements of +3.94 on JRDB and +15.03 on QuadTrack over the original OmniTrack. These results highlight the effectiveness of trajectory-informed feedback, adaptive paradigm switching, and robust long-term memory in advancing panoramic multi-object tracking. Datasets and code will be made available at https://github.com/xifen523/OmniTrack.