ReferTrack: Referring Then Tracking for Embodied Visual Tracking

AI summary

Overview

• Research area: Robotics — embodied visual tracking (EVT) with vision-language-action (VLA) models; also touches on vision-language navigation, referring expression comprehension, and sim-to-real robot deployment.
• Technical level: Intermediate. The core ideas (pick one object from a list, then predict movement) are intuitive, but the implementation assumes some familiarity with LLM token training, VLA policies, and bounding-box grounding.
• Scope: A single paper proposing a "referring-then-tracking" VLA policy that identifies a language-described pedestrian by selecting an indexed bounding box from the current forward-view detections, then predicts tracking waypoints conditioned on that selection.

What This Paper Is About

Embodied visual tracking asks a mobile robot with only onboard cameras to continuously follow one specific person described in natural language — for example, "track the person in the red and yellow superhero suit." The hard part is that in a crowd, the robot must decide which of many pedestrians is the right one before it can plan how to move, and existing VLA policies often make that decision inside abstract spatial latent codes that are hard to supervise and poorly tied to what the camera actually sees. This paper's goal is to replace that abstract reasoning with an explicit, image-grounded choice: the model picks one bounding box from an indexed catalog of current detections, then plans its trajectory given that choice.

Key Contributions

1. A "referring-then-tracking" paradigm for EVT. Instead of reasoning in abstract spatial latents, ReferTrack formulates target identification as a constrained multiple-choice problem over an indexed set of image-space bounding boxes, emitted as a single Refer-CoT token from the vocabulary {⟨ped₁⟩, …, ⟨ped_K⟩, ⟨NO_EXIST⟩}, where ⟨NO_EXIST⟩ signals the target is absent from the current forward view.
2. Temporal-viewpoint-bbox indicator (TVBI) tokens. ReferTrack keeps a first-in-first-out queue of previously selected target bounding boxes (capacity H−1) and injects their geometric features into the historical visual stream via TVBI tokens, extending the NavFoM-style temporal-viewpoint indicator (TVI) encoding with target geometry. The current frame's fine tokens are deliberately TVI-only, forcing the model to ground the target from history and raw pixels.
3. A custom Refer-QA dataset for grounding. 1.3M referring samples are synthesized from the SYNTH-PEDES person ReID dataset using the identical indexed-bbox catalog interface, then co-trained at a 1:1 ratio with 1.3M navigation samples so that static referring skill transfers to online tracking.
4. State-of-the-art single-view results and real-robot validation. With a 4B-parameter backbone and supervised fine-tuning only (no reinforcement learning), ReferTrack leads the single-view EVT-Bench comparison and is deployed on a Unitree Go2 quadruped and a Unitree G1 humanoid.

Main Findings

• Top single-view performance on EVT-Bench: 89.4% / 92.5% / 1.6 (SR / TR / CR) on Single-Target Tracking, 73.3% / 81.8% / 7.6 on Distracted Tracking, and 74.1% / 85.7% / 7.7 on Ambiguity Tracking.
• Large gains over the strongest single-view baseline, TrackVLA++: +6.8 SR and +13.0 TR on Distracted Tracking, and +22.9 SR and +22.3 TR on Ambiguity Tracking.
• Competitive with multi-camera reference systems: on the identification-heavy DT and AT splits, ReferTrack matches or exceeds reported multi-camera baselines such as CoMaTrack (74.2/80.5/2.1 on DT; 57.5/73.4/12.0 on AT) and TrackVLA++ multi-camera (74.0/73.7/3.51 on DT; 55.9/63.8/15.1 on AT), suggesting explicit image-space referring can compensate for narrower camera coverage.
• Identification, not planning, is the bottleneck: an oracle variant with ground-truth target bounding boxes reaches 81.5% SR / 84.7% TR on DT, versus 73.3% / 81.8% for the full model and 85.1% SR for the expert policy that sees all simulator states — the gap between oracle and full model is larger than between oracle and expert.
• Both components matter, but unequally: removing TVBI drops DT performance to 70.4% SR / 80.8% TR, while removing both Refer-CoT and TVBI collapses it to 55.7% SR / 71.4% TR. Explicit image-space selection is the primary source of robustness; TVBI stabilizes the selected target over time.
• Training recipe: 1.3M navigation samples (330K each for STT and AT, all 640K DT samples) plus 1.3M Refer-QA samples, two-stage SFT — Stage 1 trains only the vision projector for one epoch at 1×10⁻⁴, Stage 2 jointly trains for 20K steps with global batch size 256 at 2×10⁻⁵ for the LLM and 1×10⁻⁴ for other trainable modules.
• Real-world deployment works: the full perception-and-control loop runs at an average of 10.6 Hz on a cloud GPU server with detection taking 12 ms per step; the Go2 quadruped followed a target through cluttered obstacles even when the narrow camera field of view captured only the target's lower body, and the G1 humanoid referred the correct target under multi-person interference.

Methodology in Plain English

ReferTrack treats the robot's decision as two steps in sequence rather than one blended process.

First, a standard real-time detector ("YOLO11 + ByteTrack") finds pedestrians in the current forward-facing frame, and these are sorted into an indexed catalog — top-K by bounding-box area, plus a permanent ⟨NO_EXIST⟩ slot. Each catalog entry is turned into tokens: an identifier token like ⟨ped_k⟩ followed by a bounding-box token produced by a small MLP projector. Because the same projector is used for both catalog entries and TVBI history, the representation is shared.

Second, the model runs a single LLM forward pass over the language instruction, the catalog, and the encoded visual history. In that pass it emits one token — the Refer-CoT — naming the index that matches the instruction, or ⟨NO_EXIST⟩ if the target is not visible. A second forward pass uses that Refer-CoT token as a conditioning prefix to produce an action token, which an MLP head decodes into waypoints (each an (x, y, θ) egocentric displacement and heading change on the ground plane).

Memory is handled with two queues. History frames are encoded with SigLIP and DINOv2 features, grid-pooled into 64 fine tokens for the current frame and 4 coarse tokens for history, and kept in a sliding window of the latest H frames. An interleaved TVBI token per historical frame carries the geometric feature of the target's previously selected bounding box; frames where the target was unseen get an all-zero sentinel box so the model can tell real motion from missing-target history. The newly selected box is appended to a FIFO queue of capacity H−1 for use in later steps. During training, the queue is filled with ground-truth annotations and occasionally corrupted with a wrong index to simulate tracking errors; at inference it updates autoregressively.

Training combines a trajectory MSE loss (scaled by α = 10), a cross-entropy loss on the Refer-CoT token, and a text cross-entropy loss on Refer-QA samples that bypasses the action head. The Refer-QA data is built by cropping backgrounds to 384×384 and compositing 2–3 captioned pedestrian crops at random scales in [0.75, 1.5], each assigned a catalog ID from 0 to 19, with one absent negative caption represented by an all-zero box.

Deployment runs the model as a WebSocket service on a remote GPU server; the robot streams JPEG-compressed frames from an Intel RealSense D455 camera, and a pure-pursuit controller turns the predicted trajectory into velocity commands.

Why This Matters

The paper's central claim — that making the reasoning step an explicit pick from image-space candidates is better than reasoning in abstract latents — has direct implications for how VLA policies are built. It suggests that a 4B model trained with plain supervised fine-tuning can match or beat both larger models and multi-camera setups on disambiguation-heavy tasks, which changes the cost calculus for anyone building robot policies: less model scale, fewer cameras, and no RL stage.

It also points at a scalable supervision path. Because the referring interface is the same one used by web-scale referring and grounding data and by 2D pedestrian annotations in robotic and autonomous-driving egocentric video, identification capability can in principle be improved without collecting more scarce closed-loop navigation trajectories.

Real-world applications:

• Assistive and companion robotics: the paper frames EVT as fundamental to long-term companionship and socially aware interaction, which covers following a specific person in a care or home setting.
• Service and hospitality robots: following an instructed guest through a crowded lobby, museum, or event venue where many people are present.
• Legged and humanoid field robots: the demonstrated Go2 and G1 rollouts target cluttered indoor and outdoor spaces where a single narrow field-of-view camera is all the robot carries.
• Security, patrol, and logistics: maintaining a prescribed following distance behind a designated worker or courier in dynamic environments, with collision-freeness as a tracked metric.

Industry relevance: the paper targets exactly the constraints industrial robotics groups face — limited onboard sensing, latency budgets (10.6 Hz loop, 12 ms detection), and the cost of RL refinement. The demonstrated sim-to-real transfer from Habitat 3.0 to physical legged and humanoid hardware, plus the released code at https://github.com/MedlarTea/referTrack, makes it a practical reference point for teams building person-following capabilities.

Future Directions

• Closing the identification gap. The oracle TVBI variant reaches 81.5% SR on DT versus 73.3% for the full model, leaving a clear margin to recover through better referring rather than better planning. The paper notes that strong tracking or person re-identification modules could also supply this target-specific guidance.
• Scaling identification supervision beyond scarce trajectories. The authors explicitly propose leveraging web referring and grounding data, plus egocentric video with 2D pedestrian annotations from robotics and autonomous driving, to populate candidate catalogs without relying only on costly closed-loop navigation data.
• Reducing dependence on RL and extra cameras. Since the single-view results already match or exceed several multi-camera baselines on the ambiguity-heavy splits, an open question is how much of the reported benefit from multi-camera rigs and RL fine-tuning is now redundant — and what a combined approach would achieve.
• Robustness of the catalog interface itself. The pipeline depends on an off-the-shelf detector and on top-K bounding-box area ranking when more than K pedestrians are present; behavior under missed detections, tight overlaps, or long occlusions is not characterized in the reported results, and the paper does not report latency or accuracy figures for those failure regimes.

Target Audience

Robotics and embodied-AI researchers working on VLA policies, person following, and language-guided navigation; engineers building sim-to-real pipelines for legged or humanoid platforms; and graduate students who want a concrete example of replacing abstract chain-of-thought latents with a discrete, image-grounded decision interface. A reader with basic familiarity with transformers and bounding-box detection will get the most out of it. Readers unfamiliar with VLA architectures will still follow the high-level argument, since the paper's core idea — choose a box, then plan a path — is conceptually simple.

Authors’ abstract

Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.