Real-World Point Tracking with Verifier-Guided Pseudo-Labeling

AI summary

Overview

• Research area: Computer vision, specifically long-term point tracking (tracking arbitrary physical points through video frames) and sim-to-real adaptation via self-training.
• Technical level: Advanced. The method relies on transformer decoders, deformable attention, contrastive/soft-label objectives, and ensemble teacher-student fine-tuning, and assumes familiarity with benchmarks like TAP-Vid.
• Scope: The paper proposes a learned "verifier" meta-model that scores the per-frame reliability of trajectories produced by several pretrained point trackers, uses those scores to build pseudo-labels for fine-tuning on unlabeled real video, and can also act as an inference-time ensemble.

What This Paper Is About

Point trackers are typically trained on large synthetic datasets, and their accuracy drops on real videos because real footage has different appearance statistics, nonrigid motion, occlusion patterns, lighting and sensor artifacts, while dense frame-accurate trajectory labels for real video do not exist. Self-training with pseudo-labels from pretrained "teacher" trackers is a practical workaround, but no single teacher is uniformly reliable across frames and scenes, and naïve random selection or averaging propagates errors. The paper's goal is to learn when and where to trust each teacher, producing cleaner pseudo-labels so a tracker can be adapted on unlabeled real video with less data.

Key Contributions

1. A verifier meta-model. A learned model that takes a query point plus candidate trajectories from multiple pretrained trackers and outputs per-frame reliability distributions over those candidates, addressing both training-time supervision selection and optional inference-time ensembling. It is trained entirely on synthetic data with ground-truth trajectories and no real-world annotations.
2. A verifier-guided pseudo-labeling framework. At each frame, the candidate with the highest reliability score is selected as the pseudo-label, and visibility is set by majority voting across teachers, giving supervision for fine-tuning on unlabeled real videos while mitigating the failure modes of naïve pseudo-labeling.
3. Extensive experiments and ablations. Evaluations on four real-world benchmarks (EgoPoints, RoboTAP, TAP-Vid Kinetics, TAP-Vid DAVIS) show state-of-the-art results, plus ablations on teacher composition and on mixing synthetic and real data, highlighting data efficiency relative to prior self-training methods.

Main Findings

• Headroom exists for adaptive selection. An oracle that picks the closest-to-ground-truth teacher prediction at each frame outperforms every individual teacher across four real-world datasets, and the gap is particularly pronounced on challenging egocentric videos such as EgoPoints. Individual teachers cluster below the oracle, while a random-selection baseline sits well below it too.
• No single teacher dominates. Performance rankings vary substantially by dataset: for example, BootsTAPNext ranks last on RoboTAP but second on DAVIS.
• Verifier as an inference-time ensemble works. Across the four real-world datasets, the verifier consistently beats both the random-teacher baseline and the strongest individual teacher, with gains most pronounced on EgoPoints.
• Main results (Track-On-R). On EgoPoints, δ_avg^x rises from 61.7 (synthetic baseline) to 67.3, widening the gap to the closest competitor, AllTracker, by +5.3 in δ_avg^x (OA 90.2). On RoboTAP, AJ reaches 70.9, δ_avg^x 82.6, OA 94.0, a +2.8 improvement in AJ over the synthetic baseline. On TAP-Vid Kinetics, AJ is 57.8 and δ_avg^x 71.0 with OA 90.5, improving +2.5 in AJ over the synthetic baseline, surpassing the highest reported AJ of 57.3 (BootsTAPNext) and achieving 3.1 points higher OA. On TAP-Vid DAVIS, AJ is 68.1, δ_avg^x 80.3, OA 92.5, a +1.1 AJ improvement over Track-On2.
• Transfer without domain overlap. The real-world training set contains no robotic sequences, yet the adapted model attains the highest scores on the RoboTAP benchmark; similarly, it improves strongly on egocentric EgoPoints even though the training data are not explicitly ego-centric.
• Verifier tolerates weaker teachers. In the teacher-composition ablation on TAP-Vid DAVIS and RoboTAP, the verifier consistently beats random (round-robin) selection. In DAVIS, adding teacher D to the set {A, B, C} lowers the random baseline (79.4 to 77.5) but raises verifier accuracy (80.6 to 80.8). With all five teachers, verifier δ_avg^x is 81.1 on DAVIS and 83.1 on RoboTAP.
• Data schedule matters modestly. In the data-composition ablation, training only on real data yields the best localization on some settings (EgoPoints δ_avg^x 66.9, Kinetics 70.9, DAVIS 80.3), the mix improves OA thanks to synthetic visibility supervision (EgoPoints OA 90.7), and the scheduled mix (gradually increasing real-data loss weight) gives the best overall numbers (EgoPoints δ_avg^x 67.3 / OA 90.2; RoboTAP 82.6 / 94.0; Kinetics 71.0 / 90.5; DAVIS 80.3 / 92.5). The paper notes adding synthetic data on top of real-only training provides only marginal gains.
• Missing entries are acknowledged. Some results, such as CoTracker3 (Window) on EgoPoints under synthetic pretraining, are reported as "—" because no checkpoints or published values were available.

Methodology in Plain English

The core idea is to add a referee to a team of trackers.

Training the referee. The verifier never predicts a trajectory itself. Instead, during training it is shown synthetic videos with known ground-truth trajectories. The ground-truth path is deliberately corrupted into several alternative candidate paths (typically 6–12) by introducing errors that resemble real tracker failures: drift, jumps, occlusions, re-appearances, jitter. The target for each candidate is a soft distribution over how close it is to the truth, computed as a softmax over negative distances with sharpness τ_s = 0.3. The verifier is trained with a cross-entropy loss summed over frames, masking out occluded frames (visibility v_t), to predict this distribution.

How it looks at video. A frozen CNN encoder from pretrained CoTracker3 extracts dense visual features from every frame. The query point's appearance at its first visible frame is sampled as a reference descriptor. Deformable attention then aggregates local context around the query location and around each candidate location in each frame, always relative to that same query appearance descriptor, so descriptors are comparable. Positional embeddings of the displacement from the query and a learned identity embedding distinguishing "query" from "candidate" are concatenated and projected.

Reasoning across candidates and time. A candidate transformer decodes the replicated query feature against the candidate features. Its cross-attention is restricted per frame: each frame's query attends only to the candidates at that same frame (time treated as a batch axis), preventing cross-frame leakage. Temporal self-attention along the trajectory length then lets frames inform one another, so an uncertain frame can be refined by confident nearby frames. There is no attention mask, and all candidates are included even in occluded frames.

Turning scores into labels. The output reliability for each frame is a temperature-scaled softmax (τ = 0.1) over cosine similarities between the decoded query feature and each candidate feature. At inference, the candidate with the highest score at each frame becomes the pseudo-label, and visibility comes from majority voting across teachers.

Real-world setup. At inference, candidates come from M = 6 pretrained teachers: Track-On2, BootsTAPIR, BootsTAPNext, Anthro-LocoTrack, AllTracker, and CoTracker3 (window variant). The synthetic pretraining data for the verifier is K-EPIC (11K videos, each 24 frames). For fine-tuning, the real dataset is built from TAO, OVIS, and VSPW, keeping videos longer than 48 frames with no other filtering, yielding 4864 sequences. Query points are sampled 2/3 from SIFT detections and 1/3 from motion-salient regions found by grayscale frame differencing with mild spatial smoothing, drawn from the first half of each video at four uniformly spaced frames (e.g., frames 1, 5, ...). The base tracker is Track-On2 pretrained on TAP-Vid Kubric, then fine-tuned on a gradually reweighted mixture of synthetic (ground-truth) and real (pseudo-labeled) data. In total, the paper uses 8K videos for real-world fine-tuning.

Evaluation. Four real-world benchmarks are used: TAP-Vid DAVIS (30 real-world videos from DAVIS), TAP-Vid Kinetics (1000 videos from the validation split of Kinetics-700-2020), RoboTAP (265 robotic sequences averaging over 250 frames each), and EgoPoints. Metrics are Occlusion Accuracy (OA), δ_avg^x (average proportion of visible points tracked within 1, 2, 4, 8, and 16 pixel thresholds), and Average Jaccard (AJ); on EgoPoints, δ_avg and OA are reported following official evaluation. Videos are downsampled to 256×256 except EgoPoints, whose frames are resized to 384×512 but evaluated at 256×256. All models are evaluated in the queried-first (causal) setting, where the first visible point is the query.

Why This Matters

Impact on research. The paper reframes pseudo-labeling for point tracking from "pick a teacher" to "learn to judge teachers," showing that a meta-model trained only on synthetic perturbations can estimate reliability on real video and transfer across domains (including to robotic and egocentric footage never seen in training). It provides a unified mechanism for supervision selection and for plug-and-play inference-time ensembling, and it argues for data efficiency relative to prior self-training approaches such as BootsTAPIR and BootsTAPNext, which rely on large-scale distillation over millions of real videos, and CoTracker3's generic random-teacher scheme.

Real-world applications (drawn from the applications the paper lists for point tracking):

• Video editing, where points must be followed consistently over long clips.
• Robot manipulation and perception, which require reliable motion cues (RoboTAP is the robotic benchmark used here).
• 4D scene understanding, which depends on long-range correspondence across views and time.
• Medical analysis, where tracking points through sequences supports measurement and monitoring.

Industry relevance. The method adapts an off-the-shelf tracker using only raw in-the-wild video, built from existing object tracking and segmentation datasets (TAO, OVIS, VSPW), avoiding the cost of dense frame-accurate trajectory annotation. It also lets practitioners combine several already-deployed trackers into a stronger ensemble without retraining each one, and the reported gains in occlusion accuracy matter for downstream automation where visibility errors are costly.

Future Directions

• Data curation as a bottleneck. The conclusion states that fine-tuning effectiveness still depends on the quality and diversity of the available video data, highlighting the importance of curating that video collection (the paper's own text is truncated at this point).
• Scaling and generalizing the teacher pool. Only five teachers appear in the composition ablation and six at inference; how far verifier selection scales to larger, more heterogeneous tracker pools remains open.
• Reducing reliance on synthetic supervision during adaptation. The ablation shows real-only training already gives competitive localization and that adding synthetic data gives only marginal gains, raising the question of how much synthetic data is needed, and whether verifier training itself could be extended beyond synthetic perturbation targets.
• Extending the reliability framework beyond point tracking. The authors position the verifier as a general framework for model selection and uncertainty estimation in video correspondence tasks, which invites testing on other dense correspondence problems.

Target Audience

Researchers and practitioners working on video correspondence, point tracking, and test-time or self-supervised adaptation; engineers who need to adapt pretrained trackers to their own unlabeled video collections or to combine multiple existing trackers at inference; and readers interested in learned reliability estimation and ensemble selection as an alternative to fixed confidence thresholds.

Authors’ abstract

Models for long-term point tracking are typically trained on large synthetic datasets. The performance of these models degrades in real-world videos due to different characteristics and the absence of dense ground-truth annotations. Self-training on unlabeled videos has been explored as a practical solution, but the quality of pseudo-labels strongly depends on the reliability of teacher models, which vary across frames and scenes. In this paper, we address the problem of real-world fine-tuning and introduce verifier, a meta-model that learns to assess the reliability of tracker predictions and guide pseudo-label generation. Given candidate trajectories from multiple pretrained trackers, the verifier evaluates them per frame and selects the most trustworthy predictions, resulting in high-quality pseudo-label trajectories. When applied for fine-tuning, verifier-guided pseudo-labeling substantially improves the quality of supervision and enables data-efficient adaptation to unlabeled videos. Extensive experiments on four real-world benchmarks demonstrate that our approach achieves state-of-the-art results while requiring less data than prior self-training methods. Project page: https://kuis-ai.github.io/track_on_r