TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels

AI summary

Overview

• Research area: Computer vision — monocular 3D point tracking, dynamic-scene reconstruction, and camera pose estimation.
• Technical level: Advanced (assumes familiarity with bundle adjustment, camera intrinsics/extrinsics, depth priors, and 2D point trackers).
• Scope: A pipeline that turns a single monocular video into dense 3D point trajectories for almost all pixels of all frames, expressed in a world-centric coordinate frame with jointly estimated camera poses.
• Paper identity: arXiv:2512.08358v1 [cs.CV], 09 Dec 2025, CC BY 4.0. Authors: Jiahao Lu, Weitao Xiong, Jiacheng Deng, Peng Li, Tianyu Huang, Zhiyang Dou, Cheng Lin, Sai-Kit Yeung, Yuan Liu (affiliations listed include HKUST, XMU, USTC, CUHK, HKU, MUST).

What This Paper Is About

Monocular 3D tracking tries to recover how pixels move through 3D space over time from one ordinary video. The authors argue that prior methods have two gaps: they model 3D motion only in the camera's own coordinate frame (so camera motion and object motion are entangled), and they only track sparse points defined in the first frame, missing objects that appear later. TrackingWorld addresses both by producing dense world-centric 3D tracks for almost all pixels across all frames, while explicitly disentangling camera motion from dynamic object motion.

Key Contributions

1. A tracking upsampler applied to arbitrary 2D tracks. The upsampler module from DELTA is shown to generalize beyond DELTA's own tracks, and is used to lift arbitrary sparse 2D tracks into dense 2D tracks for a given frame via a learned weight matrix W, where P_dense = W^T P_sparse.
2. Tracking every frame with redundancy removal. Instead of tracking only the first frame, the pipeline performs 2D tracking and sparse-to-dense upsampling on all frames, then discards pixels that lie near the trajectory of previously visible 2D tracks to avoid redundant computation.
3. An optimization framework that lifts dense 2D tracks into world-centric 3D. It estimates per-frame camera poses in SE(3), then back-projects dense 2D tracks into world coordinates, using an as-static-as-possible constraint (L_asap) to model and filter out unsegmentable dynamic background objects.
4. Explicit tracking of newly emerging and background dynamic objects. Rather than assuming a single foreground dynamic object, the method tracks dynamic background points (those with ||O_static(i, ·)||_2 ≥ ε) in the world-centric frame, enabling "almost all pixels" coverage.

Main Findings

• Camera pose accuracy (Table 1): On Sintel, Bonn, and TUM-D, TrackingWorld (DELTA-based) achieves ATE 0.088 / 0.016 / 0.016, RTE 0.035 / 0.005 / 0.005, and RRE 0.410 / 0.564 / 0.333. With CoTrackerV3 as the tracker, results are ATE 0.103 / 0.016 / 0.014, RTE 0.039 / 0.005 / 0.005, RRE 0.439 / 0.561 / 0.338. For comparison, MonST3R reports ATE 0.111 / 0.029 / 0.063 and Uni4D reports 0.116 / 0.017 / 0.039. The authors note the pipeline is consistently more accurate regardless of whether tracks come from DELTA or CoTrackerV3. Cells marked "/" in the table indicate results not reported for that method/dataset.
• Depth accuracy of dense 3D tracks (Table 2): Ours (DELTA) + UniDepth reaches Abs Rel 0.218 / 0.058 / 0.084 with δ<1.25 of 73.3 / 97.3 / 92.3 on Sintel / Bonn / TUM-D. DELTA with a UniDepth prior alone reports 0.636 / 0.153 / 0.178 and 63.1 / 90.5 / 85.6; with ZoeDepth or Depth Pro priors the numbers are higher (worse) still.
• Sparse 3D tracking (Table 4, evaluated in camera coordinates): On ADT (moving camera), Ours (DELTA) scores AJ 23.4, APD_3D 32.2, OA 90.1, versus DELTA's 15.3 / 22.9 / 90.1 and SpatialTracker's 14.3 / 22.3 / 91.5. On PStudio (static camera), gains are smaller: Ours (DELTA) 15.1 / 25.6 / 75.7 versus DELTA's 15.1 / 24.6 / 75.7. OA stays identical to the underlying tracker because visibility maps are taken directly from DELTA/CoTrackerV3.
• Dense 2D tracking (long-range optical flow, Table 6): CoTrackerV3 + the upsampler ("Up") reaches EPE 1.24 and IoU 80.9 on CVO-Clean, and EPE 1.35 / IoU 80.6 on CVO-Final, comparable to DELTA (1.14 / 78.9 and 1.39 / 78.2). The CVO test set has roughly 500 videos of 7 frames per subset.
• Upsampler efficiency (Table 11, CVO-Clean): CoTrackerV3 alone gives EPE 1.45, IoU 76.8, and average time 3.00 minutes; CoTrackerV3 + Up gives EPE 1.24, IoU 80.9, and 0.25 minutes — roughly a 12× speed-up with better accuracy.
• Ablations on Sintel (Table 8): Removing tracking-every-frame raises ATE from 0.088 to 0.171; removing pose initialization raises ATE to 0.659; removing dynamic object tracking raises Abs Rel from 0.218 to 0.468; removing O_static raises ATE to 0.092 and RRE to 0.459; removing L_dc raises ATE to 0.093; removing inlier selection (N_inliers) raises ATE to 0.089. The full setting is ATE 0.088, RTE 0.035, RRE 0.410, Abs Rel 0.218, δ<1.25 73.3.
• Robustness to depth backbone (Table 9): With DELTA fixed as the tracker, the pipeline improves over raw predictions for all three depth models: Ours (ZoeDepth) ATE 0.093, Abs Rel 0.236; Ours (Depth Pro) ATE 0.101, Abs Rel 0.228; Ours (UniDepth) ATE 0.088, Abs Rel 0.218. Raw ZoeDepth, Depth Pro, and UniDepth report Abs Rel 0.814, 0.813, and 0.636 respectively.
• Robustness to mask source (Table 10): VLM + Grounding-SAM masks give ATE 0.088, RTE 0.035, RPE 0.410, Abs Rel 0.218, δ<1.25 73.3; Segment Any Motion gives ATE 0.093, RTE 0.041, RPE 0.379, Abs Rel 0.224, δ<1.25 73.3 — described as comparable performance.
• Runtime: The full framework takes about 20 minutes to produce dense world-centric 3D tracking for a 30-frame video, run on an RTX 4090 GPU.
• Qualitative results: World-centric dense tracking is visualized on the DAVIS dataset, with keyframe-spaced 3D tracks and continuous tracks across all frames.

Methodology in Plain English

The system starts by running off-the-shelf foundation models on the video: a 2D point tracker (CoTrackerV3 or DELTA's 2D tracking component), a monocular depth estimator (UniDepth), and a foreground dynamic mask segmenter (VLM + Grounding-SAM, or alternatively Segment Any Motion). The paper stresses that none of these predictions need to be accurate.

Then, in two main stages:

1. Densify the 2D tracks and cover the whole video. Sparse 2D tracks are lifted to dense per-pixel tracks using DELTA's upsampler, which predicts a weight matrix relating each dense pixel to its neighboring sparse tracks. This is repeated on every frame, not just the first, so newly appearing objects are captured. Pixels that sit near already-seen trajectories are dropped to cut redundant work.

2. Lift dense 2D tracks into a world-centric 3D frame. Camera poses are first estimated by unprojecting static-region tracks with the depth maps and minimizing a reprojection error; the video is split into C clips and poses are estimated within clips in parallel, then merged between clips. Because the foreground masks miss dynamic background objects, the method adds a per-point, per-time offset term O_static that lets any "static" point drift, and regularizes it with an as-static-as-possible L1 penalty so genuinely static geometry stays still while truly moving background objects are absorbed by the offset. This second stage jointly optimizes camera poses and 3D point locations with a bundle adjustment loss plus a depth-consistency loss. Points whose offsets exceed ε are reclassified as dynamic. Finally, dynamic tracks are initialized from the refined poses and depths and optimized with the reprojection loss, depth consistency, an as-rigid-as-possible loss, and a temporal smoothness loss. Outputs are static 3D tracks, dynamic 3D tracks, and camera poses.

Why This Matters

• Research impact: It reframes monocular 3D tracking as a world-centric problem and argues that explicitly separating camera motion from object motion improves tracking quality itself — the ablation showing a lower ATE with O_static and pose refinement supports this. It also extends dense tracking to objects that appear mid-video, a case earlier methods such as DELTA do not handle.
• Real-world applications (as motivated by the paper):
  • Fine-grained 3D motion analysis of objects in video.
  • High-fidelity novel view synthesis that requires disentangled camera and object motion.
  • Generating temporally and geometrically consistent videos.
  • Video editing and scene understanding built on dense point correspondences.
• Industry relevance: Because the pipeline is optimization-based and built on top of interchangeable pretrained foundation models (multiple trackers, depth models, and segmentation sources all work), it is a modular recipe that can be re-assembled as better components become available. The reported 20-minute runtime for a 30-frame clip and the 12× speed-up from the upsampler are the practical constraints a production system would need to weigh.
• Positioning against prior work: The paper contrasts itself with camera-coordinate methods (OmniMotion, SpatialTracker, DELTA), with world-centric concurrent work (St4RTrack, which the authors say may drift in long-term tracking, and TAPIP3D, which they describe as sparse-only and unable to recover camera motion), and with 4D reconstruction methods such as Uni4D, which the authors say assume a single dynamic foreground object rather than modelling background object motion in pose estimation.

Future Directions

• Reducing runtime. The reported ~20 minutes for a 30-frame video is the pipeline's stated operating cost; making dense world-centric tracking faster is an obvious extension the work sets up but does not itself solve.
• Long-term drift. The paper raises drift as a criticism of a concurrent pairwise-correspondence approach (St4RTrack); whether TrackingWorld's clip-based pose estimation and merging sustains accuracy over much longer sequences is not reported here.
• Residual dependence on foundation models. Ablations show robustness across ZoeDepth, Depth Pro, and UniDepth backbones and across two mask sources, but the upper bound on accuracy still comes from those priors; improving how the optimization corrects poor priors is an open question.
• Quantifying "almost all." The authors define "almost all" as filtering noisy and outlier tracks, but do not report a coverage percentage for tracked pixels, so the exact fraction of pixels tracked is not reported.

Target Audience

Researchers and graduate students working on 3D tracking, structure-from-motion, dynamic-scene reconstruction, or 4D reconstruction who already understand camera pose estimation and bundle adjustment. It is also relevant to practitioners building video editing, motion analysis, or novel view synthesis systems who need dense point correspondences in a consistent world frame and can run an optimization pipeline on a GPU. Readers looking for a beginner-level introduction to 2D point tracking should start with the related-work section's pointers to PIPs, TAP-Vid, TAPIR, CoTracker, and LocoTrack instead.

Authors’ abstract

Monocular 3D tracking aims to capture the long-term motion of pixels in 3D space from a single monocular video and has witnessed rapid progress in recent years. However, we argue that the existing monocular 3D tracking methods still fall short in separating the camera motion from foreground dynamic motion and cannot densely track newly emerging dynamic subjects in the videos. To address these two limitations, we propose TrackingWorld, a novel pipeline for dense 3D tracking of almost all pixels within a world-centric 3D coordinate system. First, we introduce a tracking upsampler that efficiently lifts the arbitrary sparse 2D tracks into dense 2D tracks. Then, to generalize the current tracking methods to newly emerging objects, we apply the upsampler to all frames and reduce the redundancy of 2D tracks by eliminating the tracks in overlapped regions. Finally, we present an efficient optimization-based framework to back-project dense 2D tracks into world-centric 3D trajectories by estimating the camera poses and the 3D coordinates of these 2D tracks. Extensive evaluations on both synthetic and real-world datasets demonstrate that our system achieves accurate and dense 3D tracking in a world-centric coordinate frame.