ReLift: Dense 3D Tracking as Lifted Trajectory Repair

AI summary

Overview

Research area: Computer Vision — dense 3D point tracking from monocular video, combining 2D point tracking with feed-forward 3D geometry estimation.

Technical level: Advanced. The paper assumes familiarity with camera intrinsics/extrinsics, optical-axis depth, viewing rays, transformer attention, and standard tracking benchmarks.

Scope: ReLift proposes a feed-forward module that repairs and completes 3D point trajectories obtained by "lifting" pretrained 2D tracks with pretrained depth and camera estimates, and shows state-of-the-art mean accuracy on WorldTrack.

What This Paper Is About

Recovering dense 3D point trajectories from a single moving camera normally requires one network to solve three problems at once: matching points across frames, estimating scene geometry, and guessing where points go when they are hidden behind something. The authors observe that pretrained 2D trackers already handle matching very well and pretrained geometry models already handle depth and cameras, so the scarce 3D ("4D") supervision is better spent on a narrower job. ReLift takes 2D tracks, converts them to 3D using estimated depth and cameras, and then learns to repair the corrupted parts of those trajectories and fill in the missing, occluded motion.

Key Contributions

1. A strong lifting baseline. The authors show that simply lifting pretrained 2D tracks into 3D with estimated depth and cameras, discarding points flagged as occluded and interpolating the resulting gaps, already matches or exceeds dedicated 3D trackers that use the same geometry.
2. ReLift, an efficient feed-forward repair module. A compact trajectory corrector alternates attention along each trajectory and across trajectories within a group, preserving reliable observations, fixing corrupted positions, and completing missing motion, while predicting unconstrained 3D residuals rather than only depth corrections.
3. Provider-agnostic training from simulated lifts. The corrector is trained without ever running a tracker, depth model, or renderer: geometric effects of lifting failures (correspondence error, depth jitter, depth drift, wrong-surface observations, missing observations) are injected as randomized corruptions of clean 3D trajectories from PointOdyssey. The same trained weights improve all six evaluated tracker–geometry combinations without retraining.
4. State-of-the-art accuracy with a speed advantage. ReLift achieves the best mean dense 3D tracking accuracy on WorldTrack, improving on the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points, with faster configurations running 2.7–4.4× faster than 4RC.

Main Findings

• Lifting alone is competitive. CoWTracker trajectories lifted with initialized gaps and VGGT-Ω geometry reach 84.9 mean all-point APD3D, compared with 83.9 for TrackCraft3R and 81.1 for DELTAv2 using the same geometry.
• Best mean accuracy on WorldTrack. With CoWTracker and 4RC geometry, ReLift reaches 89.01 mean APD3D on all points and 86.95 on dynamic points, the top mean row in the main table; the authors report improvements of 2.1 APD on all points and 3.0 on dynamic points over the strongest baselines.
• Gains under matched geometry. Using 4RC's own depth and camera estimates, the CoWTracker-based pipeline improves over 4RC's trajectory predictions by 2.1 (all points) and 5.4 (dynamic points) in mean accuracy.
• One corrector transfers across providers. The identical trained corrector improves every one of the six combinations of {AllTracker, CoWTracker} with {Depth Anything 3, VGGT-Ω, 4RC}, and mean accuracy rises monotonically across those geometry providers for both trackers without retraining.
• Repair and completion help everywhere. Correction adds 1.6–3.6 points of mean APD3D over the initialized lift, with the largest gains on ground-truth-occluded points; with CoWTracker and VGGT-Ω, all-point APD3D rises from 84.9 to 87.5, overtaking both 4RC and V-DPM on all and dynamic points.
• Simulated lifts beat provider-generated training data. Models trained on AllTracker+DA3 or AllTracker+VGGT-Ω lifts improve their own training scenes (+14.1 and +8.3 APD3D respectively) but transfer inconsistently; the model trained on VGGT-Ω lifts reduces accuracy on every evaluated dataset, while the simulated-lift model improves on every dataset with both test geometry providers. A control trained on the same retained windows confirms the advantage is not due to window selection.
• Wrong-surface corruption matters most. Removing wrong-surface observations from training causes the largest accuracy drop (for example, gains over the lift fall from +2.0 to +0.3 on all points for AllTracker + VGGT-Ω), while removing correlated noise causes a smaller drop and removing synthetic gaps leaves the overall mean essentially unchanged, though gaps give small gains on occluded points.
• 2D accuracy favors the pretrained tracker. Native CoWTracker exceeds the image-plane accuracy (APD2D) of the projected trajectories of every evaluated 3D method on each dataset, for both all and dynamic points, supporting the decomposition the paper is built on.
• Speed. VGGT-Ω configurations are competitive while running 2.7–4.4× faster than 4RC, including all provider inference (7.3 s and 11.7 s versus 31.8 s in the reported table).
• Design choices. With C = 4 frames per token, correction is more accurate and about three times faster than with C = 1; C = 8 is faster still but slightly less accurate. Track groups of P = 1024 are used because smaller groups reduce accuracy and larger groups offer little improvement at higher attention cost.

Methodology in Plain English

The pipeline has three stages, and only the last one is trained.

First, frozen off-the-shelf models do the heavy perception. A dense 2D tracker follows every pixel of the reference frame through the video and reports a visibility score for each point in each frame. A geometry model supplies depth maps, intrinsics and camera poses, all expressed relative to the reference camera.

Second, each 2D track is lifted into 3D. The 2D position defines a viewing ray through the camera; the depth model gives the distance along that ray; combining the camera center, the ray direction and that distance yields a 3D point per frame. Points whose visibility score falls below a tracker-specific threshold, or whose range is not positive, are masked out, and the resulting gaps are filled by linear interpolation between the surrounding observations, with the last observation held after the track ends. Positions are normalized by the median reference depth so that scenes of different scale look alike to the network.

Third, a learned corrector edits these trajectories. Each frame of each track becomes a small token containing the displacement from the track's starting point, the masked version of that displacement, the availability mask, the distance in frames to the nearest real observation, the masked viewing ray, and the normalized camera center. Chunks of C consecutive frames are merged into a single tracklet token, and groups of P tracks are processed together. Eight transformer blocks alternate temporal attention (linking all time chunks of one track) with across-track attention (linking different tracks at the same time chunk), which costs less than attending over all tokens jointly. A zero-initialized head predicts a 3D residual per frame, added back to the initialized trajectory and rescaled. Because the head starts at zero, an untrained corrector simply returns the lift.

The distinctive training choice is that no tracker or depth model is run during training. Starting from clean 3D trajectories and camera centers in the PointOdyssey training split, the authors synthesize the failures that lifting produces: small rotations of the viewing direction for correspondence error, random and temporally correlated multiplicative range perturbations for depth jitter and drift, occasional large range outliers for wrong-surface observations, and geometric occluders plus temporal dropouts for missing observations. Some observations at the start or end of a synthetic occlusion are deliberately kept at the occluder's depth to mimic missed occlusions. Corruption parameters are randomized across examples so the network sees a broad range of error patterns, and the network is trained to output the clean trajectory at every frame, including frames where the input was masked. The loss is a weighted squared error on positions plus a term matching frame-to-frame displacements, with moving tracks weighted four times static ones and unavailable observations weighted twice available ones. At inference, reference pixels are split into random groups of P tracks and processed independently in batches, so cost grows linearly with the number of pixels.

Why This Matters

The paper's main impact on research is architectural: it argues that dense 3D tracking should not be learned end to end, because two of its three sub-problems already have strong pretrained solutions, and because annotated 4D data is scarce. Concentrating 4D supervision on trajectory refinement produces a module that is decoupled from any specific tracker or geometry model, so it inherits future improvements in either component for free. The simulated-lift training scheme also offers a template for training on geometric annotations without rendering images or running expensive providers.

Real-world applications the authors or the pipeline point to:

• Robot manipulation policy learning, where per-point 3D motion of manipulated objects is a useful signal.
• Dynamic scene reconstruction and rendering from casually captured monocular video.
• Sensor simulation, for generating realistic moving 3D content for autonomy and robotics testing.
• Motion-controlled video generation, where 3D trajectories are used to steer generation.

Industry relevance: the method's speed profile (7.3–11.7 s full-pipeline in the reported configurations versus 31.8 s for 4RC) and its reliance on interchangeable off-the-shelf components make it practical to slot into existing video or 3D-content pipelines, and the fact that one set of weights works across providers lowers the cost of adopting newer geometry or tracking backbones.

Future Directions

• Robustness to input geometry. The authors note that performance remains sensitive to input geometry and explicitly motivate future work on this.
• Unsimulated failure modes. The corruption model does not explicitly simulate camera-pose errors or persistent switches to another object's trajectory, leaving these failure modes for future handling.
• Extending the simulation. Synthetic gaps produced little overall mean change but helped on occluded points, suggesting room to refine occlusion simulation and the relative weighting of corruption families.
• Riding on component advances. Because the corrector consumes only geometric inputs and is trained without providers, the natural next step is to drop in stronger 2D trackers or geometry models as they appear and measure how far the same repair module carries over.

Target Audience

Researchers and engineers working on 3D vision, video understanding, or dynamic scene reconstruction who want a strong, efficient recipe for dense 3D point tracking without training a large end-to-end model. It is also useful for practitioners who already have a 2D tracker and a geometry model in production and want a lightweight learned post-process, and for students studying how to decompose a hard learning problem into pretrained components plus a narrowly supervised corrector.

Authors’ abstract

Dense 3D tracking from monocular video requires establishing point correspondences over time, estimating their depth, and inferring their trajectories during occlusion. Existing methods typically learn these tasks jointly, a demanding objective given the scarcity of 4D supervision. We show that strong pretrained 2D trackers and 3D geometry models provide an effective foundation: lifted trajectories, with occluded segments simply interpolated, already rival dedicated 3D trackers using the same geometry. However, lifted trajectories inherit correspondence and depth errors, and under occlusion, their depth comes from the occluder rather than the tracked point. We introduce ReLift, an efficient feed-forward framework that recasts dense 3D tracking as geometric trajectory repair and completion. At its core, a trajectory corrector alternates attention along and across trajectories to preserve reliable observations, repair corrupted positions, and complete missing motion. Using only geometric inputs allows us to train directly on 3D trajectories with simulated lifting errors, decoupling training from images and from any specific tracker or geometry model. Concentrating 4D supervision on trajectory refinement yields a reusable model that consistently improves lifted trajectories across trackers and geometry providers without retraining, so it benefits directly from future advances in either component. On WorldTrack, ReLift achieves state-of-the-art mean accuracy, improving on the strongest baselines by 2.1 APD on all points and 3.0 on dynamic points, with faster configurations remaining competitive with leading 3D trackers while running faster than 4RC. Visit our supplementry page for more examples and code.