TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

AI summary

Overview

Research area: computer vision, specifically 3D point tracking and dynamic scene representation for video. Technical level: Advanced (assumes familiarity with point tracking benchmarks such as TAPVid-3D, transformer attention, pointmaps, and voxel representations). Scope: The paper introduces TrackEverything, a sliding-window 3D point tracker that de-duplicates co-located tracks in world-coordinate voxel space to track all visible points across videos exceeding 1000 frames within 40 GB of GPU memory.

What This Paper Is About

Point trackers trade off breadth against duration: sparse trackers follow a small set of user-specified query points over long videos, while dense trackers follow all points but only across short clips of 48–64 frames, because frame-local pixel representations re-encode the same surfaces at every timestep and blow up in memory. The paper's goal is to break that trade-off by representing a video as persistent 3D scene tracks in world coordinates, so that model complexity scales with unique physical scene geometry rather than with video duration.

Key Contributions

1. TrackEverything, a 3D point tracker for all visible points over long horizons. The authors state it is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory.
2. Voxelization-based de-duplication at window boundaries. Points from different frames that converge on the same physical surface occupy the same 3D voxel and are merged by mean-pooling into a single canonical track, so representation size scales with unique scene geometry rather than frame count.
3. An "endpoints before trajectories" decomposition. An endpoint refiner predicts each point's destination at the window boundary and classifies it as static or dynamic; a lightweight trajectory refiner then decodes dense within-window trajectories only for dynamic points, concentrating dense decoding on the moving subset of the scene.
4. 3D WAFT. An extension of warp-aligned feature transforms to 3D point clouds that replaces memory-intensive 4D correlation volumes with feature sampling at projected source and target locations, described as template matching at a fraction of the compute cost.

Main Findings

• Dense short-clip accuracy on TAPVid-3D. TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips. In Table 1 it beats VDPM by over 20% APD-P on average, and reports 45.7 APD-P / 88.7 APD-M on ADT, 28.2 / 40.3 on DriveTrack, and 30.1 / 88.7 on PStudio over the first 48 frames (34.7 / 72.6 average).
• Only method scaling to full-length all-point tracking. VDPM and D4RT are marked N/A on full-length sequences. TrackEverything reports 40.0 / 85.1 (ADT), 26.5 / 42.4 (DriveTrack), 27.2 / 88.5 (PStudio), an average of 31.2 APD-P / 72.0 APD-M, while remaining competitive with state-of-the-art sparse trackers despite tracking far more points.
• Competitive with first-frame dense tracking under a stricter setting. Against DeltaV2 with the same VGGT-Ω backend, TrackEverything is ahead on average and on ADT, and slightly behind on DriveTrack and PStudio (DeltaV2 averages 34.4 APD-P / 73.0 APD-M on 48 frames and 29.9 / 70.7 on full length), while densely tracking all points across all frames rather than only those originating in the first frame.
• Memory and latency scaling on PointOdyssey. At 384 query points, Any4D exceeds memory at 96 frames, SpatialTracker-v2 and DeltaV2-dense at approximately 200 frames, DeltaV2-sparse at approximately 500 frames, and CoTracker3 at approximately 600 frames. TrackEverything remains under 30 GB even at 900 frames. DeltaV2-dense exceeds 100 s per clip at 200 frames.
• Query-point scaling. With video length fixed at 120 frames, CoTracker3 and DeltaV2-sparse start with lower latency but are overtaken by TrackEverything at approximately 750 points (DeltaV2-sparse) and 2,000 points (CoTracker3); SpatialTracker-v2 memory exceeds limits at 1,000 points. The method accepts up to 10,000 additional query points.
• Geometry source matters and is swappable. Without retraining, replacing Pi3 with VGGT-Ω improves APD-P by +6.6 on PStudio (to 30.1) and +6.4 on ADT (to 45.7). Sensor geometry reaches 71.7 on PStudio and 46.5 on ADT, surpassing D4RT's reported 49.6 and 40.8 despite roughly 20× fewer parameters.
• Static/dynamic decomposition is an efficiency win, not an accuracy one. The classifier reaches 93.1 and 92.5 F1 on Dynamic Replica and PointOdyssey. Forcing all points through the dynamic trajectory decoder leaves APD-P identical at 31.2 but increases latency by 4.8× and peak GPU memory by 2.5×.
• Ablations. Removing 3D WAFT drops APD-P from 31.2 to 29.3 (−1.9); removing iterative trajectory refinement drops it to 26.4 (−4.8).
• Voxelization is necessary. Without voxelization the model runs out of memory. Accuracy stays near 31 APD-P from voxel size 0.005 up to 0.02 and drops thereafter (28.5 at 0.05, 23.5 at 0.2). Finer voxels cost more: 0.23 s/frame and 24.0 GB at 0.005 versus 0.03 s and 10.7 GB at 0.2. The default v = 0.02 gives 30.9 APD-P at 0.11 s/frame and 15.0 GB.
• Window length. L = 8 gives 31.3 APD-P, L = 16 gives 31.2, and L = 24 drops to 29.8; training at L = 16 is 26% faster per step than L = 8, so L = 16 is used.

Methodology in Plain English

The system processes a video in non-overlapping windows of 16 frames. Each frame is encoded by a frozen DINOv3-small backbone plus a trainable ViT adapter head, producing D = 384-dimensional features at quarter resolution. Using per-frame 3D pointmaps (from sensors or from a feedforward reconstruction model such as VGGT-Ω), those 2D features are unprojected into a world-coordinate 3D feature cloud. Tokens are quantized into per-frame voxels and merged, but that first voxelization is deliberately per-frame so that distinct surfaces occupying the same 3D region at different times are not wrongly merged.

Tracking then runs in two stages. The endpoint refiner is a recurrent transformer that, at each pass, samples features from the target frame at the point's current estimated position and from the source frame at its birth location (3D WAFT), concatenates them with the point's own feature, and adds sinusoidal spatial and temporal embeddings. Self-attention over the feature cloud lets the transformer update each point's target-timestep 3D position by a predicted residual, and also emit a visibility logit and a static/dynamic logit. The trajectory refiner then handles only the points classified as dynamic: their within-window paths are initialized by constant-velocity interpolation and refined using temporal self-attention along each track plus cross-attention from a [cls] track-summary token to the scene cloud. Tracks do not attend to each other, which lets the model decode a random subset of tracks during training yet all tracks at test time. Refinement alternates between the two refiners for K iterations per window.

At each window boundary, the cloud is voxelized again, this time across frames, because all points have been tracked to the same target timestep and two physical entities cannot occupy the same voxel at the same time. Co-located points are mean-pooled into one canonical track, which is carried into the next window alongside newly disoccluded content. Supervision uses L2 loss on 3D positions and cross-entropy on visibility and static/dynamic labels, computed only over the labelled points the training data provide.

Training details: 41M trainable parameters (61M total including the frozen 20M DINOv3-small backbone), transformer attention layers initialized from the last layers of DINOv2-small, 8 L40S-46GB GPUs, per-GPU batch size 1, learning rate 1×10⁻⁴, 100k iterations on 32-frame videos followed by 300k iterations on a mix of 32- and 64-frame videos, on Kubric, PointOdyssey, and Dynamic Replica.

Why This Matters

The paper reframes video tracking as a problem of scaling with unique scene geometry rather than with frame count, and shows that a 61M-parameter model with public training data can track every visible point across 1000+ frames on a single GPU, where prior dense 3D trackers exhaust memory beyond roughly 96 frames. For research, it supplies a persistent, de-duplicating dynamic 3D scene representation that the authors present as a substrate for downstream foundation models, and it demonstrates that tracking accuracy can inherit improvements in geometry backbones or sensor hardware without retraining—something monolithic trackers like D4RT cannot do.

Real-world applications implied by the paper's framing:

• Vision-language-action policies and robotic manipulation, where explicit 3D representations are described as beneficial but currently confined to static or quasi-static scenes.
• Vision-language models that reason about dynamic physical environments, extending explicit 3D representations beyond static settings.
• Video generation, where the paper notes explicit 3D representations have shown benefits.
• Long-horizon egocentric and driving video analysis, since evaluations include egocentric indoor videos (ADT), egocentric driving videos (DriveTrack), and multi-camera lab captures (PStudio).

Industry relevance: the efficiency claims are stated in terms of a single L40S-46G GPU, under 30 GB in the scaling analysis and within 40 GB in the abstract's framing of 1000+ frame tracking. That places all-point long-video 3D tracking within reach of commodity single-GPU inference, which matters for robotics, autonomy, and video understanding pipelines where dense scene motion is the input rather than a sparse query set.

Future Directions

• Extending downstream foundation models to dynamic scenes. The authors explicitly position the representation as a path toward extending vision-language models, manipulation policies, and video generation beyond static or quasi-static scenes.
• Decoupling from the geometry backbone. Because TrackEverything ingests external pointmaps without retraining, future geometry improvements and better sensors are a direct, unretrained path to higher accuracy (sensor geometry reached 71.7 APD-P on PStudio versus 30.1 with VGGT-Ω).
• Dense ground-truth evaluation. TAPVid-3D evaluates on a sparse set of query points because dense 3D ground truth is infeasible for real-world dynamic scenes; the paper reports dense evaluation only on held-out simulation splits (PointOdyssey, Dynamic Replica) in the supplementary.
• Trade-offs left explicitly open. Voxel size trades accuracy for latency and memory (accuracy drops beyond 0.02), and window length beyond 24 frames degrades accuracy, leaving room to study where the representation's limits actually lie. The paper does not report the number of attention layers, the number of refinement iterations K, or scaling behavior beyond 900 frames and 10,000 query points.

Target Audience

Researchers and engineers working on point tracking, dynamic 3D scene reconstruction, and 4D perception; practitioners building robotics, autonomous driving, or video-understanding systems that need dense motion over long videos; and readers interested in how explicit 3D representations can make video foundation models scale with scene content instead of frame count. Familiarity with TAPVid-3D, pointmaps, and transformer attention is helpful, since the architecture and metrics are presented in those terms.

Authors’ abstract

Existing point tracking models face a fundamental tradeoff: they can either track a sparse set of query points over long horizons, or track all points across only short clips. We introduce TrackEverything, a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates. Grounded in the insight that videos are 2D projections of an underlying 3D world, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead. Our approach introduces three key innovations. First, we employ a voxelization-based de-duplication mechanism at sliding-window boundaries to merge co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, we decompose tracking into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, we propose 3D WAFT, replacing memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud. To the best of our knowledge, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.