ProMo3D: Probing Motion Cues in Frozen 3D Foundation Models

AI summary

Overview

Research area: Computer Vision — motion segmentation and dynamic-scene understanding with feed-forward 3D foundation models (DUSt3R, VGGT, π³/π³X, Depth Anything 3).

Technical level: Intermediate. Readers should be comfortable with ViT token representations, probing classifiers, and standard motion-segmentation benchmarks (DAVIS, SegTrackv2, FBMS-59), but the core idea — fitting a tiny probe on frozen features — is conceptually simple.

Scope (one sentence): The paper tests whether motion information is already present in the frozen token representations of pretrained 3D reconstruction models and shows that a small low-rank probe trained on 160 videos can decode it into usable motion masks that transfer zero-shot to three benchmarks.

What This Paper Is About

Feed-forward 3D foundation models reconstruct geometry from unposed multi-view images, but extracting motion from them usually requires either large-scale supervised training or brittle hand-crafted heuristics tied to a specific architecture. The authors ask a narrower question: do these frozen models already encode motion-relevant cues that a lightweight readout function can expose, without re-learning motion features or fine-tuning the backbone? They answer by probing three frozen backbones and turning the learned pairwise probe into a dense motion-mask predictor called Probing-based Motion Distillation (PMD).

Key Contributions

1. Systematic localization of motion cues. A layer-wise probing study across three feed-forward 3D foundation models — π³X, Depth Anything 3 (DA3), and DUSt3R — identifies where in the frozen network static-versus-dynamic information becomes decodable, without relying on architecture-specific heuristics such as attention patterns or key–query Gram matrices.

2. Characterization of a low-rank motion subspace. Using rank-restricted probing and spectral analysis on π³X, the authors show that the probe-induced motion representation is compact: a low-rank quadratic probe with factor rank k = 8 (rank at most 2k after symmetrization) suffices, and performance saturates at very low effective rank.

3. A data-efficient method, PMD. Probing-based Motion Distillation converts pairwise motion-state compatibility scores against a single fixed anchor token into dense dynamic score maps, thresholded into coarse masks and optionally refined with SAM2. It is trained on 160 YouTube-VOS 2018 videos (validated on 80) on one 16 GB NVIDIA RTX 5070 Ti GPU.

4. Zero-shot benchmark evidence. PMD transfers to DAVIS, SegTrackv2, and FBMS-59 with no target-benchmark fine-tuning, outperforming training-free heuristics and approaching much heavier supervised and self-supervised pipelines at a fraction of the trainable capacity.

Main Findings

• Motion is decodable from frozen tokens, broadly. Layer-wise probing shows motion decodability is widespread across the multi-view geometric modules of all three backbones, and the low-rank quadratic probe consistently beats the linear and diagonal probes. Motion cues also propagate to intra-frame blocks and prediction heads, but their quality varies by backbone.

• The cues live in a low-dimensional subspace. On π³X, a 3D PCA of the k = 8 probe representation shows clear separation between static and dynamic tokens; the singular spectrum decays steeply in leading components, especially in earlier layers; and performance saturates at very low effective rank. A low-rank probe (k = 8) matches or exceeds a full-rank probe even as training data is progressively reduced.

• The probe genuinely uses token–label correspondence. Shuffling labels within each frame group (same frozen features, same label counts) drops pairwise accuracy from 95.72% to 84.48% and balanced accuracy from 91.67% to 50.34%, returning to the always-same-state baseline (84.47% accuracy, 50% balanced accuracy) averaged over 18 inter-frame blocks.

• Strong zero-shot motion segmentation. On DAVIS17, π³X-PMD reaches 73.7 J-M / 91.0 J-R coarse and 81.9 / 92.3 with SAM2; π³-PMD reaches 72.7 / 89.9 and 80.2 / 91.1. On SegTrackv2, π³X-PMD reaches 72.9 / 94.0 coarse and 86.4 / 99.3 with SAM2; on FBMS-59, 73.5 / 88.8 and 83.2 / 89.3.

• Comparison points. Training-free baselines are clearly surpassed: VGGT4D reports 56.5 / 65.6 and Easi3R+DUSt3R+SAM2 reports 60.1 / 65.3 on DAVIS17. GeoMotion, which uses five datasets and a π³ + RAFT + DINOv2 pipeline, reports 75.4 / 87.2 coarse and 81.1 / 92.6 with SAM2 on DAVIS17. Flow-SAM reports 71.6 / 84.2 / 88.7 J-M across DAVIS17, SegTrackv2, FBMS-59, and SegAnyMo 76.3 / 78.3 on SegTrackv2 and FBMS-59.

• Backbone choice matters. Multi-view pretrained variants outperform DUSt3R-PMD (62.2 / 70.0, 58.6 / 77.3, 64.0 / 74.5). The π³X–DA3 gap on DAVIS17 is 4.1 J-M for coarse masks and 4.2 points after SAM2 refinement.

• Low refinement headroom means already-good coarse masks. SAM2 adds only 1.2 and 1.3 J-R points for π³- and π³X-PMD, versus 5.4 for GeoMotion and 9.9–14.8 for Easi3R variants — high coarse J-R indicates most frames already exceed 0.5 Jaccard.

• Very small compute footprint. The low-rank probe has 0.295M trainable parameters and adds 1.62G FLOPs per frame; coarse PMD adds 4.2 ms/frame (0.87%) to frozen-backbone inference. GeoMotion uses 80.3M trainable parameters, Flow-SAM 9.83M, SegAnyMo 3.44M.

• Category generalization. Qualitative examples from DAVIS and SegTrackv2 show cleaner score maps and more coherent coarse masks than Easi3R and VGGT4D on semantic classes absent from the 160-video training subset.

• Known failure mode: semantic leakage. Static instances can receive elevated dynamic scores when another instance of the same category is moving (illustrated with background flamingos), attributed in part to object-centric YouTube-VOS supervision and frozen semantic features.

Methodology in Plain English

The recipe has three steps.

1. Freeze the backbone and label the tokens. Feed a small group of frames to a pretrained 3D model and take the token features from chosen transformer blocks. Project the ground-truth dynamic mask onto the token grid so each token is labeled dynamic or static.

2. Train a tiny probe on token pairs. Rather than classifying single tokens, the probe predicts whether two tokens share the same motion state. The authors compare three probes: a linear one, a diagonal quadratic one, and their main choice, a low-rank quadratic form whose weight matrix is built from two k×d factors (k = 8), symmetrized so the prediction is order-invariant. Training uses binary cross-entropy on sampled token pairs, with the backbone always frozen — AdamW at 1×10⁻³, 16 epochs, 200K token pairs per epoch.

3. Turn pairwise scores into a mask with a fixed anchor. At inference, one token at a fixed spatial index (a = 0) is treated as a static reference. Querying the probe between that anchor and every other token gives a static score per token; one minus that is the dynamic score. Scores are averaged across the selected blocks, reshaped to the token grid, bilinearly upsampled to image resolution, and thresholded with Otsu plus an offset of 0.2. Optionally, the coarse mask is eroded, filtered by mean dynamic score, and passed to SAM2 for refinement. Because all anchor–target scores can be computed in parallel, the overhead on top of frozen backbone inference is small.

Inference uses a 3-frame context with stride 2. For padded backbones the anchor sits in the padded region and is static by construction; on YouTube-VOS 2018 the upper-left 14×14 patch corresponding to the anchor is static in 97.73% of frames in the authors' 160-video training subset and 96.58% in their 80-video validation subset, which the authors report as a post-hoc sanity check rather than a selection criterion.

Why This Matters

Impact on research. The paper reframes dynamic-scene understanding from "train a big motion decoder" to "decode what the geometry-pretrained model already knows." It provides a reusable diagnostic — layer-wise probing with rank-restricted probes — for auditing 3D foundation models, and it quantifies the efficiency gap: 0.295M trainable parameters and 160 videos versus 80.3M parameters and five datasets for a comparable supervised decoder. It also gives a concrete, spectral account of why such a small readout works on π³X.

Real-world applications:

• Robot perception and manipulation: cheap moving-object segmentation on top of an existing 3D reconstruction stack, without a separate motion-specialized network.
• Autonomous driving and mobile mapping: separating independently moving agents from static structure while the platform itself moves, using geometry models already deployed for reconstruction.
• Video editing and post-production: rotoscoping-style foreground extraction and object isolation from ordinary monocular footage, with optional SAM2 mask refinement for sharper boundaries.
• Surveillance and scene monitoring: flagging moving regions in cluttered scenes on consumer-grade hardware, since the whole pipeline runs on one 16 GB GPU.

Industry relevance. The method requires no backbone fine-tuning and no large labeled corpus of motion masks — 160 videos and a sub-million-parameter probe. That makes it attractive for teams that already run a 3D foundation model in production and want an incremental motion capability, and for edge or cost-constrained deployments where multi-GPU supervised training is impractical.

Future Directions

1. Beyond binary masks. PMD makes binary static–dynamic decisions and cannot distinguish motion types such as in-place articulation, local deformation, or object displacement. Probing additional physical quantities — the authors suggest displacement and velocity — from frozen 3D backbones is the natural extension.

2. Fixing semantic leakage. Static instances of a moving object's category receive inflated dynamic scores. Disentangling semantic from motion cues, or changing the object-centric supervision, is an open problem, particularly for categories unseen in training.

3. Overcoming token granularity. Spatial precision is capped by ViT token resolution. Better upsampling, higher-resolution token grids, or tighter integration with promptable segmentation models could close this.

4. Anchor and camera-motion assumptions. The fixed a = 0 anchor is validated empirically rather than guaranteed, and the formulation does not estimate absolute scene motion: a fully static scene under camera motion and a uniformly moving scene both admit a no-independent-foreground interpretation. The authors note that no dedicated evaluation on all-static camera-motion datasets is reported, and their optional anchor-pinning safeguard is disabled in all reported results.

Target Audience

Researchers and engineers working on dynamic 3D reconstruction, motion segmentation, or representation probing of foundation models will get the most from this paper, particularly those who already use DUSt3R-, VGGT-, π³-, or Depth Anything 3-style backbones and want to know what those frozen features already contain. It is also useful for practitioners seeking a low-cost motion-mask baseline, and for students learning how probing studies are designed — the layer-wise, rank-restricted, and label-shuffling controls form a clean template. Readers without background in ViT features or segmentation benchmarks will need to consult the cited prior work first.

Authors’ abstract

Feed-forward 3D foundation models have recently reshaped multi-view reconstruction, yet dynamic 4D understanding still often relies on large-scale motion supervision or hand-crafted, architecture-specific heuristics. We ask whether frozen 3D foundation model tokens already encode motion-relevant information that can be exposed by lightweight readout functions, rather than re-learning motion-specific features with high-capacity decoders. Through probing on three representative backbones, we find that static--dynamic separation is decodable from frozen token representations. Our rank-restricted and spectral analyses on $π^3$X reveal that the recovered motion cues are concentrated in a low-dimensional subspace. Building on this, we propose Probing-based Motion Distillation (PMD), which converts pairwise motion-state compatibility scores against a fixed anchor token into dense motion masks. Trained on only 160 YouTube-VOS videos using a single 16 GB GPU, PMD transfers zero-shot to DAVIS, SegTrackv2, and FBMS-59 without target-benchmark fine-tuning. With multi-view pretrained geometric backbones, PMD outperforms training-free heuristics and approaches substantially heavier supervised and self-supervised methods, suggesting that motion decoding from frozen 3D foundation models can be lightweight and data-efficient.