Flow3r: Factored Flow Prediction for Scalable Visual Geometry Learning

AI summary

Overview

Research area: Computer vision, specifically feed-forward 3D/4D scene reconstruction — predicting camera poses and scene geometry from images without per-scene optimization.

Technical level: Advanced. The paper assumes fluency with pointmaps, multi-view transformers, camera tokens/patch tokens, dense correspondence (flow) estimation, and projective geometry.

Scope in one sentence: Flow3r introduces a "factored" flow-prediction module that conditions source-view geometry latents on target-view camera latents, letting visual geometry models learn from ~800K unlabeled videos and reach state-of-the-art results on eight benchmarks.

What This Paper Is About

Feed-forward visual geometry systems (VGGT, π³, DUSt3R and relatives) learn to output 3D structure and camera motion, but they depend on dense geometry and pose labels that come from multi-view reconstruction pipelines — expensive to obtain and especially scarce for dynamic real-world scenes such as in-the-wild or egocentric video. The paper asks how to use a cheaper signal instead: dense 2D correspondences ("flow") between frames, which can be extracted from any video. The central technical question is how flow should be predicted inside a geometry network so that supervising it actually improves camera and geometry learning rather than merely sharpening local features.

Key Contributions

1. A factored flow prediction module. Instead of matching local patch features, Flow3r predicts flow between views i and j using the geometry latents of view i together with only the camera latent of view j (Φ_flow(g_i, c_j)), decoded by a DPT head. The asymmetry mirrors projective geometry while avoiding explicit projection, and it extends naturally to dynamic scenes.
2. A controlled comparison of flow supervision designs. The authors train from-scratch, VGGT-like models under four regimes — 3d-sup (no flow), flow-projective (analytic projection from predicted poses and pointmaps), flow-tracking (VGGT-style pairwise patch-feature tracking head), and flow-factored — and show factored flow wins on both static and dynamic scenes.
3. A scaling study with unlabeled video. Keeping labeled data fixed at 1K OmniWorld sequences, they add 3K, 10K, and 20K unlabeled SpatialVID sequences and show monotonic improvement, with 20K unlabeled videos plus 1K labeled sequences outperforming 4K labeled sequences alone.
4. Integration into existing architectures and broad evaluation. Factored flow is added to π³ (primary model) and to a VGGT variant via two-stage training (frozen backbone to train the flow head, then end-to-end fine-tuning), yielding state-of-the-art results across eight benchmarks using ~800K unlabeled video sequences.

Main Findings

• Factored flow beats all alternatives in controlled tests. On ScanNet++ (1K labeled + 1K unlabeled), flow-factored reaches RRA@30 0.7700, RTA@30 0.7366, CD 0.026, MSE 0.078, versus 3d-sup at 0.7500 / 0.6929 / 0.030 / 0.088, flow-tracking at 0.7438 / 0.7021 / 0.030 / 0.089, and flow-projective at 0.6700 / 0.4572 / 0.033 / 0.088.
• The same holds on dynamic scenes. On OmniWorld (1K) plus SpatialVID (3K), flow-factored scores RRA@30 76.26, RTA@30 68.84, CD 0.103, MSE 0.598, compared with 66.01 / 62.37 / 0.105 / 0.637 for 3d-sup, 68.56 / 62.95 / 0.107 / 0.628 for flow-tracking, and 61.23 / 56.12 / 0.158 / 0.710 for flow-projective.
• Accurate flow does not imply useful supervision. flow-tracking learns to predict accurate flow (per the appendix) yet gives almost no improvement in pose accuracy or geometry quality, while flow-projective — despite being grounded in exact projective geometry — degrades performance. The authors attribute this to instability when errors in either decoded cameras or pointmaps propagate into the flow.
• Factored flow is deliberately not the best standalone flow estimator. The authors state it is suboptimal for standalone flow estimation because the target camera token creates an information bottleneck compared with patch features carrying richer visual cues, yet it provides a more effective supervisory signal for geometry.
• Performance scales with unlabeled data. With OmniWorld fixed at 1K, adding SpatialVID at 3K gives RRA@30 76.26, CD 0.103, MSE 0.598; at 10K, 78.45 / 0.077 / 0.560; at 20K, 81.12 / 0.075 / 0.532. Training on 4K labeled OmniWorld sequences alone yields 78.68 / 0.080 / 0.565, so 20K unlabeled videos plus 1K labeled sequences outperforms 4K labeled sequences.
• Large gains on in-the-wild dynamic video. On Kinetics700 with pseudo ground truth from MegaSAM, Flow3r records RPE trans 0.018, RPE rot 0.830, MSE 0.256, f-score@5 0.599 (π³: 0.023 / 1.006 / 0.267 / 0.585; VGGT: 0.038 / 1.392 / 0.347 / 0.479; CUT3R: 0.027 / 1.988 / 0.303 / 0.573; DUSt3R: 0.063 / 9.343 / 0.366 / 0.533). On EPIC-KITCHENS it records 0.037 / 2.729 / 0.199 / 0.622. On Sintel, 0.058 / 0.920 / 0.426 / 0.404. On Bonn, 0.094 / 6.262 / 0.052 / 0.954.
• Gains transfer to static scenes, though not uniformly. Flow3r leads on NRGBD (RTA@30 99.60, AUC@30 94.40, MSE 0.018, f-score@5 0.992) and 7-Scenes (91.66 / 75.76 / 0.102 / 0.807). On Co3Dv2 it has the best RTA@30 (97.62) and f-score@5 (0.876) but AUC@30 90.41 versus π³'s 90.53. On ScanNet its RTA@30 of 92.89 and AUC@30 of 71.00 are below VGGT's 93.71 and 71.37, while its f-score@5 of 0.943 is the highest reported.
• The gains come from unlabeled data, not from extra fine-tuning. With the π³ backbone, a labeled-data-only fine-tune gives 0.041 RPE trans / 3.005 RPE rot / 0.218 MSE / 0.620 f-score on EPIC-KITCHENS and 87.27 RTA / 72.66 AUC / 0.169 MSE / 0.735 f-score on 7-Scenes; adding flow loss on labeled data only gives 0.040 / 3.005 / 0.203 / 0.619 and 87.99 / 72.47 / 0.149 / 0.783; the full model reaches 0.037 / 2.729 / 0.199 / 0.622 and 91.66 / 75.76 / 0.102 / 0.807. The same pattern appears with the VGGT backbone.

Methodology in Plain English

A standard visual geometry network encodes images into patch tokens and runs a multi-view transformer that produces two kinds of latents per view: patch-level geometry features and a global per-image camera feature. The decoded outputs are normally supervised with ground-truth camera poses and pointmaps.

Flow3r keeps that supervised path for the labeled datasets it has, and adds a second path for data it does not have labels for. For a pair of frames, a small head takes the geometry latents of the source frame and the camera latent of the target frame, fuses them, and decodes a dense flow field through a DPT head. The predicted flow is compared against pseudo-ground-truth correspondences from an off-the-shelf teacher model (UFM) using a robust generalized Charbonnier loss, weighted by a covisibility mask. Because the source side supplies geometry and the target side supplies pose, the gradient from the flow loss pushes both components to be correct — and because the head decodes flow directly in latent space rather than projecting 3D points, it does not break on moving objects.

For the large-scale model, training is two-stage: first the pretrained π³ backbone is frozen and only the new flow head is trained on labeled data; then the whole model is unfrozen and fine-tuned end-to-end on a mix of labeled 3D/4D data and unlabeled video. Labeled data comes from eleven multi-view reconstruction datasets (CO3Dv2, Habitat, ARKitScenes, ScanNet, ScanNet++, MegaDepth, BlendedMVS, StaticThings3D, Omniworld, PointOdyssey, VKITTI), roughly 34K sequences providing poses and geometry. Unlabeled data comes from Kinetics-700, SpatialVID, and EPIC-Kitchens, roughly 800K video sequences. The π³-based model uses a DINOv2 encoder and a 36-layer multi-view transformer; the VGGT variant uses a 48-layer transformer with alternating frame-wise and global attention, with VGGT's global-frame predictions converted to relative, frame-independent supervision.

Why This Matters

Impact on research. The paper supplies evidence that flow can serve as a scalable substitute for dense geometry and pose labels, and — more importantly — that how flow is predicted determines whether the supervision helps. The negative results for flow-projective and flow-tracking are as informative as the positive result for flow-factored, since both are intuitive designs that do not improve geometry. It also suggests a path analogous to the self-supervised scaling that benefited large language models and vision transformers.

Real-world applications (each follows the settings the paper measures or names):

• Reconstructing 3D structure and camera motion from in-the-wild dynamic video, the setting where the paper reports its largest gains.
• Egocentric video understanding, explicitly named as a domain where labeled supervision is scarce and existing methods generalize poorly.
• Interaction videos involving people, objects, and animals — the paper's qualitative figure covers static scenes, dynamic scenes, and interaction videos, and reports baselines failing on motion such as "incorrect movement of the cat."
• Multi-view static reconstruction for settings such as ScanNet, NRGBD, 7-Scenes, and CO3Dv2, i.e., indoor scanning and object-centric capture.

Industry relevance. The supervisory signal is extracted from raw video, which is abundant and cheap, rather than from Structure-from-Motion-style annotation pipelines, which are expensive. Any product pipeline that must recover geometry and camera trajectories from consumer or robot video — AR/VR capture, robotics, 3D content creation, large-scale scene digitization — is a plausible beneficiary. The release of a project and code page (https://flow3r-project.github.io/) and the demonstration that the module can be bolted onto two existing backbones (π³ and VGGT) lower the barrier to adoption. The work was supported by an NVIDIA academic grant and computed on Bridges-2 at Pittsburgh Supercomputing Center, and by IARPA via DOI/IBC contract 140D0423C0074.

Future Directions

• Reducing dependence on the pseudo-label teacher. Flow3r relies on off-the-shelf models for pseudo-ground-truth flow, and the authors note there can be domains where such 2D prediction fails.
• Handling multiple independently moving components. The factored formulation handles a single combined camera-plus-scene motion field, but the authors expect Flow3r to struggle in complex scenes with several independently moving objects.
• Scaling to truly large data. Experiments currently use ~800K video sequences for flow supervision; scaling to roughly 10–100M videos is described as exciting but unexplored.
• Improving the flow module itself. The factored head is knowingly suboptimal for standalone flow estimation due to its information bottleneck, leaving room for a design that retains the geometry-aware supervision while predicting more accurate flow.

Target Audience

Researchers and graduate students in 3D vision, multi-view and dynamic reconstruction, structure-from-motion, and video correspondence, particularly those working on feed-forward geometry models or on reducing reliance on labeled 3D data. It is also relevant to practitioners who want to fine-tune an existing visual geometry backbone (π³ or VGGT) with a lightweight auxiliary head using their own unlabeled video. The paper is not an introductory read: it presumes comfort with pointmap representations, multi-view transformers, camera parameterization, and dense correspondence evaluation metrics such as RRA, RTA, AUC, Chamfer distance, MSE, and f-score.

Authors’ abstract

Current feed-forward 3D/4D reconstruction systems rely on dense geometry and pose supervision -- expensive to obtain at scale and particularly scarce for dynamic real-world scenes. We present Flow3r, a framework that augments visual geometry learning with dense 2D correspondences (`flow') as supervision, enabling scalable training from unlabeled monocular videos. Our key insight is that the flow prediction module should be factored: predicting flow between two images using geometry latents from one and pose latents from the other. This factorization directly guides the learning of both scene geometry and camera motion, and naturally extends to dynamic scenes. In controlled experiments, we show that factored flow prediction outperforms alternative designs and that performance scales consistently with unlabeled data. Integrating factored flow into existing visual geometry architectures and training with ${\sim}800$K unlabeled videos, Flow3r achieves state-of-the-art results across eight benchmarks spanning static and dynamic scenes, with its largest gains on in-the-wild dynamic videos where labeled data is most scarce.