Foundation Model Driven Optical Flow

AI summary

Overview

Research area: Computer Vision — dense optical flow estimation, specifically reusing pretrained vision foundation models (DINOv2, Depth Anything V2) inside a supervised matching pipeline.

Technical level: Intermediate. Readers should be comfortable with optical flow metrics (EPE, F1-all), transformer attention, and the idea of frozen pretrained backbones, but the paper's own framing is largely about pipeline configuration and benchmarking rather than new mathematical machinery.

Scope: A workshop-challenge paper (RoCo-Spring 2026, "The Robust Correspondence Challenge") that extends the authors' prior TTSFlow framework with DPT-based decoding of DINOv2 features and one coarse-to-fine refinement stage, then reports standard benchmarks and a corruption-robustness leaderboard score.

What This Paper Is About

Dense optical flow assigns a motion vector to every pixel between two consecutive frames. Modern methods either train specialized networks from scratch or refine iteratively at test time; this paper asks whether frozen pretrained visual and depth representations can be plugged into a supervised matching pipeline instead. The goal is to measure how far a pipeline built on frozen DINOv2 and Depth Anything V2 features gets on cross-dataset benchmarks and on the corruption-focused RoCo-Spring challenge, and to isolate the effect of two specific design changes: DPT decoding of intermediate DINOv2 blocks, and one higher-resolution refinement stage.

Key Contributions

1. DPT decoding of intermediate DINOv2 features. The paper takes transformer-block outputs 3, 6, 9, and 12 from a frozen DINOv2-S encoder, strips class tokens, and decodes them with a DPT head into a 128-channel map that is resized to 1/8 and 1/4 matching resolutions, rather than using the encoder's native token grid directly.

2. One refinement stage added to the TTSFlow fusion design. The pipeline fuses frozen DINOv2 features with a 128-channel intermediate feature from a frozen Depth Anything V2-B decoder (depth_head.scratch.refinenet1, before output_conv1), matches at 1/8 resolution with GMFlow-style global correlation, then performs a single 1/4-resolution residual refinement — hence "one refinement stage" rather than refinement-free inference.

3. A controlled DPT ablation at fixed refinement count. Table 5 compares the full pipeline with and without DPT after 360k Things-stage updates, both with one refinement stage, so the DPT gain is measured without changing the refinement budget.

4. A RoCo-Spring 2026 challenge entry (RoCo-29 · VSAI). The Spring-fine-tuned model obtains a robustness-balanced score (RbS) of 1.1038, Spring EPE 1.411, and RobustSpring proxy EPE 4.739, with a per-corruption and per-subset error breakdown in the appendix.

Main Findings

• Sintel Final improvement over the listed FlowSeek (T) configuration. After Chairs + Things training only, the model reaches 2.37 EPE on Sintel training split, Final pass, versus 2.52 for the FlowSeek (T) configuration reported here and 2.48 for GMFlow with one refinement. The authors explicitly frame this as a Final-pass advantage "within the listed configurations."

• The ordering flips on Sintel Clean and Things. Sintel Clean EPE is 1.26, above GMFlow's 1.08 and FlowSeek's 1.12; on FlyingThings3D validation, EPE is 3.28, below RAFT (4.25) and GMFlow without refinement (3.48), but above GMFlow with one refinement (2.80).

• Stronger target-adapted results on Sintel and KITTI. After Chairs + Things + TSKH training, the model reaches 0.816 Clean / 1.13 Final on Sintel, improving on TTSFlow (0.847 / 1.140). On KITTI it reaches 1.248 EPE and 4.01% F1-all, improving on GMFlow with one refinement (1.36 / 5.17%) and TTSFlow (1.99 / 7.40%); FlowSeek has slightly higher EPE (1.26) but lower F1-all (3.90%). SEA-RAFT remains best on both, under a different training configuration (eight L40 GPUs, batch 32, 432×960 crops, plus TartanAir pretraining) versus two RTX 6000 GPUs, batch 4, 320×896 crops for this work.

• DPT decoding helps, especially at large displacements. With refinement count fixed, adding DPT lowers Sintel Clean EPE from 1.33 to 1.26 and Sintel Final EPE from 2.73 to 2.37; the largest reduction is in the Final s40+ bin, from 16.69 to 14.05.

• Cross-modal fusion is prior evidence, not a new result. The fusion ablation is reproduced from TTSFlow (800k Things updates after 200k Chairs updates, no DPT or refinement): fusion reduces EPE from 3.52 to 3.02 on Things, 1.575 to 1.46 on Sintel Clean, and 3.12 to 2.81 on Sintel Final, with the s40+ Sintel Final bin improving from 19.37 to 16.99. Sintel Clean s10–40 slightly worsens (2.01 to 2.05).

• The model is mid-pack on the robustness leaderboard. RbS 1.1038 is lower (better) than seven of the twelve listed reference baselines in Table 3, but worse than SEA-RAFT (0.4584), MS-RAFT+ (0.6774), FlowFormer (0.7368), GMFlow (0.8015), and GMA (0.8704). GMFlow's reference has lower clean EPE (0.945 vs 1.411) despite the 400k Spring fine-tuning updates, though the GMFlow reference uses a Sintel checkpoint — the authors stress these are different training histories.

• Aggregate stability hides corruption-specific fragility. RobustSpring disagreement is low for brightness (0.308) and saturation (0.415) but high for rain (13.034), spatter (10.097), snow (8.943), and frost (8.579). Clean EPE is 7.193 in high-detail regions, 6.500 at unmatched pixels, and 5.887 for motions above 40 pixels.

• Inference is the slowest measured. At 362 ms per Sintel-resolution pair on a single Quadro RTX 6000, the model is slower than GMFlow with one refinement (295 ms), RAFT with 32 updates (229 ms), GMFlow without refinement (194 ms), FlowSeek (T) (144 ms), and SEA-RAFT (S) (63 ms). One refinement stage does not imply low latency.

Methodology in Plain English

The system takes two consecutive RGB frames and runs both through two frozen feature extractors. The visual branch resizes each frame to 448×448 and feeds it to DINOv2-S, which uses 14×14 patches and produces a 32×32 grid of 384-channel tokens. Rather than using the final output, the authors pull out four intermediate transformer blocks (3, 6, 9, 12), drop the class tokens, and decode them with a DPT head into a 128-channel spatial map — this is the new piece relative to their earlier TTSFlow work. In parallel, the depth branch resizes frames to 518×518 and runs Depth Anything V2-B, taking a 128-channel intermediate decoder feature; the actual scalar depth prediction is never used for matching, only the internal feature.

Both branches are fused per frame and per scale by concatenation, a 3×3 convolution, two residual blocks at 256 channels, and a 1×1 projection back to 128 channels. A six-block, 128-dimensional, single-head transformer then does global all-pairs correlation between the two frames at 1/8 resolution, and the flow is read out as an expectation over the softmax correspondence matrix rather than an argmax, which gives continuous coordinates without restricting the search range. A learned propagation step smooths these estimates.

One refinement stage follows: the 1/8 flow is upsampled by two with displacement scaling, used to warp the second frame's 1/4 features, and the shared transformer predicts a residual within a 9×9 local neighborhood, which is added to the upsampled flow. Final learned convex upsampling by a factor of four produces the full-resolution flow field. Learning comes from a masked ℓ1 loss on four full-resolution predictions with weights (0.729, 0.81, 0.9, 1) and γ = 0.9; pixels with ground-truth magnitude of at least 400 are masked out.

Training runs on two NVIDIA RTX 6000 GPUs with AdamW: FlyingChairs for 200k steps at 384×512 crops, FlyingThings3D for 360k iterations at 384×768 crops, and for target-adapted evaluation a TSKH mixture (Things, Sintel, KITTI, HD1K) for 320k iterations at 320×896 crops, plus a 200k-iteration KITTI fine-tune. The challenge entry is additionally fine-tuned on Spring for 400k iterations at 544×960 crops with batch size 4, and the predicted flow fields are submitted as .hdf5 files to the benchmark server. The matching transformer, correlation-to-flow formulation, propagation, refinement, and convex upsampling follow GMFlow; the learned parts are the downstream transformer, fusion and projection layers, propagation projections, and the upsampling predictor.

Why This Matters

Impact on research. The paper treats foundation-model features as a drop-in replacement for task-specific CNN encoders and then measures what that substitution actually buys, including where it does not win. Its most useful contribution may be methodological honesty: it reproduces its predecessor's fusion ablation rather than re-claiming it, holds refinement count fixed for the DPT comparison, and reports per-corruption and per-subset breakdowns instead of a single leaderboard number. It also documents a caveat that matters for reproducibility — the visual DPT path carries no flow gradient in the released forward path — and notes that removing a feature branch changes the fusion architecture, so branch-removal numbers (depth-only 122.87/120.12 EPE, DINOv2-only 36.15/36.42, joint 1.46/2.81 on Sintel Clean/Final) are not capacity-matched comparisons.

Real-world applications:

• Robotics and autonomous navigation, where motion estimation from commodity cameras feeds odometry and obstacle reasoning.
• Video understanding and post-production, including frame interpolation, stabilization, and motion-aware editing.
• 3D scene analysis and reconstruction, where per-pixel correspondence supports structure-from-motion and dense depth.
• Action recognition and saliency detection, both of which the paper cites as downstream consumers of dense flow.

Industry relevance. Reusing frozen backbones means a team can build a flow system without retraining a large encoder from scratch, and the single-forward-pass design (no repeated test-time updates) is architecturally simpler than RAFT-style iterative refinement. The 362 ms latency and the severe sensitivity to rain, snow, spatter, and frost are the practical counterweights: for outdoor or safety-critical deployment, the corruption breakdown in Table 7 is arguably more decision-relevant than the average benchmark EPE.

Future Directions

• Corruption-targeted robustness. Rain (13.034), spatter (10.097), snow (8.943), and frost (8.579) drive the disagreement far more than brightness (0.308) or saturation (0.415). The paper does not identify which module — DINOv2 features, depth features, fusion, or the matching operator — is responsible, leaving that attribution open.

• More refinement without losing efficiency. The authors propose lightweight multiple refinement, adaptive correction, or multi-frame temporal reasoning as routes to better handling of extreme motion, severe appearance changes, occlusions, and fine motion boundaries, which they name as the main limitation of features not optimized for optical flow.

• Efficiency attribution. The timing record does not separate the two foundation branches, DPT decoding, fusion, matching, propagation, and upsampling, and does not specify precision, batch size, warm-up, synchronization, or repetition count. Module-level profiling under a fixed protocol is an obvious next step for a 362 ms system.

• Training-matched comparisons. The paper repeatedly flags that its comparisons with SEA-RAFT, FlowSeek, WAFT, CroCo v2, and D4RT are configuration comparisons rather than architecture-isolated or training-matched ones, and that longer training (560k vs 360k Things-stage checkpoints) improves Things validation (3.28 → 2.95) and Sintel Clean (1.26 → 1.16) while slightly worsening Sintel Final (2.37 → 2.42).

Target Audience

Researchers and graduate students working on optical flow, dense correspondence, or foundation-model transfer who want a concrete, honestly-scoped data point on how frozen DINOv2 plus Depth Anything V2 features perform inside a GMFlow-style matching pipeline. It is also useful for practitioners entering robustness or corruption benchmarks such as RoCo-Spring, and for engineers evaluating whether a frozen-backbone, single-refinement design is worth its 362 ms inference cost against lighter alternatives.

Authors’ abstract

We investigate how frozen pretrained visual and depth representations support dense optical flow when coupled with a supervised matching network. Building on TTSFlow, this paper studies Dense Prediction Transformer (DPT) feature decoding and one coarse-to-fine refinement stage. Our method fuses features from frozen DINOv2 and Depth Anything V2 models and estimates optical flow through a trainable transformer-based matching pipeline. After optical-flow training on FlyingChairs and FlyingThings3D, it achieves 2.37 endpoint error (EPE) on the Sintel training split, Final pass, compared with 2.52 for the FlowSeek (T) configuration reported here. Our RoCo-Spring 2026 entry, RoCo-29.VSAI, obtains a robustness-balanced score of 1.1038, Spring EPE of 1.411, and RobustSpring proxy EPE of 4.739, with a lower balanced score than seven of the twelve listed reference baselines. These results characterize the complete pipeline, which reuses frozen foundation backbones and performs one refinement stage. Code is available at: \url{https://github.com/Visual-and-Spatial-AI-Lab/FFlow}