Optical Flow and Dense Correspondence

A stationary wall moves across the image when the camera pans; a rotating ball can change appearance while its center barely moves; a hidden surface disappears and creates no valid correspondence.

Optical flow estimates apparent image displacement, conditioned on visibility and imaging; it does not directly provide object motion in world coordinates.

Modern flow systems combine appearance matching with iterative refinement. Every stage below has a published way to be measured. The last one — marking uncertainty and occlusion — is the stage this lesson returns to twice, because it is the one an average error score erases.

Classical and learned methods still rely on some notion of matchability. The smoothness entry in this table is not a stylistic preference. It is arithmetic.

Two frames give one measurement at a point. The flow at that point has two components. Horn and Schunck said so in 1980: flow “cannot be computed locally, since only one independent measurement is available from the image sequence at a point, while the flow velocity has two components”. Their abstract names the second constraint they added to close the gap: “A method for finding the optical flow pattern is presented which assumes that the apparent velocity of the brightness pattern varies smoothly almost everywhere in the image.” The same memo flags its own exception, giving examples “where the assumption of smoothness is violated at singular points or along lines in the image”. The assumption arrived with its failure case attached.

Middlebury's own paper rebuilds that step in 2011. Brightness constancy linearises to the optical flow constraint u∂I/∂x + v∂I/∂y + ∂I/∂t = 0, which is “just one constraint on the two unknowns at each pixel” — “the origin of the Aperture Problem”. Aggregating that residual in L2 is, in their equation 5, “the Horn and Schunck algorithm (Horn and Schunck 1981)”. They also name what the aggregation quietly assumes: Gaussian, independent, identically distributed errors, a picture that “is rarely true in practice, particularly near occlusion boundaries where pixels at time t may not be visible at time t + 1”.

The visibility row has a measured size as well. Under a sub-heading called “Unmatched Pixels”, the Sintel paper counts them: “With large motions, these are common and we find that nearly 8.5% of the pixels in Sintel are unmatched.” The penalty is not a matter of degree. “Algorithms with EPEs of less than 0.5 pixel on Middlebury have EPE of approximately 10 pixels on MPI-Sintel; a 20-fold increase. In unmatched regions we observe EPEs of more than 40 pixels.” In those regions “errors here are an order of magnitude higher than in matched regions”. And because the motion-boundary statistics deliberately exclude unmatched pixels, the authors advise that “to get a full picture of the difficulty caused by occlusions one should separately consider errors both near motion boundaries and inside unmatched regions”. The release carrying those numbers is 1628 ground-truth flow frames at 1024 × 436, split into 1064 training and 564 withheld test frames.

These cases reveal different limits. They should not be collapsed into one average error, and the spread between them is now measured rather than argued.

RobustSpring, an ICLR 2026 benchmark, does the measuring: “RobustSpring applies 20 different image corruptions, including noise, blur, color changes, quality degradations, and weather distortions, in a time-, stereo-, and depth-consistent manner to the high-resolution Spring dataset, creating a suite of 20,000 corrupted images that reflect challenging conditions.” The 20 corruptions fall into five families — colour, blur, noise, quality and weather — and produce 40,000 frames, 20,000 stereo pairs. Each model is scored by how far its corrupted prediction moves from its own clean one.

The movement is large and it is not uniform. RAFT sits at a 6.79% 1px error on clean Spring. Under corruption it shifts by an average of 20.18 points of 1px outlier rate, and by 42.41 px of endpoint error under rain alone. SEA-RAFT is cleaner at 3.69% and shifts less on average, 17.52 points — but 60.52 points under snow. PWCNet shifts 90.80 points under snow. Accuracy and robustness do not track each other at all: “Transformer models (GMFlow, FlowFormer) perform best overall but struggle with noise”, while “Stacked models (e.g. SEA-RAFT, FlowNet2) uniquely resist noise”. A single leaderboard position tells you nothing about which of the six scenes below will break the system you are choosing.

Treat correspondence quality, visibility, and temporal use as separate questions.

The last step — testing the downstream task rather than the flow field — is the one teams skip as impractical. A public benchmark has built it in since 2007. Beside its two flow-accuracy measures, Middlebury reports two interpolation measures: an interpolation error IE, “the root-mean-square (RMS) difference between the ground-truth image and the estimated interpolated image”, and a gradient-normalized error NE whose scaling constant is set to ε = 1.0 graylevels per pixel squared. The evaluation site still describes itself in exactly those terms, as reporting “two measures of flow accuracy (endpoint and angular error) and two measures of interpolation quality”.

The instructive part is what scoring a downstream task cost. Interpolation quality is a different question from flow accuracy, so it needed different footage: “For IE and NE, which measure interpolation accuracy, we use only four of the above datasets (Mequon, Schefflera, Urban, and Teddy) and replace the other four with the high-speed datasets Backyard, Basketball, Dumptruck, and Evergreen.” Army, Wooden, Grove and Yosemite drop out. High-frame-rate sequences take their place. Expect the same in your own audit: the data that answers step 2 will usually not answer step 5.

A matching cost may be broad because texture repeats, while an ensemble may disagree because the input is out of distribution; these uncertainty signals do not mean the same thing.

Validate confidence against actual correspondence error and occlusion; downstream systems should be able to ignore or escalate unsupported vectors.

There is a published protocol for that validation. Seven authors set it out in 2018 as a sparsification plot. Remove pixels in order of estimated uncertainty and watch the average endpoint error of what remains fall. Then compare that curve against an “Oracle Sparsification” that removes pixels in order of their true ground-truth error. The gap between the two curves reduces to one number. “To quantify the sparsification error with a single number, we use the Area Under the Sparsification Error curve (AUSE).” On Sintel train clean, their FlowNetH-Pred-Merged reaches AUSE 0.095 at EPE 3.36 and 60 ms, against 0.212 for an empirical dropout estimate and 0.191 for an SGDR ensemble. A confidence estimate that good is worth acting on: “Removing 20 percent of the pixels results in halving the average endpoint error.”