Depth, Stereo, and Projective Geometry

Depth can be measured or inferred through different physical and statistical mechanisms.

The four routes differ above all in how much of an image they actually measure. KITTI's 64-layer, 10 Hz laser scanner, projected into the image, covers approximately 5% of it. A monocular network returns a number for every pixel of the same frame. Both outputs are called depth maps. Only one of them is mostly measurement.

For rectified stereo cameras, the same scene point appears at different horizontal positions; greater disparity generally indicates smaller depth when focal length and baseline are fixed.

Correspondence is difficult in textureless areas, reflections, repeated patterns, occlusions, and lighting differences. Calibration error is not a caveat to append. It is a measured bias with a measured cost.

The Middlebury 2014 stereo release is 33 six-megapixel datasets built to 0.2-pixel disparity accuracy. Building them, Scharstein and his co-authors found that standard OpenCV rectification did not leave the pairs aligned to anything like that tolerance. “At a resolution of 6 MP residual vertical disparities of several pixels are common, especially in image corners.”

Their bundle-adjustment refinement cut the average absolute y-error from 0.77 to 0.096 pixels, a factor of 8.1, and the maximum from 2.76 to 0.67, a factor of 4.1. Then they priced the difference. Re-running matching on the “imperfect” rather than the “perfect” rectification raised disparity error by 9% on the 18 datasets whose average absolute y-disparity was under 0.5 px. On the three whose y-disparity fell between 1.5 and 3.0 px, it raised the error by 154%.

Žbontar and LeCun report the same defect from the other side: vertical disparity errors of up to nine pixels from standard OpenCV rectification on Middlebury. They survive it by deliberately injecting a small vertical offset during training. A matcher tuned on a well-rectified pair is being asked, in the field, to search a line that is not where it was told.

Depth outputs can look similar while supporting different product claims.

The metric row's requirement — scale information supplied from somewhere outside the image — is exactly the requirement Apple's Depth Pro was built to remove. It is worth seeing what a current system actually claims rather than leaving the requirement abstract. Apple published Depth Pro at ICLR 2025, with code and weights released, and the abstract does not hedge: “The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics.” The model produces a 2.25-megapixel depth map in 0.3 seconds on a standard GPU, and estimates focal length from the single image itself.

Read carefully, that moves where the scale comes from without abolishing the question. The metres are now inferred by the model from the image, rather than handed to it by a calibration file. That is a different provenance for the same number. Provenance is what the four rows above are really distinguishing.

These conditions challenge physical sensors and learned predictors in different ways, and each one breaks a specific assumption rather than merely adding noise.

Use error measures and tasks that expose range and boundary behavior.

Step 2 is not a counsel of thoroughness. Error genuinely has a shape in distance, and two independent teams measured that shape on the same sensor. Khoshelham and Oude Elberink put the Kinect result in their abstract: “The random error of depth measurement increases with increasing distance to the sensor, and ranges from a few millimeters up to about 4 cm at the maximum range of the sensor.” That maximum range is 5 m. Their conclusions turn the curve into an operating limit: for indoor mapping applications, the data should be acquired within 1–3 m of the sensor.

A second team fitted the same behaviour to an equation. Measuring lateral and axial noise on a rotating planar target at z-distances of 0.5–2.75 m across a range of surface angles, Nguyen and his co-authors gave the axial component as σz(z,θ) = 0.0012 + 0.0019(z − 0.4)² metres for 10°≤θ≤60°. That is quadratic in distance. Lateral noise grew only linearly.

A single aggregate error figure averages the near range, where this sensor is good to millimetres, with the far range, where it is out by centimetres. It reports a number that describes neither.

Active sensors may fail on sunlight, dark surfaces, reflective materials, or ranges outside specification; stereo fails differently, while monocular predictions can fill holes using learned priors.

The proportions involved are not close. A 93k-frame semi-dense depth dataset derived from KITTI raw came with a measurement of what the 64-layer, 10 Hz laser actually delivers to an image: “If projected to the image, the depth measurements cover approximately 5 % of the image.” Uhrig and his co-authors then priced the alternatives against each other. Raw LiDaR: density 4.0%, at a 1.62% KITTI outlier rate. Their cleaned annotation: 16.1% at 0.31%. Semi-global-matching stereo: 82.4% density at 4.52%. Density and correctness trade directly.

Wong and Soatto, evaluating a different method, describe the same KITTI input independently as sparse maps with a density of about 5%, with semi-dense depth existing only for the lower 30% of the image.

Fusion should preserve source validity and disagreement rather than silently replacing every missing measurement, and confidence must be evaluated for each source condition. A pipeline that hands downstream code one dense array has thrown away the distinction between the 4.0% that a laser returned and the remainder that an algorithm proposed. Those two populations have outlier rates that differ by more than a factor of two.

Depth systems need more than colorful qualitative examples, and the condition slices they are asked for are not hypothetical.

The Insurance Institute for Highway Safety first ran its nighttime pedestrian autobrake evaluation on 30 August 2022. 23 vehicles went through it: four rated superior, seven advanced, eight basic and four no credit. The comparison that matters is with the same test in daylight. “Eight of the 12 vehicles that earn a basic rating or no credit in the nighttime test got superior or advanced ratings in the daylight evaluation.”

The Institute's field research points the same way. Pedestrian crash rates were 27% lower for vehicles with pedestrian AEB across all light conditions, and no different at all for crashes at night on roads without streetlights.

A regulator then wrote the slice into a binding test. NHTSA cited the finding that systems performing well in daylight performed poorly in the dark when it adopted FMVSS No. 127 on 9 May 2024: “Importantly, this final rule requires that PAEB systems be able to avoid pedestrian crashes in dark testing conditions”. Compliance is due 1 September 2029, and the rule is estimated to save at least 362 lives and mitigate 24,321 non-fatal injuries a year. A lighting slice omitted from a test plan is a lighting slice somebody else may eventually mandate.