Camera Geometry, Coordinates, and Calibration

A detection at pixel coordinate (620, 410) says where an image sample lies; it does not directly state the object's three-dimensional position, size, or distance.

Converting pixels into spatial estimates requires a camera model, coordinate frames, and additional constraints such as known geometry, multiple views, or depth sensing.

Depth from a single image is “inherently ambiguous, with a large source of uncertainty coming from the overall scale”. That is the 2014 paper's own account of the problem, and it is why Eigen and colleagues “apply a scale-invariant error to help measure depth relations rather than scale”. A model can be right about which surface is nearer and still have no idea how far away either one is.

The useful part is that they measured the hole instead of naming it. On NYU Depth their log-space RMSE was 0.28. When an oracle substituted the ground-truth mean log depth into each prediction, the error fell to 0.22. The same substitution took the Make3D baseline from 0.41 to 0.33, a 20% relative improvement. Roughly a fifth of the total error was nothing but the unknown global scale. For the full coarse+fine model, the reported scale-invariant RMSE on NYU Depth is 0.219, against an abs-relative difference of 0.215 and a linear RMSE of 0.907 m.

They put it plainly.

“The global scale of a scene is a fundamental ambiguity in depth prediction. Indeed, much of the error accrued using current elementwise metrics may be explained simply by how well the mean depth is predicted.” — Eigen and colleagues, 2014.

Most geometry bugs are frame or convention bugs rather than failures of advanced mathematics. The chain starts at image coordinates, the pixel columns and rows on the stored grid. It passes through normalized camera coordinates, where intrinsics are removed so points represent viewing directions. Then the camera frame, then the world or rig frame that extrinsics place the camera in, and finally the object frame a pose may be expressed against.

The KITTI recording platform makes that chain concrete, and makes the convention problem hardware rather than notation. The rig, described in 2013, carries four 1.4-megapixel Point Grey Flea2 cameras with 4 mm lenses on two stereo rigs of about 54 cm baseline, plus a Velodyne HDL-64E and an OXTS RT3003 GPS/IMU. Three sensor families, three different ideas of which way is forward. The cameras use x=right, y=down, z=forward. The Velodyne uses x=forward, y=left, z=up. So does the GPS/IMU. Nothing in an image or a point cloud announces which convention produced it. Only the declared transform does, which is why the declaration is part of the calibration and not part of the documentation.

The extrinsics were not established once and trusted, either. The team re-calibrated the sensors every recording day and shipped a separate calib_cam_to_cam.txt for each one. They gave the reason: “To avoid drift over time, we calibrated the sensors every day after our recordings. Note that even though the sensor setup hasn't been altered in between, numerical differences are possible.”

Those per-day differences are large enough to be visible downstream. In 2019 the Monodepth2 authors simply flattened them: “We use the same intrinsics for all images, setting the principal point of the camera to the image center and the focal length to the average of all the focal lengths in KITTI.” A rig that nobody touched still produced a spread of intrinsics wide enough that a later group averaged them away.

These parameter groups answer different geometric questions. Intrinsics describe how camera-frame rays map to image coordinates: focal lengths and principal point, dependent on image scaling and crop, used to back-project pixels. Extrinsics describe the rotation and translation between two frames — placing the camera in a rig or world, changing after physical movement, requiring a declared direction, used for sensor fusion. The distortion model approximates the lens's departure from ideal projection: usually radial terms, sometimes tangential ones, valid only for the calibrated lens setup, used for rectification.

A printed sheet of paper is enough to estimate the first two. Zhang's method, from 1998, is still the engine OpenCV's calib3d module names in its notation for the camera intrinsic matrix, and it is the pinhole-plus-radial-and-tangential model MathWorks documents in the Computer Vision Toolbox. Its abstract states the whole contract: “The technique only requires the camera to observe a planar pattern shown at a few (at least two) different orientations. Either the camera or the planar pattern can be freely moved. The motion need not be known. Radial lens distortion is modeled.”

The real-data numbers matter more than the convenience. The camera was a 640×480 PULNiX with a 6 mm lens. The target was 17 cm × 17 cm, 8×8 squares, 256 corners. From two images the closed-form solution reached an RMS reprojection error of 0.761 px, and the maximum-likelihood refinement cut it to 0.295 px. That much is the expected story of a refinement step.

The distortion coefficient is the warning. From those same two images the closed-form radial coefficient k1 came out at +0.161, pincushion. The refined estimate was −0.227, barrel. Same camera, same corners, opposite sign: an intermediate fit had the lens bending the wrong way while still producing plausible sub-pixel output. It is not a quirk of one configuration either. Zhang ran the algorithm on the first 2, 3, 4 and all 5 of his images, and with all five the pattern repeats — +0.136 before refinement, −0.228 after, with RMS error 0.881 px falling to 0.335 px. A parameter file that looks finished can still be wrong about the direction the lens curves.

Each failure can produce plausible overlays while corrupting metric conclusions, and none of them is hypothetical.

On 5 March 2026 NHTSA acknowledged Ford safety recall 26V123 — Ford's own 26C11 — covering 889,950 vehicles: 2020-2022 Escape and Lincoln Corsair, 2020-2024 Explorer and Lincoln Aviator. The acknowledgement letter states the problem: “The image on the center display may flip or invert, resulting in an incorrectly displayed rearview image when the vehicle is placed in reverse. As such, these vehicles fail to comply with the requirements of Federal Motor Vehicle Safety Standard number 111, "Rear Visibility."” The component was logged as BACK OVER PREVENTION:SOFTWARE. The camera worked. The display worked. Only the orientation of the image was wrong, and that alone put nearly nine hundred thousand vehicles outside a federal safety standard.

Treat calibration as a monitored asset rather than a one-time setup ritual. Declare conventions: axis directions, units, image origin, pixel-center rules, transform direction. Estimate parameters with suitable targets, viewpoints and optimization. Inspect residuals spatially, across the frame rather than as one average. Test metric tasks on independent fixtures. Monitor the physical changes — focus, zoom, crop, resolution, mount — that invalidate what you estimated.

Steps three to five have a canonical demonstration of what skipping them costs. Hubble's spherical aberration came from a single uncross-checked metrology error. The field lens in the reflective null corrector used to figure the 2.4 m primary mirror sat approximately 1.3 mm too far from the lower mirror, and that displacement accounts for the observed blur in detail. NASA's board of investigation, chaired by Lew Allen, reported that in November 1990. The reference used to shape the mirror was never itself verified.

The independent checks were not missing. Two other optical tests, an inverse null corrector and a refractive null corrector, both showed the error at the time. The report's executive summary records what happened next: “Both indicators of error were discounted at the time as being themselves flawed.” The as-built conic constant of the primary came out near K = −1.0144, against a design value of −1.0023: about 253 to 277 nm rms of wavefront error, and roughly 2.3 µm of surface error at the mirror's edge. Step 4 is not a formality. A disagreeing fixture is evidence, not noise.

Step 4 also has a published template. FMVSS No. 111 turns a rearview camera's geometry into a pass/fail test with fixed targets, fixed distances, an angular-size floor and a timing bound. That is the shape any internal metric acceptance test should borrow, whether or not the product is a vehicle.

A homography can map points between views of the same plane or between images created by pure camera rotation; it is widely used for document rectification, floor-plane mapping, and panorama stitching.

It becomes misleading when objects leave the assumed plane or the scene contains significant parallax. The validation set must include those violations.

A reproducible geometry package should include more than a parameter file. FMVSS No. 111 shows what a binding version of one looks like. NHTSA's final rule of 7 April 2014 expanded the required field of view to a 10-foot by 20-foot zone behind the vehicle, effective 6 June 2014, with phase-in beginning 1 May 2016 and full compliance required on 1 May 2018. The codified test places seven cylinders, each 0.8 m tall and 0.3 m in diameter, at 0.3 m, 3.05 m and 6.1 m behind the rear bumper and ±1.52 m off the centreline. The rearview image must show them, and it is graded on angular size: “All three test objects located at positions A, B, and C specified in S14.1.4 shall average not less than 5 minutes of arc; and Each individual test object (A, B, and C) shall not be less than 3 minutes of arc.” The image must also appear within 2.0 seconds of the start of a backing event. Distances, target sizes, an angular floor and a deadline. That is what an independent metric check looks like when someone has to enforce it.