Computer Vision as Measurement and Decision Engineering

A warehouse camera records a pallet near a loading bay; the image depends on lens position, exposure, motion, compression, and which parts of the scene are hidden.

A model may detect a pallet, estimate its distance, track its movement, or decide whether a safety zone is blocked: those are different tasks with different evidence and failure costs.

The field's habit of measuring itself came from one benchmark. Russakovsky and eleven co-authors described the ImageNet Large Scale Visual Recognition Challenge in 2015 as “a benchmark in object category classification and detection on hundreds of object categories and millions of images”. It “has been run annually from 2010 to present, attracting participation from more than fifty institutions”. Their paper compares “the state-of-the-art computer vision accuracy with human accuracy”. That comparison became the scoreboard.

A number that large is still a number about one collection of photographs. And how large the gap between two collections really is has itself been contested in print. Recht and three colleagues rebuilt the ImageNet and CIFAR-10 test sets, re-running the original collection procedures. They reported in 2019: “We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet.” Nothing about the models had changed. Only the photographs had.

Then a separate MIT group re-analysed the same ImageNet-v2 replication. Engstrom and five co-authors corrected for statistical bias in the replication procedure itself and concluded in 2020 that “only an estimated 3.6% ± 1.5% of the original 11.7% ± 1.0% accuracy drop remains unaccounted for”. Two labs, one benchmark, two different sizes for the same hole. An accuracy figure is a property of a dataset and of how that dataset was assembled. The measurement of the dataset is itself a measurement, with error bars.

This map prevents a model score from being mistaken for direct knowledge of the scene. Six layers stand between the world and an action. In the physical scene, objects, surfaces, light, motion, and occlusion exist before any image is captured. In the imaging process, optics, sensor settings, viewpoint, timing, and compression create a partial measurement. In the dataset and labels, sampling and annotation convert selected measurements into training evidence. In the representation and model, features and learned parameters transform pixels into task-specific outputs. In the decision policy, thresholds, rules, temporal logic, and human review determine what happens next. In the operational outcome, the system affects safety, cost, access, diagnosis, or user experience.

The layers are easiest to see when they come apart under load. On 18 March 2018, in Tempe, Arizona, a self-driving test vehicle struck a pedestrian. The NTSB's preliminary report, HWY18MH010, released that May, walks down the stack in four pages.

The imaging layer worked. The Uber ATG automated driving system registered radar and LIDAR observations of the pedestrian about 6 seconds before impact, travelling at 43 mph. The representation layer did not settle: “As the vehicle and pedestrian paths converged, the self-driving system software classified the pedestrian as an unknown object, as a vehicle, and then as a bicycle with varying expectations of future travel path.” The decision layer reached its own conclusion 1.3 seconds before impact — that emergency braking was needed — and had no authority to act on it. Uber had disabled emergency braking manoeuvres under computer control.

Every layer failed differently, and only the last one produced the outcome. In December 2020 GAO placed the same crash among accidents “caused in part by the drivers' overreliance on the automated technologies in the vehicle”. A single verdict of "the vision system failed" would have located the fault in the wrong layer.

One image can support several tasks. Yet each task defines a different output contract. Classification predicts which categories describe an image or crop and returns one or several labels; its typical failure is a background shortcut, and its evidence is class and slice performance. Detection locates and classifies individual objects, returning boxes, labels, and scores, and typically misses small objects. Segmentation assigns labels to pixels or instances, returning masks or regions, and typically erodes boundaries. Geometry and motion estimate depth, pose, correspondence, or trajectories, returning coordinates or fields, and typically fail through scale or drift error. Retrieval and generation return ranked items or generated images and typically fail through semantic mismatch.

"Background shortcut" is the cheapest of those phrases to write and the most expensive to discover late. One study measured it precisely. Zech and five co-authors trained pneumonia-detection CNNs on 158,323 chest radiographs from three institutions — 112,120 from NIH, 42,396 from Mount Sinai, 3,807 from Indiana University — and found better internal than external performance in 3 of 5 natural comparisons. Then they asked the networks a question nobody had trained them to answer. Their 2018 paper in PLOS Medicine reports the answer: “CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs.”

The case became the standard illustration of shortcut learning. Geirhos and colleagues, surveying the phenomenon in 2020, restated it as a model that “had unexpectedly learned to identify particular hospital systems with near-perfect accuracy”, by picking up a hospital-specific metal token. The classification contract had been satisfied on the internal test set. The measurement question — does this chest have pneumonia — had not been answered at all. The output contract tells you what shape the answer has. Only slice evidence tells you what the answer is about.

Human observers resolve many visual ambiguities using context that a deployed system may not possess. How much that context is worth can be measured, and on faces it has been. NIST's December 2019 report on demographic effects in face recognition, NISTIR 8280, ran four operational US government datasets through the largest test of its kind: “Together these datasets allowed us to process a total of 18.27 million images of 8.49 million people through 189 mostly commercial algorithms from 99 developers.”

The headline result is not an average, and that is the point. “Across demographics, false positives rates often vary by factors of 10 to beyond 100 times”. They run highest in West and East African and East Asian faces on the high-quality application photos, and lowest in Eastern European faces. False negative differentials usually vary by factors below 3. One reported accuracy number can sit on top of a hundredfold spread in the error that actually causes harm.

Use this sequence to prevent a camera project from becoming an untestable model demo. First, define the decision: state what action or judgment the system must support. Second, specify the visible evidence: list viewpoints, sensors, timing, resolution, and metadata available at decision time. Third, choose the output contract: decide whether the product needs labels, boxes, masks, coordinates, rankings, or generated content. Fourth, enumerate difficult conditions, including occlusion, blur, rare objects, domain shifts, and ambiguous annotations. Fifth, connect metrics to consequences: map task errors to missed events, false interventions, review load, latency, and safety.

One authorized system shows what steps one and three look like when they genuinely come first. On 11 April 2018 the FDA granted De Novo request DEN180001 to IDx LLC for IDx-DR, with the decision recorded as "granted (DENG)". It created a new device classification named “diabetic retinopathy detection device”. The output contract in that system is the decision itself: it emits a two-option referral rather than a score for a clinician to threshold.

The pivotal trial behind the authorization “enrolled 900 subjects, with no history of DR at primary care clinics”, at 10 such clinics. Abràmoff and four co-authors reported in npj Digital Medicine in 2018: “The AI system exceeded all pre-specified superiority endpoints at sensitivity of 87.2% (95% CI, 81.8–91.2%) (>85%), specificity of 90.7% (95% CI, 88.3–92.7%) (>82.5%), and imageability rate of 96.1% (95% CI, 94.6–97.3%)”.

Read that sentence for its structure rather than its numbers. The endpoints were pre-specified, with a threshold to beat written down before the trial ran, because the decision the device makes had been defined before the images were collected. Imageability sits in the list beside sensitivity and specificity. That is the imaging layer being scored as part of the product, rather than assumed away.

This course focuses on visual evidence, task design, image and video data, geometry, evaluation, domain-specific systems, and deployment; it uses architecture concepts only when they clarify a vision failure.

The deeper mechanics of CNNs, Transformers, autoencoders, and diffusion networks belong to the architecture path. Other paths cover generic optimization, governance, and MLOps more fully.