Object Detection: From Presence to Localization

An image classifier can report “forklift present” while ignoring whether the vehicle blocks an aisle, appears twice, or occupies three pixels. Detection adds location, instance identity and count to the output contract. What a badly written contract costs is on the public record.

A test vehicle in Tempe, Arizona killed Elaine Herzberg on 18 March 2018. Uber ATG's automated driving system first detected her 5.6 seconds before impact. It never classified her as a pedestrian. The label alternated among vehicle, bicycle and other. “The system never classified her as a pedestrian—or correctly predicted her path—because she was crossing N. Mill Avenue at a location without a crosswalk, and the system design did not include consideration for jaywalking pedestrians,” the National Transportation Safety Board wrote in 2019. The design also discarded tracking history for any object whose classification changed. So the identity of the thing in the road was thrown away and rebuilt on every flip. The system placed an object fully on the SUV's path only 1.2 seconds before impact.

Presence was never the failure. Something was detected for 5.6 seconds. Class, path and instance identity were coupled, so a detector that appeared to have 5.6 seconds of warning had 1.2. The task therefore depends on box conventions, matching rules, object visibility, image scale, and the number of candidate locations considered. It also depends on whether those fields are allowed to reset one another.

Each predicted instance combines several fields, and they have to be evaluated together. A class label names the category. A localization gives a box or another geometric extent. A confidence or ranking score orders candidates for thresholding. An instance identity separates multiple objects of the same class within the frame. Optional attributes may add occlusion, orientation, keypoints, masks or tracking features.

The Tempe timeline shows what happens when those fields are wired to each other. In the Uber ATG system the class label and the instance identity were coupled. Each flip among vehicle, bicycle and other discarded the tracking history and reset the object's predicted path. Five fields that look like five independent columns of a prediction record behaved in deployment as one field with four dependents. The 5.6 seconds of accumulated evidence collapsed to 1.2.

Modern systems vary in how candidate objects are represented and assigned. The three families come with published numbers rather than adjectives.

A two-stage proposal detector generates candidate regions before refining classes and boxes. Faster R-CNN folded the proposal stage into the network itself, and its abstract states the budget: “For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image,” wrote Ren and three co-authors in 2015. Three hundred proposals per image, five frames per second, 73.2% mAP. That is the shape of the trade, and it is a budget you can audit. The design then took the field. MSRA won the ILSVRC 2015 provided-data detection track with 0.620741 mean AP, ahead of Qualcomm Research at 0.535745 and CUImage at 0.527113, on a system it describes as based on deep residual nets and the “Faster R-CNN” system from that paper. Amax won the additional-data track with 0.57848 mean AP, from a “Faster RCNN” baseline of its own (mAP 45.6% for VGG-16, 47.2% for GoogleNet).

A dense one-stage detector predicts objects across many spatial locations and scales. It is direct and often efficient. But it must assign every location as positive or negative, it produces many overlapping candidates, and it usually needs post-processing to reduce them.

A query-based set detector predicts a fixed set of object slots with global matching. DETR removed non-maximum suppression and anchor generation altogether, treating detection as direct set prediction under a bipartite-matching global loss. On COCO val, the 2020 paper reports, it matched a heavily tuned Faster R-CNN-FPN+ at 42.0 AP. The tie is the least interesting number in the table. DETR scored 20.5 AP on small objects against Faster R-CNN's 26.6, and beat it on large objects 61.1 to 53.4. The trade runs in both directions, and the single headline figure shows neither side of it. An independent group reported the same weakness a year later and named a cause: “However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps,” wrote Zhu and five co-authors of DETR in 2021.

Object count and scale interact with visibility, context, and annotation rules. Each of the six below has been measured on a public benchmark rather than merely noticed.

Start with a system that makes scale, matching, and threshold behavior visible. Audit the object-size distribution by class: widths, heights, areas, aspect ratios, crowding. Fix the image policy by declaring crop, resize, padding and multi-scale behavior. Train a standard baseline with its default assumptions preserved for comparison. Inspect matched errors, separating classification, localization, duplicate and missed-object failures. Then test deployment slices — distance, lighting, occlusion, site, device, object density.

Step five is the one that gets skipped, and there is a measurement of what skipping it hides. Vehicles with pedestrian automatic emergency braking had pedestrian crash rates 27% lower and injury crash rates 30% lower than vehicles without. That came from nearly 1,500 police-reported crashes involving 2017–2020 model-year vehicles, the first real-world study of the technology to cover a broad range of manufacturers, published by Jessica Cicchino of the Insurance Institute for Highway Safety on 3 February 2022. Then IIHS cut the data by one slice: “However, when the researchers looked only at pedestrian crashes that occurred at night on roads without streetlights, there was no difference in crash risk for vehicles with and without pedestrian AEB.” The slice is not a corner case. Cicchino's own summary is that “these systems are much less effective at night, when three-quarters of fatal pedestrian crashes happen”.

The follow-on bench testing reported to NHTSA says the same thing about the detector rather than the fleet: “of the 114 model year 2023 PAEB systems tested at night, 12 (11 percent) fully avoided the pedestrian mannequin in every test condition”, against 33 of 194 (17 percent) tested in daylight. An aggregate benefit of 27% and a slice benefit of zero are the same system, measured twice.

Background is not one coherent class. It contains confusing textures, object parts, reflections, signs, and unlabeled instances.

Hard-negative mining can help the model tell them apart. But mining against incomplete labels can punish correct detections. Review high-scoring negatives and verify annotation coverage before treating them as background.

The one-stage accuracy gap was traced to exactly this. Lin and four co-authors named “the extreme foreground-background class imbalance encountered during training of dense detectors” as “the central cause”. Their fix was the loss function: “by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples”, training concentrates “on a sparse set of hard examples” and prevents “the vast number of easy negatives from overwhelming the detector during training”. That is focal loss, published in 2017. The imbalance was the architecture problem all along.

Two datasets take the further step of treating a negative as something recorded rather than assumed. LVIS is built as a federated dataset — “A federated dataset is a dataset that is formed by the union of smaller constituent datasets, each of which looks exactly like a traditional object detection dataset for a single category,” write Gupta and colleagues in 2019. Each category gets a positive set exhaustively annotated for it, and a negative set verified to contain none of it. A detection of that category on an image in neither set is simply not scored. The paper announces 2.2 million instance masks over 1000+ entry-level categories in 164k images; the released LVIS v1.0 annotates 159,623 images with about 2 million masks over 1,203 categories.

Open Images V7 runs the same construct at production scale: 14,610,229 training boxes over 600 object classes on 1,743,042 training images, plus 303,980 boxes on 41,620 validation images and 937,327 on 125,436 test images. Boxes are exhaustive only for verified-positive classes. Its documentation states the design plainly: “As a result of our annotation process, each image is annotated both with verified positive image-level labels, indicating some object classes are present, and with verified negative image-level labels, indicating some classes are absent.” The two efforts confirm each other from opposite directions. The LVIS authors note that “With release v4, developed concurrently with our work, Open Images has used a federated dataset design for their object detection task.”

A serious review should reveal whether the detector's geometry matches the use case. Each artifact below has a published form you can point at rather than describe.