Detection Matching, Suppression, and Operating Points

A suppression rule has to survive real crowds, and somebody measured what that costs. CrowdHuman, a benchmark Megvii published in 2018, carries 15,000 training images with 339,565 annotated persons — 22.64 per image. Its authors then counted how often two distinct annotated people overlap each other: “Moreover, There are averagely 2.4 pairs whose IoU is greater than 0.5 in the CrowdHuman dataset.” The tables give that count as 2.40 and trace it down the threshold: 9.02 pairs per image above IoU 0.3, 4.89 above 0.4, 1.01 above 0.6.

Those pairs are the whole problem. A hard suppression rule cannot tell them from duplicates. The same count is 0.32 pairs per image in CityPersons, and 0.02 in COCOPersons and Caltech. So a rule that deletes every box overlapping the winner beyond IoU 0.5 is very nearly free on ordinary images, where 0.02 pairs per image are genuinely that close. In a crowd the same rule is deciding the fate of 2.40 real people per image, with identical geometry and identical scores in front of it.

None of that came from the class vocabulary. Candidate ranking, overlap rules and crowd geometry determined the operational count.

A prediction is not correct until it is paired with a reference instance under declared rules. The rules most benchmarks still run on were written down by the PASCAL VOC challenge in 2010. Detections are assigned to ground-truth objects in order of decreasing confidence. A detection counts as correct only if the overlap ratio a_o = area(Bp ∩ Bgt)/area(Bp ∪ Bgt) exceeds 0.5. What happens to the leftovers is settled with arithmetic rather than principle: “Multiple detections of the same object in an image were considered false detections e.g. 5 detections of a single object counted as 1 correct detection and 4 false detections—it was the responsibility of the participant’s system to filter multiple detections from its output.”

Five decisions are packed into that sentence and its neighbours. Order the predictions — VOC orders by decreasing confidence. Require category agreement. Measure geometric overlap, here a_o against 0.5. Enforce one-to-one matching, so that the second, third and fourth box on one object cannot be paid a second time. Then count what is left over: unmatched predictions are false positives, unmatched references are false negatives. Five boxes on one object score 1 and 4, not 5.

The rule travelled. The ImageNet Large Scale Visual Recognition Challenge paper records that “The criteria for object detection was adopted from PASCAL VOC (Everingham et al., 2010). It is designed to penalize the algorithm for missing object instances, for duplicate detections of one instance, and for false positive detections.” In both documents, filtering duplicates is explicitly the system's job and not the metric's. That is why suppression is a modelling decision that gets scored as if it were an implementation detail.

Intersection over Union compares overlapping area with the union of two boxes or masks. The same coordinate shift therefore costs a tiny object far more of its area than a large one. ILSVRC put the arithmetic on the page. A detection window of 20×20 pixels that fully contains a 10×10-pixel object — roughly 5 pixels of error per side, about average human annotation error — scores IoU 100/400 = 0.25. Under the standard rule it is counted wrong. “The PASCAL VOC metric uses the threshold thr(B) = 0.5. However, for small objects even deviations of a few pixels would be unacceptable according to this threshold.” The benchmark did not keep the constant. It replaced it with a size-dependent threshold, thr(B) = min(0.5, wh/((w+10)(h+10))), a relaxation that “changes the threshold only on objects which are smaller than approximately 25 × 25 pixels, and affects 5.5% of objects in the detection validation set”.

The 0.5 was a judgement call from the beginning. Everingham and his co-authors say the 50% threshold “was set deliberately low to account for inaccuracies in bounding boxes in the ground truth data”, and they measured what the dial is worth. Reducing the overlap threshold to 10% raises measured AP on the 'car' class by around 7.5%, while AP drops steeply for thresholds above 50%. Several points of reported performance sit in a tolerance nobody trained.

IoU thresholds therefore encode a notion of acceptable localization that may not match business tolerance. Report how the detector does across thresholds and object sizes.

COCO's own protocol makes the convention explicit. Its headline AP averages over ten IoU thresholds, from 0.50 to 0.95 in steps of 0.05. The evaluation code ships those thresholds as its default, so the convention travels with the tool. Scale is broken out separately: small means an area under 32 squared pixels, medium runs to 96 squared, large is anything above. Every metric is computed allowing at most 100 detections per image. That is a workload decision hiding inside an evaluation script.

Post-processing choices behave differently in crowded or overlapping scenes, and each of the four has a measured cost.

Hard NMS removes lower-scoring boxes above an overlap threshold. It is simple, fast, common in dense detectors, and sensitive to score ordering. The paper that proposed Soft-NMS states the failure mode plainly: in non-maximum suppression, “if an object lies within the predefined overlap threshold, it leads to a miss”. In CrowdHuman that lands on 2.40 real pairs per image.

Soft suppression reduces scores instead of deleting candidates immediately. It preserves alternatives, at the price of tuning and computation. Soft-NMS is “an algorithm which decays the detection scores of all other objects as a continuous function of their overlap with M”, and it obtains “consistent improvements for the coco-style mAP metric on standard datasets like PASCAL VOC 2007 (1.7% for both R-FCN and Faster-RCNN) and MS-COCO (1.3% for R-FCN and 1.1% for Faster-RCNN) by just changing the NMS algorithm without any additional hyper-parameters”. It is still threshold-dependent.

Box merging or voting combines coordinates from related candidates. Where the candidates really do describe one object, that stabilizes localization. Where they do not, it blurs separate instances. Var voting, published in 2019, replaces a surviving box's coordinates with a variance-weighted average of its neighbours' after suppression: “Applying var voting with the standard NMS improves the AP by 1.4%. Applying var voting after soft-NMS still can improve the AP by 1.3%.” On MS-COCO with VGG-16 Faster R-CNN the gain is almost entirely localization. AP50 rose by only 0.1, while AP75 rose by 1.8, AP_M by 1.8 and AP_L by 1.6. Merging improves where a box sits, not whether it exists.

Set prediction trains unique object slots through one-to-one assignment, which reduces the design pressure that produces duplicates in the first place. DETR matches predictions to objects bipartitely and needs no NMS. Its authors ran a standard NMS over its outputs anyway, at each decoder layer. It helped after the first layer, where the model cannot yet inhibit double predictions. Then it stopped helping: “It hurts AP in the last layers, as it incorrectly removes true positive predictions.” The duplicate decision moved into the matching objective. It did not stop being a decision.

Use matched predictions to assign each error to a useful category. There is a published instrument for doing exactly that. TIDE, a toolbox released in 2020, bins every false positive and false negative into six types — classification, localization, both, duplicate detection, background, and missed ground truth. It weights each by the ΔAP an oracle that fixed only that error would recover. The categories arrive in the units of the headline metric, so a taxonomy becomes a budget.

The definitions are mechanical. A duplicate detection error is a box with IoU_max ≥ t_f against a ground truth of the correct class that a higher-scoring detection has already matched. That is the same leftover the PASCAL VOC protocol turns into a false positive, now given its own line in the accounting.

Applied across seven detectors on COCO, TIDE traced one model's profile straight back to a post-processing constant. Of FCOS: “Note that it also has the highest duplication error because it uses an NMS threshold of 0.6 instead of the usual 0.5.” The distinguishing feature of that detector's error profile is a number in its post-processing, not something it learned.

The diagnostic order follows the same logic. Separate missed references, asking whether no candidate, a low score, or suppression caused the miss. Identify duplicate outputs, measuring how many predictions cluster on one true instance. Split class errors from localization errors — TIDE keeps classification, localization and 'both' apart precisely because a correct class with poor geometry needs a different remedy than class confusion. Analyze ranking, inspecting whether correct candidates exist but sit below false positives. Then recompute operating curves across thresholds, sizes and densities before believing any fix.

Average precision integrates precision-recall behavior as the score threshold changes. Its value depends on matching, IoU criteria, class averaging, interpolation, and ignored cases.

DETR shows how much a single number can hide. On COCO val it reaches 42.0 AP — exactly the score of the tuned Faster R-CNN-FPN+ baseline, at a comparable parameter count of 41M against 42M. The tie does not survive decomposition. AP_S is 20.5 for DETR against 26.6 for the baseline, and AP_L is 61.1 against 53.4. Two systems agree to the decimal on the headline figure while scoring 20.5 against 26.6 on small objects and 61.1 against 53.4 on large ones. A team that cares about small objects and a team that cares about large ones should not pick the same model here. The headline AP will not tell either of them that.

AP is excellent for standardized comparison, and its conventions travel with the evaluation code. The COCO API sets its thresholds under the header comment “[.5:.05:.95] T=10 IoU thresholds for evaluation”, which is most of why cross-paper comparison works at all. Yet it does not directly state false alarms per hour or missed hazards. Pair it with deployment-facing measures.

Operating-point analysis should translate curves into quantities that stakeholders can reason about. Every one of these has been measured somewhere, at a size worth knowing. The full var voting pipeline moved a baseline from 23.6 to 29.1 AP on MS-COCO, at an inference cost of 99 ms to 101 ms per image. About two milliseconds — and almost all of the benefit landing at tight IoU rather than loose.