Computer Vision Metrics Across Tasks

Object detection evaluation first decides whether a predicted box matches a reference object. That decision comes from a rule somebody wrote down. The PASCAL Visual Object Classes challenge wrote it down in 2010. A detection counted as correct only if the overlap ratio between the predicted and the ground-truth box exceeded 0.5. Repeats were charged for, and the paper says so plainly: “Multiple detections of the same object in an image were considered false detections e.g. 5 detections of a single object counted as 1 correct detection and 4 false detections”.

Then the rule moved and the detections stayed where they were. On the class car the measured AP “drops steeply for thresholds above 50%”, and lowering the requirement to 10% raised measured AP by around 7.5%. Same boxes, same model, different number. A small localization shift can push overlap below the threshold and convert a plausible detection into a false positive plus a false negative. The detector did not get worse. The rule moved.

COCO set the scale most detectors are now measured on. Released in 2014, it “contains photos of 91 objects types” and carries “a total of 2.5 million labeled instances in 328k images”. None of that fixes the matching rule. The threshold that turns a predicted box into a hit comes from the evaluation protocol, not from the dataset.

Different outputs require different notions of agreement. Classification compares class decisions or probabilities for an image. Detection matches predicted boxes or regions to reference instances. Segmentation compares pixel or region assignments. Pose measures landmark distance under scale or visibility rules. Tracking has to judge detection and identity continuity at once.

That last row is the one a public benchmark has recently rewritten. The older tracking measures split the task badly, and in 2020 seven researchers proposed a replacement: HOTA, Higher Order Tracking Accuracy. Luiten and his co-authors give the diagnosis in one line: “Previous metrics overemphasize the importance of either detection or association.” HOTA decomposes into detection, association and localization sub-metrics instead.

The KITTI multi-object tracking benchmark now says “We evaluate submitted results using the metrics HOTA, CLEAR MOT and MT/PT/ML. We rank methods by HOTA.”, and it dates the switch: “On 25.02.2021 we have updated the evaluation to use the HOTA metrics as the main evaluation metrics”. A tracker's rank on that leaderboard changed on a day in February 2021. Not one line of its code changed.

A detector's AP can change without retraining. AP is produced by a published parameter block, not by the model.

COCO's detection evaluation page defines 12 metrics. The primary AP is averaged over 10 IoU thresholds, .50 to .95 in steps of .05, and the page gives the reason: “Averaging over IoUs rewards detectors with better localization.” The same page fixes the area bands. Small is area below 32^2, medium 32^2 to 96^2, large above 96^2. It caps the count too: “All metrics are computed allowing for at most 100 top-scoring detections per image (across all categories)”. And it records that roughly 41% of COCO objects are small, 34% medium and 24% large.

The two protocols part company at the threshold. Oksuz and colleagues describe the split the same way: “While a single intersection-over-union (IoU) threshold, which is 0.5, is used in PASCAL VOC [8]; a range of IoU thresholds (from 0.5 to 0.95) are used in MSCOCO”. Not one item in that block is a property of your detector.

Both compare predicted and reference regions. Intersection over Union divides the intersection by the union of the two regions. It ranges from zero to one, penalizes extra and missing area, and is the quantity a matching threshold is applied to. The Dice coefficient is twice the intersection divided by the total predicted and reference area. It is closely related to F1 computed over pixels, is standard in medical segmentation, and reads higher than IoU for the same overlap.

They are the same family, and they share its blind spot. Both score area. Neither scores where the boundary sits. That is why the Metrics Reloaded consortium recommends pairing an overlap-based metric with a boundary-based one, taking normalized surface distance as the default partner. Area says how much of the shape you got. It does not say where you got it.

Image-generation metrics such as FID compare feature distributions, and they move on things that are not the model. In 2022 three researchers saved the same FFHQ images as lossless PNG and as JPEG at quality 100, 90 and 75, then measured FID against the PNG version: 0.23, 6.08 and 20.96, with PSNR at or above 39 dB. Parmar and his co-authors say what happened: “While the images are perceptually similar, this induces changes in the Inception-V3 activations, resulting in large FID.”

The size of that artefact is the point. StyleGAN2 scores around 3 on FFHQ, and “variations across recent methods are typically within 1 FID on FFHQ”. Saving through PIL's default quality of 75 “results in a high score (21)”. A file-format decision made before evaluation begins is larger than the entire measured spread between competing generators.

A separate group at Google Research found another kind of instability. FID contradicts human raters. It does not reflect gradual improvement of iterative text-to-image models, it does not capture distortion levels, and it “produces inconsistent results when varying the sample size”.

Human studies may assess realism, prompt adherence, preservation, preference, or harmful artifacts. A lower distributional distance does not prove that every image is faithful, diverse, safe, or free of memorization.

Combine benchmark geometry with the conditions of image formation, in five steps: freeze annotation rules; define matching; report scale slices; stress acquisition; connect to use.

Step 1 is not bookkeeping. Northcutt and colleagues counted 2,916 validated label errors in the ImageNet validation set, about 6% of it. Of those, 1,428 were correctable and 597 multi-label. With corrected labels, ResNet-18 outperforms ResNet-50 once the prevalence of originally mislabeled test examples rises by just 6%. A separate team re-annotated the same validation set and found recent classifiers' gains to be “substantially smaller than those reported on the original labels”. The reference the geometry is compared against is itself an artefact with an error rate. Repair it and the ranking moves.

Step 3 is what catches a single accuracy figure covering two populations. In 2018 Buolamwini and Gebru audited three commercial gender classifiers on the Pilot Parliaments Benchmark. Their abstract: “We evaluate 3 commercial gender classification systems using our dataset and show that darker-skinned females are the most misclassified group (with error rates of up to 34.7%). The maximum error rate for lighter-skinned males is 0.8%.” The worst case was IBM's, on darker-skinned females. Between best- and worst-classified groups the gap was 34.4 points. One overall accuracy number shows none of that.

Step 4 has been run at national scale. NIST pushed 18.27 million images of 8.49 million people through 189 mostly commercial algorithms from 99 developers. Its report, in December 2019: “We found empirical evidence for the existence of demographic differentials in the majority of contemporary face recognition algorithms that we evaluated”. False positive rates often varied by factors of 10 to beyond 100 across demographics. So slice by lighting, compression, blur, device, geography and domain shift. Then measure latency, abstention, human correction and downstream task impact.