Vision Evaluation and Error Analysis Across Tasks

Choose metrics that reflect the structure of the prediction. Classification compares labels, scores and decisions per example: confusion matrix, precision and recall, calibration, with a defect class as the everyday case. Detection and segmentation match spatial predictions to objects or pixels under IoU thresholds, scoring AP and recall and slicing by boundary or size, as with a lesion mask. Tracking and retrieval evaluate ranked, linked or identity-consistent sequences — rank metrics, ID switches, track fragmentation, vehicle tracking. Generation and restoration have no single ground truth. They are judged instead on fidelity, diversity, adherence to a prompt or source contract, memorization and safety, as in an image editor.

Which metric a field agrees to use is itself a contested, dated decision. Multi-object tracking went years without a settled answer. The HOTA paper, published in the International Journal of Computer Vision in 2020, opens with the diagnosis: “Multi-object tracking (MOT) has been notoriously difficult to evaluate. Previous metrics overemphasize the importance of either detection or association.” MOTA over-weights detection. IDF1 over-weights association. HOTA decomposes instead into sub-metrics covering five basic error types, so detection quality and association quality can be read apart.

The KITTI tracking benchmark switched its official ranking to HOTA on 25 February 2021. What the leaderboard measures changed that day. Nothing in any submitted model did.

On the generation side the instability is formal. Because the FID bias term depends on the model being scored, two systems' FIDs are not two readings of one instrument.

A model can pass one layer and fail another. Data validity asks whether the sample is representative, correctly labeled, independent and well documented. Task performance covers accuracy, localization, ranking, calibration and robustness. Decision policy is thresholds, abstention, capacity and fallback. Human interaction is correction, anchoring, trust, workload and accessibility. Operational outcome is safety, delay, cost, equity, quality and incident rate.

Calibration shows how a layer can be measured, declared settled, and then need measuring again. A 2017 paper at ICML reported a regression nobody was looking for: “While neural networks today are undoubtedly more accurate than they were a decade ago, we discover with great surprise that modern neural networks are no longer well-calibrated.” Guo and colleagues put expected calibration error “typically between 4 to 10%” at 15 bins, across standard vision and NLP datasets. The most effective fix was single-parameter temperature scaling. One parameter repaired a failure that accuracy had entirely concealed.

Four years later the question was re-run at scale. Minderer and colleagues took 180 models from 16 families and evaluated each on 79 ImageNet-scale datasets and 28 metric variants. They found the most recent models, notably the non-convolutional ones, among the best calibrated.

So the 2017 finding was a fact about the architectures of 2017. A product that inherited its threshold policy from that generation is running a decision layer tuned to a measurement that has since partly reversed. Each layer carries its own date.

Object detection may match a prediction to one ground-truth object using IoU and confidence order. Tracking adds temporal identity. Retrieval uses ranked relevance.

How that temporal identity is weighted against detection is a live design choice, not a convention. MOTA over-weights detection. IDF1 over-weights association. HOTA decomposes into sub-metrics covering five basic error types, so neither swamps the other. The KITTI Vision Benchmark Suite now states “We rank methods by HOTA” on its multi-object tracking evaluation page, and dates the switch to 25 February 2021. A tracker's rank has a metric and a date attached to it.

Inspect alternative thresholds and duplicate handling. A model can improve at loose overlap while degrading the precise localization the product actually requires.

Define slices before launch and expand them from observed failures. The size of what an average can hide has been measured twice, by different institutions with different methods and different subjects.

NIST's Face Recognition Vendor Test ran 18.27 million images of 8.49 million people through 189 mostly commercial algorithms from 99 developers. The report came out in December 2019. Its executive summary states: “Across demographics, false positives rates often vary by factors of 10 to beyond 100 times.”

A far smaller study had reached the same territory two years earlier. Buolamwini and Gebru measured error rates up to 34.7% for darker-skinned females on three commercial gender classifiers, against a maximum of 0.8% for lighter-skinned males.

A hundredfold spread in false positives and a 34.7%-against-0.8% gap are both compatible with a respectable aggregate score. That is the argument for defining slices in advance. An average does not report its own variance across the groups inside it.

Error analysis should narrow causes rather than produce a gallery of bad examples.

A toolbox for exactly that arrived at ECCV in 2020. TIDE's abstract states the contribution precisely: “We segment errors into six types and, crucially, are the first to introduce a technique for measuring the contribution of each error in a way that isolates its effect on overall performance.” The six are Cls (classification), Loc (localization), Both, Dupe (duplicate), Bkg (background) and Miss (missed ground truth). Each is scored by the mAP that removing it would recover.

That isolation is the point. Two detectors at the same mAP can be spending their error budget in different places, and the total does not say which. A gallery of bad images cannot rank the fixes. A per-type mAP recovery can.

The loop that follows has five steps. Sample errors systematically, stratifying by type, score, severity and slice rather than selecting memorable cases. Verify labels and inputs, separating annotation error, ambiguity, capture failure and model failure. Cluster mechanisms by scale, context, boundary, class, device or workflow. Test hypotheses with ablations, counterfactuals, retraining or policy changes on held-out data. Then prioritize by impact, estimating frequency, harm, fixability, review burden and regression risk.

A mobile robot detects products, reads labels, estimates stock, and sends uncertain cases to employees. Rather than inventing a metric list, copy the shape of a protocol a regulator has already accepted.

On 11 April 2018 the FDA granted De Novo DEN180001 to IDx-DR, an autonomous system for diabetic retinopathy. The thresholds it had to clear were fixed before the trial ran: 85.0% sensitivity and 82.5% specificity. Of 900 participants enrolled at 10 primary-care sites, 819 were fully analyzable (92%). The decision summary reports the headline result in one sentence: “IDx-DR correctly identified 173 of the 198 fully analyzable participants with fundus mtmDR+, thus observed sensitivity was at 87.4% (95%CI, 81.9% - 92.9%).”

The rest of the scorecard is just as specific. Observed specificity was 89.5% (95% CI 86.9–93.1%). The enrichment-corrected figures were 87.2% and 90.7%. Imageability was reported too, at 96.1% (819/852), together with PPV 72.7% and NPV 95.7% at 23.8% prevalence.

Notice what that carries besides an accuracy figure: thresholds set in advance, confidence intervals on every rate, the fraction of cases the system could not analyze at all, corrected as well as observed numbers, and predictive values stated at a named prevalence. Build the shelf audit the same way.