Multi-Object Tracking and Identity

On 18 March 2018, in Tempe, Arizona, a vehicle controlled by Uber ATG's developmental automated driving system struck and killed a pedestrian. The system was not blind to her. It detected her 5.6 seconds before impact. Detection was not the failure.

Identity was. The National Transportation Safety Board investigated the crash and reported that “The ADS changed the pedestrian's classification several times, alternating between vehicle, bicycle, and other”. The software had been built so that a change of label erased the object's past: “If the perception process changed the classification of a detected object, it no longer considered the tracking history of that object when generating a new trajectory.”

With the history discarded at every relabelling, the system never predicted her path. Boxes kept arriving. The hypothesis that they described one moving person did not survive from one classification to the next. After the crash, ATG changed the system to retain tracking history across reclassification.

Many practical trackers alternate observation, prediction, assignment, and lifecycle management. (1) Detect observations — boxes, masks, or keypoints in the current frame. (2) Predict track states — use recent motion and uncertainty to estimate likely locations. (3) Compute association costs — combine geometry, appearance, class, and temporal consistency. (4) Assign observations — match detections to active tracks under gating and exclusivity rules. (5) Manage lifecycles — create, confirm, pause, recover, and terminate identities.

The Tempe fault sat in stage five, not stage one. A rule that ends an identity whenever the class label changes is a lifecycle rule, and the NTSB recorded its effect in one sentence: “with each change in object classification, the ADS perceived the pedestrian as a new object without considering its location history”. Nothing in the detector was wrong. The loop's bookkeeping was.

No individual cue remains reliable in every scene, and the cheapest cues carry more of the result than they look able to.

SORT made that point in 2016 with almost no machinery: a Kalman filter and the Hungarian algorithm, nothing else. It “updates at a rate of 260 Hz which is over 20x faster than other state-of-the-art trackers”. What moved the numbers was not the association machinery but what it was fed. Its authors say so directly: “detection quality is identified as a key factor influencing tracking performance, where changing the detector can improve tracking by up to 18.9%”.

Appearance came a year later. DeepSORT learned a deep association metric offline on a large-scale person re-identification dataset, then applied it during tracking by nearest neighbor queries in appearance space. The reported result: “our extensions reduce the number of identity switches by 45%, achieving overall competitive performance at high frame rates”. A 45% reduction is a large gain. It is not elimination. The remaining switches are the ones a product still has to survive.

Four families of evidence, each with the failure that ends it. A motion model predicts where an object should appear next: strong for smooth short-term movement, weak after long occlusion, sensitive to frame timing, and defeated by abrupt motion. An appearance embedding compares visual identity across observations. It is useful after displacement, confused by uniforms or look-alikes, unstable under pose and lighting, and it is the cue that raises privacy concerns. Spatial overlap matches nearby boxes or masks: fast and interpretable, dependent on detector jitter, and precisely the cue that fails at crossings and after missed frames. Scene geometry — lanes, entrances, depth, camera calibration — constrains plausible motion and can improve multi-camera tracking. It needs a reliable environment model, and it fails after the camera moves.

Review errors around events rather than only isolated frames.

(1) Separate detection and association: establish whether the object was missed outright or matched to the wrong track, because the two failures have different fixes. (2) Inspect lifecycle events — births, confirmations, gaps, recoveries, and terminations. Tempe is the extreme case of a lifecycle bug: the births were spurious, one per reclassification, and the NTSB traced the collision to that rule rather than to a missed detection. (3) Stress crossings and occlusions, and set the bar with a real distribution rather than an impression of "dense". MOT20, published in March 2020, is eight sequences — four train, four test — totalling 13,410 frames, 3,457 pedestrian tracks and 1,652,040 annotated pedestrian boxes, 2,102,385 annotations across all classes. Its introduction states the reason it exists: “This benchmark addresses the challenge of very crowded scenes in which the density can reach values of 246 pedestrians per frame.” Xiao and Luo, writing in Entropy, give the same figure its scale: “the mean pedestrian density reaches 246 per frame, which is 10 times higher than in MOT16 as well as MOT17”. (4) Measure time consequences: quantify identity switches, fragmentation, latency, and delayed recovery. (5) Validate product aggregates: counts, dwell time, trajectories, alerts, and cross-camera handoffs.

Some summaries emphasize detection quality, others association quality, and others trajectory completeness; a tracker can improve one while degrading another. Report identity switches, fragmentation, false tracks, missed duration, localization, and application-specific trajectory errors; state whether evaluation uses public detections, private detections, or end-to-end outputs.

HOTA, published in the International Journal of Computer Vision in February 2021, was written around exactly this complaint. Its objection to what came before is that “previous metrics overemphasize the importance of either detection or association”. HOTA instead “explicitly balances the effect of performing accurate detection, association and localization into a single unified metric for comparing trackers”. The part that matters for debugging rather than for leaderboards is the decomposition: it “decomposes into a family of sub-metrics which are able to evaluate each of five basic error types separately”. A single balanced number ranks trackers. The sub-metrics tell you which of the five things went wrong.

Identity labels require policies that ordinary detection datasets do not need, and those policies are not merely annotation conventions.

DukeMTMC is what happens when they are settled late. In 2014, eight cameras on Duke University's campus filmed roughly 2,000 students at 1080p and 60fps — in the words of the investigation that ended it, “over 14 hours of synchronized surveillance video ... with over 2 million frames of 2,000 students”. Adam Harvey and Jules LaPlace recorded the outcome on Exposing.ai: “In response to this report and an investigation by the Financial Times, Duke University has terminated the Duke MTMC dataset.”

Termination did not stop use. A 2021 study of 1000 papers found that “DukeMTMC and MS-Celeb-1M were retracted in April 2019, but continued to be used in 2020—largely, through derivatives”, and put a count on it: “In our 20% sample, DukeMTMC and its derivatives were used 73 times in papers published in 2020.”

For the European case, a regulator has already answered the retention question. GDPR Article 4(14) defines biometric data as “personal data resulting from specific technical processing ... which allow or confirm the unique identification of that natural person, such as facial images”, and Article 9(1) prohibits processing “biometric data for the purpose of uniquely identifying a natural person” absent an Article 9(2) exception. The European Data Protection Board applied that to this exact design in its Guidelines 3/2019 on video devices, adopted on 29 January 2020, at paragraph 82: “If a controller wishes to detect a data subject re-entering the area or entering another area (for example in order to project continued customized advertisement), the purpose would then be to uniquely identify a natural person, meaning that the operation would from the start fall under Article 9.” Paragraph 79 adds that intermediate templates must be “immediately and securely deleted” once a match or no-match result has been obtained. Reconnecting a track after a gap is not an edge case of the guidance. It is the example the guidance uses.

Two cameras may observe the same person at different times, scales, and viewpoints; association depends on clock synchronization, transition constraints, overlapping fields, and appearance shift.

The cost of skipping those constraints is on the public record. The FTC sued Rite Aid in December 2023 over the facial recognition it ran in its stores. The complaint alleges that “between December 2019 and July 2020, Rite Aid’s facial recognition technology generated over 5,000 match alerts in stores that were more than 100 miles from the store that created the relevant enrollment”. It also describes over 900 alerts for a single enrollment across more than 130 stores in one five-day period, and over 2,000 alerts implausibly close in time across distant locations. A person cannot be in two of those places at once. A system with no transition constraint and no clock reasoning has no way to notice. The complaint further alleges that alerts in plurality-Black and plurality-Asian areas were significantly more likely to carry low confidence scores. The settlement bars Rite Aid from using facial recognition for surveillance for five years.

A cross-camera ID should therefore carry uncertainty and provenance: which cameras, at which times, on what confidence, under which transition assumption. Evaluate handoffs separately from within-camera continuity.