Anomaly Thresholds, Alert Budgets, and Evaluation

Physiologic monitors watched 461 adults in five intensive care units for 31 days. Somebody counted what they produced. The Results section of the published abstract is the whole problem in four sentences: “A total of 2,558,760 unique alarms occurred in the 31-day study period: arrhythmia, 1,154,201; parameter, 612,927; technical, 791,632. There were 381,560 audible alarms for an audible alarm burden of 187/bed/day. 88.8% of the 12,671 annotated arrhythmia alarms were false positives.” Drew and colleagues published that in 2014.

Nothing in those numbers says how well any single detector ranks. An audible alarm burden of 187 per bed per day is an operating policy. 88.8% false among the arrhythmia alarms anyone actually adjudicated is a yield. The 2,558,760 total is a volume. Anomaly evaluation is not finished when a ranking metric looks strong, because what reaches a human is the score plus a threshold plus a review budget plus whatever adjudication ever arrives. Jacques and Williams, in an AHRQ Patient Safety Network perspective on the same phenomenon, give the general figure independently: “Research has shown that 80%–99% of ECG monitor alarms are false or clinically insignificant.” The queue is the thing being evaluated.

No single metric captures ranking, capacity, timing and consequence at once. Average precision and the precision–recall curve summarize ranking under class imbalance across thresholds; they depend on observed prevalence and encode no analyst capacity. Precision at K measures confirmed yield among the top K scores. It matches a fixed review budget, and it says nothing about the cases below K. Recall at capacity measures how many known incidents fall inside an alert budget. It needs an incident denominator, and selective labels bias it. Detection delay measures time from event onset to useful alert. It needs an onset definition, and it trades off against false alarms.

The intensive care count separates these cleanly. The 381,560 audible alarms are a volume statement. The 88.8% false positives among 12,671 annotated arrhythmia alarms are a yield statement, and only about the reviewed set. Neither figure tells you what fraction of real arrhythmias the monitors caught. The denominator of true events was never established. That is the recall-at-capacity question, and the study's numbers cannot answer it.

Each level can succeed while the next one fails. Score ordering asks whether known anomalous cases rank above reference cases. Threshold behavior asks how many alerts an operating point produces, and which cases cross it. Event capture asks whether distinct incidents are detected without a flood of duplicate alerts. Workflow outcome asks whether review uncovers useful cases quickly enough to justify the cost and the intervention.

The first level is the one usually assumed safe. It is not. Wu and Keogh went through the Yahoo, Numenta and NASA anomaly benchmarks — the label sets that score ordering is normally judged against — and concluded: “The majority of the individual exemplars in these datasets suffer from one or more of four flaws.” Liu and Paparrizos characterise those four as triviality, anomaly density, mislabeling and run-to-failure bias. Mislabeling is the one that reaches level one directly. An ordering metric computed against a mislabeled reference set measures the reference set. Wu and Keogh's response was to build a replacement, the UCR Time Series Anomaly Archive, which Liu and Paparrizos count at 250 curated time series drawn from a single dataset. Small and hand-curated, because the large uncurated alternative was worse.

Sparse labels and operational feedback can make naive metrics optimistic. One scoring convention can make them meaningless. Liu and Paparrizos, presenting a benchmark study at NeurIPS in 2024, state the result without hedging: “randomly generated predictions under point adjustment can even outperform SOTA methods, with its point-adjusted F score approaching one as the average length of the anomalies increases”.

Choosing a threshold is an institutional act with named owners and dated obligations. Two bodies have written the procedure down in enforceable form.

Step 1, define the alert budget, and step 5, calibrate the threshold policy, are what an accreditor codified after counting the damage. The Joint Commission found 98 alarm-related events in its Sentinel Event database between January 2009 and June 2012. 80 resulted in death, 13 in permanent loss of function, and five in unexpected additional care or an extended stay. Its Sentinel Event Alert of 8 April 2013 names the queue those events sat inside: “It is estimated that between 85 and 99 percent of alarm signals do not require clinical intervention”. The remedy was a numbered standard, not advice: “NPSG.06.01.01: Improve the safety of clinical alarm systems.” It took effect on 1 January 2014 in two phases. Leaders establish alarm system safety as a hospital priority as of 1 July 2014 (EP 1). The most important alarm signals are identified during 2014 (EP 2). Policies and staff education follow as of 1 January 2016 (EP 3, EP 4). Woo and Bacon describe the same sequence from outside: “Later that year, the Joint Commission released its 2014 National Patient Safety Goal on Alarm Management in two phases of implementation”.

Step 2, build delayed labels, needs a maturity window. In some domains the window is fixed by law rather than chosen. FinCEN's rule on when to file, 31 CFR 1020.320(b)(3), reads: “A bank is required to file a SAR no later than 30 calendar days after the date of initial detection by the bank of facts that may constitute a basis for filing a SAR. If no suspect was identified on the date of the detection of the incident requiring the filing, a bank may delay filing a SAR for an additional 30 calendar days to identify a suspect. In no case shall reporting be delayed more than 60 calendar days after the date of initial detection of a reportable transaction.” The Office of the Comptroller of the Currency states the same deadline for its own rule at 12 CFR 21.11: “A financial institution is required to file a suspicious activity report no later than 30 calendar days after the date of initial detection of facts that may constitute a basis for filing a suspicious activity report.” Thirty days, sixty at the outside, is a maturity window an evaluation can be built on. “Chargebacks arrive months later” is not.

The remaining steps follow from those two. Step 3 evaluates rankings with precision–recall behavior, precision at K, recall at capacity and event-level delay. Step 4 samples the unseen — some below-threshold cases, some random ones — to estimate how far selective labels have moved the numbers. Step 6 monitors alert volume, yield, duplicates, reviewer agreement, severity and drift. A figure like 187 audible alarms per bed per day only exists once someone counts.

The detector fired. Data-exfiltration malware was installed on Target's systems on 30 November 2013, then updated on 2–3 December. Each time, Target's FireEye intrusion-detection system triggered urgent alerts, under the generic name “malware.binary”. What happened next is recorded in the US Senate Commerce Committee's majority staff report of 2014 — an analysis the committee is careful to say is “based on media reports and expert analyses” rather than its own investigation: “Target's security team neither reacted to the alarms nor allowed the FireEye software to automatically delete the malware in question.” On why, the report offers the sentence this lesson exists for: “It is possible that Target staff could have viewed this alert as a false positive if the system was frequently alarming.”

The consequence was later argued in court. The federal district court in Minnesota held, on 2 December 2014: “Plaintiffs have plausibly alleged that Target's actions and inactions—disabling certain security features and failing to heed the warning signs as the hackers' attack began—caused foreseeable harm to Plaintiffs.”

No ranking metric would have flagged this failure, because the ranking was right. The hardest cases are the ones never reviewed and the alerts never acted on, not the ones with an incorrect recorded label. Report label coverage and sampling policy alongside metrics. Treat queue, adjudication and delayed outcomes as one system. The threshold policy you set this quarter is what generates the feedback you will evaluate the next version on.