Kinds of learning
Anomaly, Novelty, and Density: Finding What Does Not Fit
Distinguish anomaly detection, novelty detection, density estimation, and rare-event classification, with attention to contamination and alert review.
By the end you can
- Differentiate outliers, anomalies, novelty, and rare labeled events
- Explain how density and distance assumptions define unusual behavior
- Recognize contamination and alert-fatigue risks
- Design evaluation around reviewed alerts and operational capacity
Key idea
Rare does not mean wrong, and common does not mean safe
A legitimate international payment may be rare for one customer. A familiar-looking transfer may be fraudulent. Rarity is not the point. Anomaly detection identifies deviation under a model; it does not identify harm, error, or intent.
The operational meaning comes from context, review, and consequences. Treating every statistical outlier as a threat creates noise. It can also conceal the cases that matter.
Anomaly is a relationship to an expected pattern, not a moral or causal verdict.
Case
Hospital alarms: 2,558,760 in 31 days, and 80 deaths in 98 events
Hospitals have measured both halves of that at once. The Joint Commission said in April 2013 that “it is estimated that between 85 and 99 percent of alarm signals do not require clinical intervention”. The same alert, Sentinel Event Alert Issue 50, reports 98 alarm-related events in its sentinel event database between January 2009 and June 2012. Eighty of them ended in death, 13 in permanent loss of function, and five in unexpected additional care or an extended stay.
That 85–99 percent is an estimate. One hospital then counted. Every physiologic monitor alarm in five adult intensive care units at UCSF Medical Center was logged for 31 days in March 2013 — 77 beds, 461 consecutive patients. Drew and colleagues published the tally in PLOS ONE in 2014: “A total of 2,558,760 unique alarms occurred in the 31-day study period: arrhythmia, 1,154,201; parameter, 612,927; technical, 791,632.” Of that total, 381,560 were audible. That is an average of 187 audible alarms per bed per day. Nurse scientists then annotated 12,671 of the arrhythmia alarms by hand, and 88.8% were false positives. An independent AHRQ evidence review, the “Alarm Fatigue” chapter of Making Healthcare Safer III, puts false alarms across studies at 72–99 percent.
Both records describe one system. The alarms were usually right about the signal and usually irrelevant to the patient. Staff who learned to discount 187 audible alarms per bed per day were responding rationally to the base rate, until the day they were not. The estimate and the census agree, and neither says the detectors were broken. An arrhythmia alarm that is 88.8% false at the bedside can still be a faithful report of what the waveform did.
Figure
Comparison
Four tasks that share similar language
These problems differ in what data are available and what the system is asked to detect. The distinction is not academic. It is visible in how a dataset is physically built.
MVTec AD, released at CVPR in 2019, is novelty detection in the strict sense. It holds 5,354 high-resolution images across 15 categories — 5 textures and 10 objects. The split is the whole point: 3,629 training and validation images against 1,725 test images. The dataset covers 73 distinct defect types, about five per category, with 1,888 pixel-accurate ground-truth regions marking where each defect lies. Of the training split the authors write: “The training set contains only images without defects.”
That one sentence is what “training data should represent normality” means operationally. The reference is curated clean. Every defect lives on the far side of the split. A model that learns the training distribution has learned normality by construction rather than by assumption. Outlier detection offers no such guarantee: the anomalies are already mixed into the data being modelled, which is what contamination means. Density estimation asks a different question again — not a ranking of unusualness but a distribution over observations. Rare-event classification gives up the clean reference deliberately, because it needs labelled positives to train on at all.
Outlier detection
Find unusual observations when the training set may already contain anomalies.
- Training data can be contaminated
- Goal is descriptive or cleaning-oriented
- No known normal-only reference
- Example: suspect sensor records
Novelty detection
Learn normal behavior from a mostly clean reference and flag new departures.
- Training data should represent normality
- Future cases are scored against that reference
- Useful for new fault patterns
- Sensitive to changing normal behavior
Density estimation
Model how observations are distributed rather than only ranking unusualness.
- Can assign relative likelihood or density
- Requires distributional assumptions
- High dimensions are challenging
- Useful for simulation or scoring
Rare-event classification
Predict a labeled but uncommon outcome.
- Uses known positive and negative labels
- Supervised metrics are available
- Imbalance affects training and evaluation
- Example: confirmed equipment failure
Visual
An anomaly detector is one stage of an alert system
The score becomes useful only after thresholds, context, investigation, and feedback are designed. Step 4, allocating review, is the stage most often assumed rather than built.
A regulator once wrote down what happens when it is not. On 1 August 2012, from about 8:01 a.m. ET, an internal Knight Capital system sent 97 automated “BNET reject” e-mails to a group of Knight personnel. Each one named the error “Power Peg disabled”. All of this happened before the 9:30 a.m. open. The signal was correct, it was specific, it named the failing component, and it arrived in real time. The SEC’s order against Knight, issued on 16 October 2013, records what the design did with it: “Knight did not design these types of messages to be system alerts, and Knight personnel generally did not review them when they were received.”
After the open, SMARS sent millions of child orders. In roughly 45 minutes Knight accumulated 4 million executions in 154 stocks for over 397 million shares. The loss was over $460 million, the civil penalty $12,000,000. Ninety-seven correct warnings had already arrived. Nothing in the pipeline routed them to anyone whose job was to read them. No improvement to the detector at step 2 would have changed the outcome, because the detector was not the stage that failed.
1. Build a reference
Define normal operation by entity, time, environment, and data quality.
2. Produce an unusualness score
Measure deviation using distance, reconstruction, density, or another criterion.
3. Apply context
Add maintenance windows, customer history, policy rules, or known exceptions.
4. Allocate review
Set thresholds and queues that fit analyst capacity and risk.
5. Record outcomes
Capture confirmed issues, benign novelties, unresolved alerts, and missed incidents.
Case
Target, December 2013: the alerts fired and nobody acted on them
Target’s 2013 breach is the case where the detector worked and the stage after it did not. The attackers installed exfiltration malware on 30 November and updated it twice more in early December. Target’s FireEye system “triggered urgent alerts with each installation of the data exfiltration malware”. Nothing followed: “Target’s security team neither reacted to the alarms nor allowed the FireEye software to automatically delete the malware in question.”
Those findings come from the U.S. Senate Commerce Committee’s majority staff report of 26 March 2014, which also offers the explanation this lesson is about. The alert arrived under the generic name “malware.binary”, and “it is possible that Target staff could have viewed this alert as a false positive if the system was frequently alarming.” Some 40 million credit and debit card accounts left the network afterwards. Personal information for up to 70 million customers was taken as well.
Example
Normal behavior is conditional
A single global baseline can mark the wrong cases as unusual.
- Industrial equipment: vibration differs across machine models and operating loads.
- Cybersecurity: network traffic changes by role, location, weekday, and maintenance period.
- Healthcare: laboratory ranges depend on population, instrument, and clinical context.
- Finance: transaction patterns differ by customer history and merchant category.
- Logistics: route times shift with weather, holidays, and temporary closures.
- Data pipelines: schema changes can appear anomalous while reflecting an approved release.
Distance stops discriminating from about 10–15 dimensions
When many features are noisy or irrelevant, points can appear similarly distant and local neighborhoods become unstable. A detector may then respond to scale, missingness, or measurement artifacts rather than to meaningful novelty.
That has a proof and a threshold. Beyer and three co-authors opened a 1999 paper with the result: “We show that under a broad set of conditions (much broader than independent and identically distributed dimensions), as dimensionality increases, the distance to the nearest data point approaches the distance to the farthest data point.” Their experiments show the effect arriving with as few as 10–15 dimensions. This is not a pathology of thousand-dimensional embeddings. A modest feature table can reach it.
The effect has a name and a disputed consequence. Distance concentration is “the tendency of distances between all pairs of points in high-dimensional data to become almost equal”, in the definition Radovanović and colleagues gave in the Journal of Machine Learning Research in 2010. They also record what it is usually taken to mean here: “a prevailing view of the effect of high dimensionality on distance-based outlier detection is that, due to distance concentration, every point seems to be an equally good outlier, thus hindering outlier-detection algorithms”. They attribute that view to Aggarwal and Yu in 2001, and then dispute it. The real damage, they argue, runs the other way: high dimensionality induces “antihubs”, points that show up as artificial outliers. Aggarwal and two co-authors had put the general warning bluntly in 2001: “in high dimensional space, the concept of proximity, distance or nearest neighbor may not even be qualitatively meaningful”.
Representation learning and feature selection can help. They also add assumptions. Always inspect which variables drive alerts, and whether the same mechanism appears across slices.
Steps
Evaluate an anomaly system under a review budget: 8 charts per confirmed case
Ordinary accuracy is rarely informative when confirmed anomalies are scarce and alerts are capacity-limited.
The Epic Sepsis Model shows what those numbers look like in a deployed system. It is a proprietary early-warning tool running across hundreds of US hospitals. Wong and colleagues validated it externally at Michigan Medicine and published the result in JAMA Internal Medicine in 2021: 38,455 hospitalizations of 27,697 patients, between 6 December 2018 and 20 October 2019. At the vendor’s recommended threshold of 6 it reached an AUC of 0.63 (95% CI, 0.62–0.64), with sensitivity 33%, specificity 83% and a positive predictive value of 12%. It fired on 18% of hospitalizations — 6,971 of 38,455. Clinicians had to evaluate 8 alerted patients to find one who developed sepsis. The authors’ own summary is one sentence: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.”
An independent validation in two county emergency departments cites that study and reports sensitivity 14.7%, specificity 95.3% and PPV 7.6% in its own 6-hour window. Ostermayer and colleagues published it in JAMIA Open in 2024. Different site, different window, same shape of result.
Read the review budget off figures like these rather than off a headline metric. An alert on nearly one hospitalization in five is a staffing decision before it is a modelling decision. And 8 reviews per confirmed case is a number a unit either has capacity for or does not. Evaluation that stops at AUC 0.63 never reaches that question.
1. Define review capacity
State how many alerts can be investigated per hour, day, or site.
2. Evaluate ranked alerts
Measure confirmed findings and serious misses at realistic queue sizes.
3. Separate novelty types
Distinguish harmful incidents, benign changes, data errors, and policy exceptions.
4. Test normal shifts
Simulate seasonality, new devices, new users, and approved process changes.
5. Audit alert burden
Measure repeated alerts, investigation time, and unequal burden across entities.
6. Close the feedback loop
Use reviewed outcomes carefully without learning only from cases the old detector selected.
Case
Four defects in the benchmarks — and a random score that outranks every model on them
The public benchmarks are not a shortcut around that work. Wu and Keogh went through the time-series anomaly datasets most published methods report on: Yahoo, Numenta, NASA, OMNI. The majority of their individual problems, they argued, carry at least one of four defects: “triviality, unrealistic anomaly density, mislabeled ground truth and run-to-failure bias”. Their conclusion, in IEEE Transactions on Knowledge and Data Engineering, was that most published comparisons of anomaly detection algorithms “may be unreliable”, and that much of the field’s apparent progress “may be illusionary”. They built the UCR Time Series Anomaly Archive as a replacement. Later work that adopted that archive states the consequence plainly: gains scored on the older benchmarks “may not be indicative of actual improvements generated by deep learning models”.
There is a sharper version of the problem, and it sits in the scoring convention rather than in the data. Most published time-series detectors report their results under a protocol called “point adjustment”. Kim and four co-authors examined it at AAAI in 2022. Their abstract: “In this paper, we theoretically and experimentally reveal that the PA protocol has a great possibility of overestimating the detection performance; even a random anomaly score can easily turn into a state-of-the-art TAD method.”
The table behind that sentence is the argument. A uniform random anomaly score reaches an adjusted F1 of 0.969 on SWaT, 0.965 on WADI, 0.931 on MSL and 0.961 on SMAP. That is higher than every deep model they compared on those four benchmarks, including GDN at 0.935 on SWaT and OmniAnomaly at 0.866. The same random score, without the adjustment, scores 0.216, 0.109, 0.190 and 0.227. The honest caveat is in the same table. On the fifth dataset, SMD, the random score’s adjusted F1 of 0.804 does fall below the trained models (OmniAnomaly 0.944, USAD 0.938). Noise does not win everywhere. It wins often enough that an adjusted-F1 figure tells you nothing until you know which protocol produced it.
Anomaly detection does not replace domain diagnosis
A detector can prioritize unfamiliar patterns. Root cause usually requires process knowledge, additional measurements, and investigation. In safety-critical settings, known failure modes may deserve explicit supervised detectors or rules alongside novelty detection.
Combine statistical surprise with consequence, context, and escalation. The most unusual event is not always the most important one. And as 97 e-mails naming “Power Peg disabled” and an alert labelled “malware.binary” both showed, the most important event is not always the one anybody was asked to read.
Key takeaways
- Outlier detection, novelty detection, density estimation, and rare-event classification are distinct tasks, and MVTec AD makes the difference physical: 3,629 defect-free training images against 1,725 test images.
- Normal behavior is conditional on entity, time, environment, instrumentation, and policy.
- An anomaly score measures deviation under a chosen model rather than confirmed harm — 88.8% of the 12,671 annotated arrhythmia alarms at UCSF were false positives, and the monitors were reporting the waveform correctly.
- High-dimensional geometry and contaminated references can produce misleading alerts; the nearest-neighbour distance starts approaching the farthest-neighbour distance from as few as 10–15 dimensions.
- Evaluation should match review capacity: the Epic Sepsis Model fired on 18% of hospitalizations, 6,971 of 38,455, at 8 alerted patients per confirmed sepsis case.
- Known hazards, domain rules, and investigation workflows often complement anomaly detection — Knight's 97 correct “Power Peg disabled” e-mails went to a queue no stage of the design allocated review to.