Anomaly Detection: Scope, Reference, and Rarity

These four words overlap in practice. They are not synonyms. Rare is a claim about frequency in an observed sample. Erroneous is a claim about measurement, logging or processing. Risky is a claim about expected consequence. Novel is a claim about distance from a reference concept.

The fourth depends on something the first three do not: how clean the reference is. scikit-learn splits its detectors on exactly that line. Outlier detection is the case where “The training data contains outliers which are defined as observations that are far from the others”. Novelty detection is the case where “The training data is not polluted by outliers and we are interested in detecting whether a new observation is an outlier”.

The survey literature names the same split. The clean-reference case is what Chandola and colleagues called semisupervised anomaly detection in 2009 — techniques that assume “the training data has labeled instances only for the normal class”. Unsupervised techniques assume instead that normal instances are far more frequent than anomalies in the test data, and “suffer from high false alarm rate” when that assumption does not hold.

So the cleanliness of the reference is not a detail of setup. It decides which of two different problems you are solving. It also decides what a false alarm rate will look like when you are wrong about it.

A high-value international payment is extreme compared with the global population. For the account involved it matches a monthly supplier pattern and arrives at the expected time. Anomaly detection is conditional on a reference. Global rarity, peer-group rarity, temporal surprise and harmfulness are four different concepts.

The 2009 survey that fixed the field's vocabulary makes this structural rather than rhetorical. A point or collective anomaly, it says, becomes a contextual anomaly once the context is supplied. The label is a property of the frame you declare. It is not a property of the record.

A clear contract prevents rare-case ranking from becoming an undefined risk score. Define the anomaly unit, the reference, the context, the response and the evidence. Two of those five steps have public records of what happens when they are skipped.

Step 2, the reference. Between 8 November 2012 and 23 June 2016 the customer Fowler Oldfield deposited approximately £365m, around £264m of it in cash. The bank had “initially understood it would not handle cash from the Fowler Oldfield business”. That is a declared reference population containing no cash, against which £264m of cash could not register as unusual. The Financial Conduct Authority prosecuted National Westminster Bank Plc under the Money Laundering Regulations 2007, and on 13 December 2021 Mrs Justice Cockerill fined it £264,772,619.95 at Southwark Crown Court, reduced by one third from £397,156,944.14 for the guilty plea.

The same monitoring stack had a second defect, and the regulator's press release names it: “In addition, the bank’s automated transaction monitoring system incorrectly recognised some cash deposits as cheque deposits. As cheques carry a lower money laundering risk than cash, this was a significant gap in the bank’s monitoring of a large number of customers depositing cash, of which Fowler Oldfield was one.” The bank's own disclosure of the hearing says it “failed to adequately monitor one of our customers between 2012 and 2016 for the purpose of preventing money laundering”, and cites “weaknesses in some of the bank's automated systems”.

Step 5, the evidence. The rule that matches alerts to events decides what the evaluation reports. Point adjustment is one such rule: “if at least one moment in a contiguous anomaly segment is detected as an anomaly, the entire segment is then considered to be correctly predicted as anomaly”. Kim and colleagues tested what that rule does to a scoreboard in 2022, and wrote: “we theoretically and experimentally reveal that the PA protocol has a great possibility of overestimating the detection performance; even a random anomaly score can easily turn into a state-of-the-art TAD method”. On WADI, existing methods fail to exceed the F1PA of a randomly generated score, and an untrained model matches existing methods even when point adjustment is forbidden.

Two years later a different pair arrived at the same verdict independently. Liu and Paparrizos name biased evaluation measures as one of three core defects of the field: “by revealing issues and biases in evaluation measures, we identify the most reliable and accurate measure, namely, VUS-PR”. An alert budget with an undeclared matching rule measures the rule, not the detector.

An observation that is ordinary at one scale can be anomalous at another. Chandola and colleagues fixed that ladder in ACM Computing Surveys in 2009, and it has three rungs.

Point anomalies are single instances that differ from the reference distribution. Contextual anomalies require a condition to be stated first; the survey's worked example is a temperature of 35°F, normal at time t1 in winter and anomalous at t2 in summer. The authors put the principle this way: “A data instance might be a contextual anomaly in a given context, but an identical data instance (in terms of behavioral attributes) could be considered normal in a different context.” Collective anomalies are illustrated with an atrial premature contraction in an ECG. The low value by itself is not the anomaly. The sequence is.

The survey also states that a point or collective anomaly becomes a contextual one once context is supplied, so the three are frames rather than fixed labels. Twelve years later a different group of eight authors carried the same three forward in the Proceedings of the IEEE, with its own citations: “These include point anomalies, conditional or contextual anomalies [169], [171], [191]–[195], and group or collective anomalies [146], [193], [196]–[199].”

The last two rungs of the diagram are settings rather than scales. A novel regime is judged against a reference assumed clean. A contaminated outlier setting is fitted on data that already mixes normal and anomalous cases.

Operational meaning comes from the anomaly frame, not the algorithm name. That has been measured rather than asserted. ADBench ran 30 algorithms — 14 unsupervised, 7 semi-supervised, 9 supervised — over 57 datasets in 98,436 experiments in 2022. Its finding: “None of the unsupervised methods is statistically better than the others, as shown in the critical difference diagram of Fig. 4a (where most algorithms are horizontally connected without statistical significance).”

The ranking moves when the anomaly type moves. LOF is statistically the best on local anomalies, KNN on global anomalies, and LOF drops to poor on clustered anomalies, because “there is no algorithm performing well on all types of anomalies” and “Performance of unsupervised algorithms highly depends on the alignment of its assumptions and the underlying anomaly type.” TSB-AD, run in 2024 over 1,070 curated time series from 40 datasets with 40 algorithms, reaches the compatible verdict that “simpler architectures and statistical methods often yield better performance”. Declaring the frame settles more than picking the detector does.

The score does not tell an investigator what happened or why. It orders cases according to a model of unusualness under a reference. The operational system must add thresholds, context, explanation, review, feedback, and fallback.

The Dutch childcare benefits scandal is what the missing explanation layer costs. From 2013 the Dutch tax authorities scored every childcare-benefit application with a self-learning “risk classification model”, then routed the top of the queue to a person. Amnesty International's 2021 investigation describes the handover: “The system selected applications with the highest risk scores for manual review by a civil servant. The civil servant in turn had the power to label an application as fraudulent. The civil servant, however, was given no information as to why the system had given the application a high-risk score for inaccuracy.” Tens of thousands of parents were falsely accused.

On 7 December 2021 the Dutch Data Protection Authority fined the Minister of Finance €2.75 million. Among the violations it found was that the Tax Administration “used applicants' nationality (Dutch/not Dutch) as an indicator in a system that automatically designated certain applications as risky”. Its decision notice records that nationality had not been used in determining risk since October 2018, and that 1.4 million people were still registered as dual nationals in May 2018.

Without the response and evidence layer, an anomaly detector produces a queue with no defensible decision policy behind it. A reviewer handed a rank and no reason cannot disagree with it.