Missingness, Outliers, Corruption, and Repair

Sample records and trace them through the pipeline before reaching for a statistical fix. The five steps run in order because each one produces the evidence the next one needs. A rate without a provenance trace cannot be classified. A classification without a downstream check cannot be defended when the number moves again six months later.

The artefact the process leaves behind matters as much as the fix. A cleaning rule a reviewer can trace back to a diagnosed cause is a rule that can be revisited. A rule sitting in the code with no record of what it was answering is one nobody will dare to remove.

When a record violates a critical invariant, send it to a quarantine path. That preserves the evidence and protects the published dataset, and the system can then classify, repair, replay or discard the record explicitly. The alternative is silent coercion. Its cost is not theoretical, and it has been counted twice.

Excel converts some gene names to dates by default. A 2016 screen of 35,175 supplementary Excel files, from 18 genomics journals published between 2005 and 2015, measured what that default does to the literature. Of the 3,597 papers carrying supplementary Excel gene lists, 704 contained gene-name errors — 19.6%, about one in five. Ziemann and colleagues stated the mechanism plainly in their abstract, in Genome Biology: “The spreadsheet software Microsoft Excel, when used with default settings, is known to convert gene names to dates and floating-point numbers.”

Five years later a larger rescan, with Ziemann again among the authors, covered articles published between 2014 and 2020. It found errors in 30.9% of those with supplementary Excel gene lists: 3,436 of 11,117. The rate went up, not down. A published warning did not stop a default from firing.

That is what a plausible-but-false value costs at scale, and the same failure is available in any pipeline. Parse an invalid date as epoch zero, or convert an unknown category to a common class, and you get a value that is well-formed, prints cleanly, and is wrong. Track quarantine rate, age, source, reason and resolution instead. A growing backlog is an operational failure even when the main pipeline stays green.

Decide in advance whether downstream datasets are provisional while quarantined records are unresolved. Late repairs may require a new snapshot, a label revision, or a note to consumers.

Two points equally far from the median can require opposite actions. The most dangerous extreme value is the one that is perfectly well-formed.

The Mars Climate Orbiter's carrier signal was last seen at approximately 09:04:52 UTC on 23 September 1999. The mishap investigation board's Phase I report of 10 November 1999 put the cause in one sentence: “The MCO MIB has determined that the root cause for the loss of the MCO spacecraft was the failure to use metric units in the coding of a ground software file, “Small Forces,” used in trajectory models.” The thruster performance data inside the SM_FORCES file was expressed in English rather than metric units. Every number in that file was numerically plausible. None of them tripped a check. The trajectory came in about 170 kilometres lower than planned at Mars orbit insertion. The board also recorded why the defect survived: “the root cause was not caught by the processes in-place in the MCO project.”

A wrong unit does not look extreme in the column that carries it. It surfaces as an outlier — or as nothing at all — several transformations downstream. That is why an extreme value is judged by tracing it to its source, not by its distance from the median.

A null can mean “not applicable,” “not measured,” “not yet available,” “redacted,” “source failed,” or “join did not match.” Replace every null with one value and those distinctions are gone. An extreme value can be a genuine rare case, a different unit, a duplicated aggregate, a broken sensor or a parsing error. Remove it automatically and you may be deleting the very events the model exists to recognize.

Cleaning is a diagnostic and semantic task before it is a statistical transformation. Trace the unusual value to the behavior of its source, then decide what each state means for the consumer.

Whatever the offline job repairs, the serving path has to repair the same way. A training-only imputation rule or clipping threshold is one more form of training–serving skew.

The operational response depends on why the value is absent. That question has a formal literature, and a date on it.

Ignoring the process that removed a value is only safe under stated conditions. Rubin named those conditions in 1976, in Biometrika, and his abstract says exactly what they buy: “These conditions are the weakest general conditions under which ignoring the process that causes missing data always leads to correct inferences.” Where they do not hold, the process that removed the value is part of the model whether or not anyone modelled it.

The same three states were restated in operational form in 2010, by a National Research Council panel report on missing data in clinical trials, which attributes them to “Rubin (1976) and Little and Rubin (2002)”. Data are MCAR when “missingness does not depend on values of the covariates, auxiliary and outcome variables”. They are MAR when missingness is independent of the missing responses conditional on the observed ones. They are MNAR when MAR fails.

The five states below are the engineering form of the same question. They are what a pipeline can actually observe about a null. They are also the evidence from which the MCAR/MAR/MNAR judgement has to be argued.

No strategy is universally best. Each one changes the population or the meaning of a feature.

The objection to filling a hole with a single number is not a matter of taste. It is written down, and two bodies published it independently in the same year. The EMA's guideline on missing data in confirmatory clinical trials, dated 2 July 2010 and in effect from 1 January 2011, states it in section 6.3.1: “A potential disadvantage of single imputation methods is that these methods risk biasing the standard error downwards by ignoring the uncertainty of imputed values.” The National Research Council panel of that same year made it Recommendation 10: single imputation methods such as LOCF and BOCF “should not be used as the primary approach to the treatment of missing data unless the assumptions that underlie them are scientifically justified.”

The cost is not in the imputed value. It is in what the column stops showing. A filled cell and an observed cell are indistinguishable downstream, so the uncertainty the fill introduced disappears from every interval computed after it. That is the argument for pairing any imputation with an indicator, and for treating the choice as a documented assumption rather than a default.