MechDetect: Detecting Data-Dependent Errors

AI summary

Overview

Research area: Machine learning and data quality — specifically the statistical mechanisms that generate errors in tabular data.

Technical level: Intermediate. The paper is readable with a basic grasp of binary classification, cross-validation, and hypothesis testing, but it builds directly on the missing-data statistics literature (MCAR/MAR/MNAR) and uses non-parametric tests and AUC-ROC throughout.

Scope: The paper proposes and empirically evaluates MechDetect, an algorithm that classifies the error-generating mechanism of a tabular column as MCAR, MAR, or MNAR by training three variants of a binary classifier on the data and error mask and comparing their cross-validated AUC-ROC distributions with Mann-Whitney-U tests.

What This Paper Is About

Data errors are common in tables, but most work focuses on detecting or cleaning errors rather than on understanding how they were generated. The paper argues that knowing the error mechanism matters because different mechanisms call for different cleaning procedures. MechDetect takes a tabular dataset plus a binary error mask and decides, for a given column, whether the errors are independent of the data (MCAR), depend on other observed columns (MAR), or depend on the values in the erroring column itself (MNAR).

Key Contributions

1. MechDetect algorithm: A simple, model-agnostic procedure that takes a tabular dataset X and error mask E and returns an error mechanism label plus two p-values, built on three supervised learning tasks (Complete, Shuffled, Excluded) and two sequential Mann-Whitney-U tests.
2. Extension beyond missing values: While grounded in the missing-data literature, the method applies to any error type as long as an error mask is available — for example masks derived from outlier detection, metadata constraints, or other detection approaches.
3. Coverage of all three mechanisms: Existing tests largely target MCAR only, or discriminate MNAR from MAR using a linear model; MechDetect handles MCAR, MAR, and MNAR and learns non-linear dependencies between data and errors via tree-based classifiers.
4. Large empirical evaluation: Experiments on 101 real-world tabular datasets drawn from four benchmark sources, with perturbations injected at five error rates and MechDetect applied 21,920 times per mechanism, plus a separate analysis of behaviour when no clean data is available.

Main Findings

• Strong overall accuracy on clean data: At an error rate of 0.5, MechDetect achieved a mean accuracy of 89.04% in identifying the injected error mechanism. Across all measured error rates and mechanisms, the mean accuracy was 89.14%.
• Per-mechanism performance differs: Median accuracy for MAR was 100%, for MCAR approximately 95% (with some dispersion), and for MNAR approximately 86% (with more dispersion).
• Robustness to error rate: Fig. 4 shows mean accuracy largely stable across error rates. Accuracy for detecting MCAR decreases as an error rate of 0.5 is approached, and this is balanced by an increase in accuracy for detecting MNAR. Detection of MAR appears largely unaffected by the error rate.
• AUC-ROC behaviour matches expectation: For MCAR, classifier AUC-ROC is concentrated around 0.5 (chance). In the Shuffled task for MAR and MNAR, AUC-ROC is also around 0.5, as expected since shuffling destroys the dependency. For MAR, the Complete and Excluded distributions are near identical with centres close to 1.0 — so excluding column j does not substantially hurt the classifier, a likely consequence of dependencies between column j and other columns in real-world data.
• MNAR is separable: In the MNAR case, omitting column j in the Excluded task substantially reduces AUC-ROC on average, which is what lets the second test distinguish MAR from MNAR.
• Performance degrades without clean data: When the tasks are trained on perturbed data X̃ instead of clean X, median accuracy for MCAR is approximately 0.0 with little dispersion, for MNAR approximately 1.0 with little dispersion, and for MAR approximately 0.65 with moderate dispersion. For MCAR this happens because the missingness symbol appears in the input, trivially encoding the target and violating MCAR's independence assumption, so the MCAR hypothesis is falsely rejected. The authors note this limitation applies specifically to missing values, not to other error types.

Methodology in Plain English

The idea is a controlled comparison. For a column of interest, the researchers train the same classifier three times:

• Complete: train on the full table to predict that column's error mask.
• Shuffled: train on the full table to predict a randomly permuted version of that error mask. This is a baseline — it breaks any real relationship between data and errors.
• Excluded: train on the table with the column of interest removed, still predicting that column's error mask.

If the Complete model beats the Shuffled model, errors must depend on the data (so they are MAR or MNAR rather than MCAR). If the Complete model also beats the Excluded model, the column's own values carry information about where its errors occur — evidence for MNAR; otherwise MAR.

The classifier is scikit-learn's HistGradientBoostingClassifier, a tree-based method inspired by LightGBM, chosen for its ability to capture non-linear relationships and its strong record on tabular data. Performance is measured with 10-fold cross-validation to produce ten AUC-ROC scores per task. The three sets of scores are compared pairwise using Mann-Whitney-U tests, and because two tests run in sequence, a Bonferroni correction is applied (each test thresholded at α/2).

Evaluation used 101 datasets from benchmarks by Grinsztajn et al., Fischer et al., the OpenML benchmarking suites, and data cleaning research, restricted to datasets with no pre-existing missing values, fewer than 100,000 rows, and 500 columns. The benchmark target column was excluded. Missing values were injected with the tab_err library at error rates of 0.1, 0.25, 0.5, 0.75, and 0.9. Accuracy was defined as (TP+TN)/(TP+TN+FP+FN), where a true positive means MechDetect detected the same error distribution that was present in the data.

Why This Matters

Research impact: MechDetect reframes error handling as a mechanism-inference problem rather than a detection problem, and unlike prior MCAR tests (Little's likelihood-ratio test, Li and Yu's nonparametric test, Spohn et al.'s random-forest/KL-divergence test) or Wang et al.'s linear score test for MNAR versus MAR, it addresses all three mechanisms and permits non-linear dependencies. It is, to the authors' knowledge, the first work to detect all three error mechanisms.

Real-world applications:

• Deciding which imputation or cleaning strategy to apply, since different mechanisms call for different remedies.
• Data pipeline and ML component development, where clean data is often available during development even if not at deployment.
• Auditing tabular datasets in domains with structured records — for instance sensor readings, clinical tables, or operational databases — where error masks can be obtained from outlier detection or metadata constraints.
• Calibrating error-handling logic during model development, where the authors argue prior work has shown data-driven error detection and cleaning requires clean data anyway.

Industry relevance: Data quality monitoring is described as a core challenge in modern information processing systems. The method is deliberately simple and reusable — any binary supervised algorithm can substitute for the gradient boosting classifier — and the authors release their implementation and all datasets in a reproducible code repository, which lowers the barrier to adoption in production data pipelines.

Future Directions

1. Relaxing the clean-data assumption: The authors identify increasing performance when no clean data is available as a key direction, since calibration on perturbed data currently breaks MCAR detection.
2. Errors other than missing values: Evaluating how MechDetect performs on perturbed data with non-missingness error types, where the "shortcut" problem of the missingness symbol appearing in the input does not apply.
3. Handling correlated columns: Strong correlation between column j and other columns can make the Excluded and Complete distributions identical, undermining the MAR/MNAR distinction — the paper flags this as an open limitation.
4. Extending the observable-values assumption: The method assumes values in column j are observed alongside the error mask; the authors note this may not hold in practice and may need addressing.

Target Audience

Data quality engineers and ML practitioners who need to choose cleaning or imputation strategies for tabular data; statistics and machine learning researchers working on missing-data mechanisms and error generation; and anyone building or auditing data pipelines where the mechanism behind missing or erroneous values affects downstream modelling decisions. Readers should be comfortable with classification metrics (AUC-ROC), cross-validation, and hypothesis testing.

Authors’ abstract

Data quality monitoring is a core challenge in modern information processing systems. While many approaches to detect data errors or shifts have been proposed, few studies investigate the mechanisms governing error generation. We argue that knowing how errors were generated can be key to tracing and fixing them. In this study, we build on existing work in the statistics literature on missing values and propose MechDetect, a simple algorithm to investigate error generation mechanisms. Given a tabular data set and a corresponding error mask, the algorithm estimates whether or not the errors depend on the data using machine learning models. Our work extends established approaches to detect mechanisms underlying missing values and can be readily applied to other error types, provided that an error mask is available. We demonstrate the effectiveness of MechDetect in experiments on established benchmark datasets.