MacrOData: New Benchmarks of Thousands of Datasets for Tabular Outlier Detection

AI summary

Overview

Research area: Machine learning — benchmarking and evaluation for tabular outlier detection (anomaly detection).

Technical level: Intermediate (the individual building blocks are standard, but the paper moves quickly through diffusion-model analysis, copulas, and structural causal models).

Scope (1 sentence): The paper audits the field's dominant outlier-detection benchmark, ADBench, and introduces MacrOData — a suite of 2,446 tabular datasets across three new benchmarks, plus an extensive evaluation of 14 detectors and a public leaderboard.

What This Paper Is About

Outlier detection on tabular data underlies applications like fraud and fault detection, but the field's progress is measured almost entirely on ADBench, a benchmark of only 57 datasets. The authors argue this small scale limits diversity, statistical power, and fairness, and they show ADBench is dominated by global outliers that resemble Gaussian noise — which is why simple distance methods stay competitive on it.

Their goal is to replace this situation with a much larger, more diverse, carefully curated benchmark suite with standardized splits, held-out private labels for blinded testing, semantic metadata, and a wide hyperparameter-aware evaluation.

Key Contributions

1. A critical "demystification" of ADBench. The authors analyze the state-of-the-art diffusion-based detector DTE and the foundation model FoMo-0D, and connect them analytically and conceptually to classical KNN, concluding ADBench mainly contains global outliers that resemble Gaussian noise. They also document ADBench's small scale (57 datasets), near-duplicate datasets, non-tabular embeddings, low-dimensional bias (nearly 50% of datasets have fewer than 20 features), and trivial or near-random difficulty tiers.

2. Three new benchmarks totaling 2,446 datasets. OddBench (790 real-world datasets with semantic anomalies such as fraud, failure, and defect, curated from Tablib), OvRBench (856 real-world datasets with statistical outliers, built by One-vs-Rest repurposing of classification tasks), and SynBench (800 synthetic datasets). Dataset sizes span 1,000 to 1,012,180 samples and 3 to 1,024 features in OddBench; 1,000 to 2,215,023 samples and 2 to 10,935 features in OvRBench; and 1,000 to 6,000 samples with 2 to 100 features in SynBench.

3. Benchmark infrastructure. Standardized train/test splits (random 50% of inliers for training; the other 50% of inliers plus all outliers for testing, with inlier-only training), public/private partitions (100 private datasets each from OddBench and OvRBench; 690 and 756 public respectively), semantic metadata and tags, anonymization of private datasets (metadata stripped, feature and sample order shuffled), and representative 50-dataset subsets per benchmark chosen to match the first three moments (mean, standard deviation, skewness) of AUROC performance.

4. Large-scale evaluation with practical guidelines. Evaluation of 14 classical, deep, and foundation-model detectors across all 2,446 datasets and many hyperparameter configurations, reporting relative metrics (Average Rank, ELO, Win Rate, rescaled AUROC/AUPRC, Champion Delta) plus permutation-test p-values, runtime, and hyperparameter sensitivity.

Main Findings

• ADBench's idiosyncrasy explains its leaderboard. DTE's analytical form models outliers as Gaussian noise governed by a diffusion process, and its non-parametric variant DTE-NP produces a ranking of data points identical to ranking by average squared distance to the k nearest neighbors. FoMo-0D is pretrained on Gaussian-mixture-model data with outliers sampled from inflated-variance components — i.e., also Gaussian-like noise. This unifies why DTE-NP, KNN, and FoMo-0D are similarly competitive on ADBench.
• ADBench likely tests global, not subspace, outliers. Because DTE models noise diffusion in all dimensions and KNN measures distances across all features indiscriminately, its datasets most likely feature global outliers that stand out across many features.
• ADBench composition problems. It includes near-duplicates (e.g., Cardio and Cardiography, Satellite and Satimage-2, BreastW and Mammography), omits widely used datasets such as Arrhythmia, and contains embeddings of 6 image and 5 text datasets. Nearly 50% of its datasets have fewer than 20 features.
• Saturation and floor effects. "Trivial" datasets (BreastW, Thyroid, Fraud) let most algorithms exceed 90% ROC AUC, while "very hard" datasets (Yeast, Vertebral, Speech, IMDb) often yield near-random performance below 60% ROC AUC.
• Data integrity concerns. The Thyroid dataset has only five features versus the six in the standard UCI repository, and the Wine dataset's evaluation rests on only 10 anomalies.
• OddBench's construction funnel. Starting from 627 million Tablib tables, the preliminary filter cut the pool to 11 million datasets; the separability check required a Random Forest to reach at least 60% ROC-AUC; 2,907 candidates reached manual review, 1,300 of which were judged truly anomaly-detection related, and 790 survived duplicate removal.
• OvRBench's label rules. Datasets needed a single feature named "label" or "target" with cardinality between 2 and 10 and a majority category covering at least 50% of values; the majority class became the inlier class and the rest were subsampled as anomalies at a randomly drawn ratio in [0.05, 0.2].
• SynBench's generative families. The paper describes three priors — Gaussian mixture models (contextual subspace outliers), structural causal models (measurement and structural outliers), and copulas (probabilistic and dependence outliers) — which together produce five outlier types. Note: Section 3 refers to "five different data priors" while Section 3.3 describes "three distinct data priors with five types of outliers," an inconsistency in the text itself.
• Supervised methods outperform unsupervised ones on these benchmarks. The authors point to this as a signal of remaining headroom (reported in their Appendix, not in the provided text).
• Reported evaluation results are not in the provided content. The paper text supplied here is truncated at the start of Section 4.2 ("Result"), so no per-method performance numbers, rankings, runtimes, or hyperparameter-sensitivity results can be quoted.

Methodology in Plain English

The authors first re-examine the field's standard benchmark. Rather than only noting that it is small, they look inside two of its top performers — a diffusion-based detector and a pretrained foundation model — and show mathematically and conceptually that both effectively assume outliers look like Gaussian noise, which is the same assumption a classic nearest-neighbor method makes. This explains the benchmark's results and motivates a new suite.

To build the new benchmarks, they start from 627 million web-scraped tables (Tablib) and apply a cascade of filters: remove unusable columns and rows (columns with more than 25% null or infinite values are dropped), remove datasets with fewer than 1,000 samples or fewer than 3 numerical features, remove duplicates via a custom hash, and require that a supervised Random Forest can separate anomalies from inliers with at least 60% ROC-AUC. Two human review stages then confirm semantic relevance and catch duplicates the hash missed. OddBench uses one half of Tablib and keeps only datasets whose anomalous label value is a genuine real-world anomaly such as "fraud" or "failure"; OvRBench uses the other half plus established classification benchmark repositories (OpenML-CC18, AutoML, TabZilla, Talent-CLS, TabRepo, BCCO-CLS, TabArena) and converts classification tasks to detection tasks.

SynBench generates data from controlled statistical priors so that the true outliers and their type are known by construction. Each dataset gets a fixed train/test split, metadata generated with a large language model, optional private status with labels withheld, and a place in either the full set or a 50-dataset representative subset.

For evaluation, 14 detectors are run over all 2,446 datasets. Shallow models get a full hyperparameter grid; deep models get five randomly sampled configurations from a predefined search space because of time and memory limits; foundation models require no fitting or tuning. Because hundreds of datasets differ in difficulty, the authors avoid aggregating raw scores and instead use rank-based and relative metrics, plus permutation tests for pairwise comparisons and inter-quartile range across hyperparameter configurations to measure sensitivity.

Why This Matters

Impact on research: A benchmark of 57 datasets makes reported gains sensitive to which datasets were picked and vulnerable to dataset idiosyncrasies — the authors argue that this risk of mistaking benchmark quirks for methodological progress is exactly what a much larger, more diverse suite mitigates. The public/private split supplies a blinded, leaderboard-based way to track claims over time, and the annotated metadata plus individual per-dataset results open the door to meta-learning and pretraining work that small benchmarks cannot support.

Real-world applications (as named in the paper's motivating domains):

• Fraud detection in finance
• Healthcare monitoring and diagnosis support
• Environmental monitoring
• Fault and failure detection in industrial or software systems (reflected in the OddBench anomaly keywords such as "failure", "defect", "error")

Industry relevance: Practitioners choosing a detector for a new tabular problem need evidence about hyperparameter sensitivity and runtime, not just peak accuracy. The representative 50-dataset subsets and reported time trade-offs are designed to make that selection affordable, and the private leaderboard provides an ongoing, tamper-resistant way for vendors and researchers to compare methods.

Future Directions

• Meta-learning and unsupervised model selection. The authors explicitly call out meta-learning over large labeled repositories as a research direction that small benchmarks could not support, and Hyperspectral/OD hyperparameter optimization methods depend on exactly this kind of data.
• Foundation models for outlier detection. FoMo-0D and OutFormer are zero-shot detectors pretrained on labeled or synthetic data; larger, more diverse pretraining corpora raise the question of whether these models can move beyond Gaussian-like global outliers.
• Going beyond Gaussian-like and global anomalies. Since ADBench's apparent idiosyncrasy is a bias toward global, Gaussian-noise-like outliers, an open question is how detectors perform on the subspace, causal, and dependence outliers that SynBench and OddBench are designed to contain.
• Closing the gap between supervised and unsupervised detection. The authors report that supervised algorithms significantly outperform unsupervised ones on their benchmarks, framing this as evidence of large possible improvement for unsupervised methods.

Target Audience

Outlier-detection and anomaly-detection researchers who need a trustworthy evaluation suite; benchmark and dataset-curation researchers interested in the filtering and metadata pipeline; practitioners in fraud, healthcare, and monitoring who select detectors for tabular data; and developers of deep or foundation-model detectors who need blinded, leaderboard-backed comparisons.

Authors’ abstract

Quality benchmarks are essential for fairly and accurately tracking scientific progress and enabling practitioners to make informed methodological choices. Outlier detection (OD) on tabular data underpins numerous real-world applications, yet existing OD benchmarks remain limited. The prominent OD benchmark AdBench is the de facto standard in the literature, yet comprises only 57 datasets. In addition to other shortcomings discussed in this work, its small scale severely restricts diversity and statistical power. We introduce MacrOData, a large-scale benchmark suite for tabular OD comprising three carefully curated components: OddBench, with 790 datasets containing real-world semantic anomalies; OvrBench, with 856 datasets featuring real-world statistical outliers; and SynBench, with 800 synthetically generated datasets spanning diverse data priors and outlier archetypes. Owing to its scale and diversity, MacrOData enables comprehensive and statistically robust evaluation of tabular OD methods. Our benchmarks further satisfy several key desiderata: We provide standardized train/test splits for all datasets, public/private benchmark partitions with held-out test labels for the latter reserved toward an online leaderboard, and annotate our datasets with semantic metadata. We conduct extensive experiments across all benchmarks, evaluating a broad range of OD methods comprising classical, deep, and foundation models, over diverse hyperparameter configurations. We report detailed empirical findings, practical guidelines, as well as individual performances as references for future research. All benchmarks containing 2,446 datasets combined are open-sourced, along with a publicly accessible leaderboard hosted at https://huggingface.co/MacrOData-CMU.