Baselines and Sanity Checks

A project may need several baselines. Each tests a different assumption, and the historical-heuristic row is the one most often waved through.

Google Flu Trends is the case that settles it. The system over-predicted influenza-like illness in 100 of the 108 weeks from August 2011 to September 2013. It overshot the actual 2011-2012 level by more than 50%. Lazer, King and two colleagues reported the comparison in Science in March 2014: “Even 3-week-old CDC data do a better job of projecting current flu prevalence than GFT”. The baseline here is not a toy. It is last-known-value applied to official counts that were already three weeks stale, and it won.

An independent reassessment had reached the same verdict earlier. Olson and colleagues wrote in PLOS Computational Biology, in October 2013: “We identified substantial flaws in the original and updated GFT models at all three geographic scales, including completely missing the first wave of the 2009 influenza A/H1N1 pandemic, and greatly overestimating the intensity of the A/H3N2 epidemic during the 2012/2013 season.” A system can be celebrated, widely cited, and still be losing to the number a clerk could have written down.

A good baseline is simple but not careless — and in forecasting the simple ones have a documented record of winning.

Popular machine-learning methods were run against eight classical statistical methods on 1,045 monthly series from the M3 Competition. Makridakis and two colleagues published the result in PLOS ONE in March 2018. Their abstract records that “we found that the former are dominated across both accuracy measures used and for all forecasting horizons examined”. The results discussion adds the sentence that matters here: “Even Naive 2 (a seasonal Random Walk (RW) benchmark) is more accurate than half of the ML methods.”

Five years later the same shape appeared in deep learning. Zeng and colleagues found that one-layer linear models beat every Transformer-based long-term forecaster they tested: “Experimental results on nine real-life datasets show that LTSF-Linear surprisingly outperforms existing sophisticated Transformer-based LTSF models in all cases, and often by a large margin.” Two literatures, two decades, one lesson. The naive predictor is the control, not the courtesy.

These tests move from pipeline integrity to evidence of real signal. What happens when a whole field skips them has been counted twice.

The AIX-COVNET collaboration searched bioRxiv, medRxiv, arXiv, EMBASE and MEDLINE for machine-learning models that diagnosed or prognosticated COVID-19 from chest radiographs and CT, between 1 January and 3 October 2020. They identified 2,212 studies; 415 survived initial screening, and 62 survived quality screening. Roberts and colleagues state the outcome in the abstract, in Nature Machine Intelligence in March 2021: “Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases.” Sixty-two papers cleared a quality screen. Zero cleared use.

The parallel living review reached the same conclusion independently. Wynants and colleagues appraised 31 models in 27 studies, drawn from 2,696 screened titles, and wrote in the BMJ in April 2020 that “All models were rated at high or unclear risk of bias” and that “we do not recommend any of these reported prediction models to be used in current practice”. Every one of those models had a metric, and most had a good one. The ladder below is what would have been missing from them.

Randomly permuting training labels breaks the relationship between inputs and targets. Many other properties of the dataset survive the permutation. A flexible model may still fit the shuffled training labels, but held-out performance should fall toward a no-signal level.

If it remains strong, investigate leakage, duplicate structure, evaluation code, or labels accidentally shared across partitions. The exact chance level depends on task and metric.

This is not folklore. It is a statistical test with a defined null distribution. Ojala and Garriga set out two permutation tests for classifier performance in the Journal of Machine Learning Research in 2010. Their abstract describes the first one — the wording, including “assess”, is as printed: “The first test assess whether the classifier has found a real class structure in the data; the corresponding null distribution is estimated by permuting the labels in the data.” scikit-learn implements exactly that test, as sklearn.model_selection.permutation_test_score. It reports the empirical p-value as (C + 1) / (n_permutations + 1), where C is the number of permutations whose score is greater than or equal to the true score. The formula has a consequence worth knowing before you run it. The best achievable p-value is 1/(n_permutations + 1), so the permutation count sets a floor no amount of signal can go below.

The shuffle became famous at ICLR in 2017, in a paper by Zhang, Bengio and three colleagues on why deep learning requires rethinking generalization. Their experiments “establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data”. That result “is qualitatively unaffected by explicit regularization, and occurs even if we replace the true images by completely unstructured random noise”. A training loss driven to zero therefore proves less than it looks. It is compatible with there being no relationship in the data at all. Only the held-out number carries the information you were after.

Ablation turns a complex system into a set of testable claims. The clearest demonstration of what a feature ablation is for comes from chest radiographs.

Convolutional networks for pneumonia detection were trained on 158,323 chest radiographs from three hospital systems: 112,120 from NIH, 42,396 from Mount Sinai and 3,807 from Indiana University. One line in the abstract, by Zech and colleagues in PLOS Medicine in November 2018, says what those networks had available: “The prevalence of pneumonia was high enough at MSH (34.2%) relative to NIH and IU (1.2% and 1.0%) that merely sorting by hospital system achieved an AUC of 0.861 (95% CI 0.855–0.866) on the joint MSH–NIH dataset.”

Sorting by hospital is the shortcut baseline, and it scored 0.861. The networks could execute it. They identified the source hospital for 99.95% of NIH images, 22,050 of 22,062, and 99.98% of Mount Sinai images, 8,386 of 8,388. The best internal model scored AUC 0.931 and fell to 0.815 when tested externally. Remove the site signal and you learn how much of the 0.931 was pneumonia. Leave it in and you cannot tell.

The pattern repeated under pandemic conditions. DeGrave, Janizek and Lee reported in Nature Machine Intelligence that “we demonstrate that recent deep learning systems to detect COVID-19 from chest radiographs rely on confounding factors rather than medical pathology, creating an alarming situation in which the systems appear accurate, but fail when tested in new hospitals”.

A one-point metric gain can be valuable when applied to millions of low-cost decisions, or irrelevant when it adds latency, review burden, and instability. The reverse also happens. A model that barely improves aggregate accuracy may substantially help a rare high-cost slice.

The Epic sepsis validation shows how much of the interpretation lives outside the metric. Alerts fired on 6,971 of 38,455 hospitalizations — 18% — while 1,709 sepsis patients, 67% of them, went unflagged. Neither figure is the AUC of 0.63, and no discussion of 0.63 alone would have surfaced either. Wong and colleagues named the cost in the same sentence as the counts: “thus creating a large burden of alert fatigue.”

Report absolute and relative changes, uncertainty, affected volumes, error trade-offs, and operational consequences. “Ten percent better” is incomplete unless the denominator and baseline are named.

Baselines belong in the evaluation suite. They should not disappear after the first experiment. Six of the seven reproducible neural recommenders lost to nearest-neighbour and graph-based heuristics only because somebody kept those heuristics available and tuned them properly, years after the original papers had moved on.