Evaluation
Baselines, Comparators, and Counterfactual Thinking
Design credible comparators that reveal whether a model adds value beyond existing rules, workflows, and simple statistical alternatives.
By the end you can
- Choose baselines that match the real deployment decision
- Distinguish predictive comparison from causal claims about intervention impact
- Use negative controls and trivial predictors to detect evaluation defects
- Explain why a complex model can lose despite a higher standalone score
A model without a comparator has no victory
A 2019 systematic review screened 927 studies and kept 71 of them: every one a machine-learning model put head to head with a logistic regression on the same clinical prediction task. Then the reviewers did the thing the field rarely does. They split the comparisons by how well the comparison itself had been run.
“For 145 comparisons at low risk of bias, the difference in logit(AUC) between LR and ML was 0.00 (95% confidence interval, -0.18 to 0.18). For 137 comparisons at high risk of bias, logit(AUC) was 0.34 (0.20 to 0.47) higher for ML.” — Christodoulou and colleagues, Journal of Clinical Epidemiology, 2019.
The advantage did not live in the models. It lived in the protocols. Where the comparison was careful, the gap between the complicated method and the simple one was exactly zero, with a confidence interval symmetric around it. Where it was careless, machine learning won by 0.34. That is the whole subject of this lesson, in two numbers from one dataset.
Evaluation asks whether the candidate improves the actual choice set. The comparator may be a rule, a queue, a human process, a previous model, or a policy of doing nothing. How well that comparator was built determines most of what the reported margin means.
Across 145 comparisons at low risk of bias, the difference in logit(AUC) was 0.00.
Case
Eighteen recommender papers against a nearest-neighbor baseline
A reproduction study took the neural recommender literature at its word. It “considered 18 algorithms that were presented at top-level research conferences in the last years” and tried to run each one again. The arithmetic is in the abstract. “Only 7 of them could be reproduced with reasonable effort.” Of those seven, “6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques”. That was Ferrari Dacrema and colleagues, at RecSys in 2019.
What makes this a lesson about comparators rather than about neural networks is the next step. The simple methods were not missing from the original papers. They were there, and undertuned. That is how an unrelated team could reverse a published ranking without proposing anything new. Rendle and colleagues, at Google, re-ran the neural collaborative filtering experiments in 2020 and found that with proper hyperparameter selection a simple dot product substantially outperforms the learned similarities that had replaced it. The baseline had not lost the original comparison. It had not been run properly.
Figure
Visual
The comparator ladder
Different baselines diagnose different questions, and each rung has a documented case where it changed the verdict.
The operational status quo is the rung most often skipped. Epic's proprietary sepsis prediction model was validated by somebody other than its vendor: 38,455 hospitalisations of 27,697 patients at Michigan Medicine, 2,552 of them with sepsis (6.6%). The hospitalisation-level AUC came out at 0.63 (95% CI 0.62-0.64). The vendor had reported 0.76 to 0.83. Alerts fired on 6,971 of the 38,455 hospitalisations, 18% of them, and 1,709 of the sepsis patients — 67% — were missed. That was Wong and colleagues, in JAMA Internal Medicine in 2021. Ostermayer and colleagues validated the same model again in 2024, across 145,885 encounters at two county emergency departments, and measured 14.7% sensitivity and 7.6% positive predictive value within a 6-hour window.
The simple model baseline has been measured in bulk rather than argued about. In that review of 71 studies, the 145 comparisons judged at low risk of bias put the logit(AUC) difference between logistic regression and machine learning at 0.00, from -0.18 to 0.18. Only the 137 comparisons at high risk of bias favoured machine learning, by 0.34 (0.20 to 0.47). Iwagami and colleagues reproduced the pattern in an independent head-to-head on 118,074 validation discharges in 2024: a GBDT c-statistic of 0.764 against 0.755 for LR-LASSO. A logistic regression is not a courtesy floor placed under the candidate. On this evidence it is a live opponent.
- 01
Operational status quo
Current people, rules, thresholds, delays, and resource limits.
- 02
Trivial statistical baseline
Prevalence, mean, seasonal naive forecast, or random ranking.
- 03
Simple model baseline
Logistic regression, shallow tree, BM25, or linear calibration.
- 04
Strong incumbent
Best existing model under the same data and evaluation protocol.
- 05
Ablated candidate
The new system with one component removed to isolate contribution.
Example
Sanity checks that should be hard to beat
Baselines also expose broken protocols. The best of them have a known expected answer, so a deviation is unambiguous. The permutation control below is a randomization test taken from non-parametric statistics. Zhang and colleagues built their 2017 analysis on it, and Arpit and colleagues confirmed independently at ICML the same year that deep networks fit noise — and showed that their optimisation behaviour on noise differs measurably from their behaviour on real data.
- Majority predictor: If a classifier barely exceeds prevalence under severe imbalance, its apparent sophistication may be irrelevant.
- Shuffled labels: The expected reading is fixed in advance, and Zhang and colleagues wrote it down: “More precisely, when trained on a completely random labeling of the true data, neural networks achieve 0 training error. The test error, of course, is no better than random chance as there is no correlation between the training labels and the test labels.” The model can still fit the training set perfectly. The held-out score must fall to chance. Any above-chance held-out score after permutation is a defect signal — leakage or contamination, not a result. It also follows that training-set performance carries no information about generalisation.
- Feature removal: If deleting a supposedly crucial signal changes nothing, the narrative about model behavior needs revision.
- Time-naive forecast: Many forecasting systems fail to beat “same hour last week” once evaluated on future periods.
- Random ranking: Retrieval metrics should exceed a distribution-matched random ordering by a meaningful margin.
Comparison
Prediction comparison is not intervention impact
Two questions use different evidence, and one of them has been an adopted regulatory requirement for a quarter of a century. The international guideline on choosing a control group, ICH E10, reached Step 4 of the ICH process on 20 July 2000 and was adopted by the US FDA as guidance in May 2001. It states the counterfactual as a definition rather than a methodological preference: “The control group experience tells us what would have happened to patients if they had not received the test treatment or if they had received a different treatment known to be effective.” Control groups exist for one major purpose. They separate outcomes caused by the test treatment from outcomes caused by other factors.
E10 also names the failure mode this lesson keeps circling. A non-defined external comparator built out of general medical knowledge is “particularly treacherous”, because general impressions are so often inaccurate. A team that reports its model is better than what people usually do has chosen exactly that comparator, and has chosen it without measuring it.
The defined version costs more and settles more. In the MASAI trial, reported in the Lancet Oncology in 2023, 80,033 Swedish women were randomly assigned 1:1 between AI-supported screen reading and standard double reading. Randomisation is what turns the second column of this comparison from an argument into a measurement. Both arms drew from the same population under the same programme, so the difference between them belongs to the policy rather than to who showed up.
Predictive comparison
Which system produces better outputs on a defined reference set?
- Can use held-out labels
- Supports model selection
- Depends on metric and sample
- Does not isolate workflow impact
Intervention comparison
What changes because the organization deploys one policy rather than another?
- Needs a credible counterfactual
- Often uses randomization or causal design
- Includes human adaptation
- Measures downstream outcomes
Key idea
The human-plus-model comparator
A model can outperform an individual reviewer and still underperform a well-designed team workflow. The comparator that decides it is whatever the people are already doing. Subtract that, and Epic's sepsis model had very little left: “The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice.” — Wong and colleagues, JAMA Internal Medicine, 2021. The marginal contribution is 183 patients out of 2,552. It was bought with alerts on 18% of all 38,455 hospitalisations.
The comparison can run the other way too. In MASAI the comparator was itself a system of work: every mammogram read independently by two radiologists. The AI-supported arm produced 244 screen-detected cancers against 203, at an identical false-positive rate of 1.5% in both arms. Neither result is about a model in isolation, because neither comparator was a model in isolation.
Comparators should therefore include the complete decision process: who sees what, in which order, with which fallback and time budget.
183 patients out of 2,552 is what the model added to what the clinicians were already doing.
Analogy
A new route through a city
A route one kilometer shorter may be slower, less reliable, or impossible for a delivery truck. Distance is simply the easiest thing to measure about a journey, which is why it so often ends up standing in for the journey.
The comparator worth beating is the one that carries the constraints the truck actually faces.
A shorter route is not automatically a better trip.
Steps
Build the baseline suite
Use multiple comparators, because each catches a different failure. And give each comparator the effort you gave the candidate. That is where these studies say the result is actually decided.
Step 3 is the one with the largest documented failure rate. The recommender reproduction found the simple methods already sitting in the published papers, losing. A Google team then tuned them properly, and a simple dot product substantially outperformed the learned similarities that had replaced it. The clinical review prices the same mistake across a whole literature: 0.00 difference in logit(AUC) across the 145 comparisons at low risk of bias, 0.34 in favour of machine learning across the 137 at high risk. Identical preprocessing, identical data access and identical hyperparameter search for the baseline are not fairness gestures. They are the difference between those two numbers.
1. Document the status quo
Measure current outcomes, delays, workload, and failure modes before introducing the candidate.
2. Add a trivial predictor
Use prevalence, mean, recency, or random ordering as a floor.
3. Train a simple model
Keep preprocessing and data access identical to the candidate.
4. Compare the incumbent
Re-evaluate the existing production system on the same frozen evidence.
5. Run ablations
Remove features or components to estimate which additions actually matter.
Report incremental value and incremental cost
The decision memo should include gains over each baseline, the uncertainty around those gains, and the added costs: latency, labeling, maintenance, review burden. MASAI is the shape of an honest one. Detection: 244 screen-detected cancers against 203, or 6.1 versus 5.1 per 1,000 screened, a ratio of 1.2 (95% CI 1.0-1.5) at p=0.052 — a difference the trial could not separate from chance, with an identical false-positive rate of 1.5% in both arms. Workload: 46,345 screen readings against 83,231, a 44.3% reduction. The authors placed the claim where the evidence was: “AI-supported mammography screening resulted in a similar cancer detection rate compared with standard double reading, with a substantially lower screen-reading workload, indicating that the use of AI in mammography screening is safe.”
Outside a trial, the same accounting was repeated at scale. The German PRAIM implementation study followed 463,094 women and reported a detection rate of 6.7 versus 5.7 per 1,000, a relative increase of 17.6% (95% CI +5.7% to +30.8%), with recall rates of 37.4 versus 38.3 per 1,000. Eisemann and colleagues published it in Nature Medicine in January 2025.
A complex model that wins by a negligible margin may be the inferior system. Evaluation should make that tradeoff visible rather than hiding it behind a single score.
The detection difference sat at p=0.052; the 44.3% cut in reading workload did not.
Case
The M3 series where eight statistical methods dominated
Across the 1,045 monthly series of the M3 Competition, eight traditional statistical methods beat the popular machine-learning ones on both accuracy measures used and at every horizon examined. The comparison ran in PLOS ONE on 27 March 2018, and the abstract leaves no room for interpretation: “After comparing the post-sample accuracy of popular ML methods with that of eight traditional statistical ones, we found that the former are dominated across both accuracy measures used and for all forecasting horizons examined.” The machine-learning methods also cost far more to compute. That was Makridakis and colleagues.
The finding did not stay put, which is the more useful half of the story. Cerqueira and colleagues re-ran the comparison in 2022 and found that the ranking holds only under extremely low sample sizes, and reverses in direction as series length grows. So the 2018 result is a claim about the conditions of one comparison: 1,045 monthly series, at the lengths those series happen to have. It is not a claim about two families of methods. Change the comparator's conditions and the winner changes with them.
Position
A reported gain is a claim about the baseline it was measured against
A margin is a statement about two systems, and only one of the two is the one being offered. Eighteen neural top-n recommendation algorithms from top-level conferences went into a reproduction study at ACM RecSys '19. Only seven “could be reproduced with reasonable effort”, and six of those seven were often outperformed by nearest-neighbour or graph-based heuristics. The finding is not that anyone computed a number wrongly. The comparator was in the papers all along. A Google team showed at RecSys the following year what it was worth once someone selected its hyperparameters properly: a simple dot product substantially outperforming the learned similarities that had replaced it.
Forecasting shows the same shape on a different task, including the part where the shape moves. Across the 1,045 monthly series of the M3 Competition, popular machine-learning methods were “dominated across both accuracy measures used and for all forecasting horizons examined” by eight traditional statistical ones, while costing far more to compute. Then Cerqueira and colleagues found in 2022 that this ranking survives only at extremely low sample sizes, and reverses as the series get longer. Neither the 2018 result nor the 2022 one settles a field. Both are reports on a comparison, and they disagree because the comparison was set up differently.
The clinical literature has been counted, so the size of the effect is known rather than suspected. Across 71 studies drawn from 927 screened, “the difference in logit(AUC) between LR and ML was 0.00” in the 145 comparisons at low risk of bias, while the 137 comparisons at high risk of bias handed machine learning 0.34 (0.20 to 0.47). None of this says the simpler method usually wins. It says that a score reported with no named alternative, measured on the same data with the same care, is rarely evidence about the decision anyone is making. And it says that a large part of the published advantage of complicated methods is an artefact of how their opponents were run.
The comparator is the half of the result that rarely gets checked.
Key takeaways
- Every evaluation needs a comparator tied to the choice the organization actually faces: across 145 comparisons at low risk of bias, a review of 71 clinical studies measured the logit(AUC) difference between logistic regression and machine learning at 0.00.
- The operational status quo, a trivial predictor, a simple model, and an incumbent answer different questions — Epic's sepsis model scored an AUC of 0.63 at Michigan Medicine, against the 0.76-0.83 the vendor reported.
- Negative controls have known expected readings: trained on a completely random relabelling, neural networks reach 0 training error with test error at chance, so any above-chance held-out score after permutation is leakage or a broken protocol.
- Predictive superiority does not by itself establish that deployment caused better outcomes. ICH E10 reached Step 4 on 20 July 2000 and was adopted by the US FDA in May 2001; it defines the control group as what would have happened without the test treatment, and calls an undefined external comparator “particularly treacherous”.
- Human-plus-model workflows may outperform either component considered alone: measured against contemporary clinical practice, the sepsis model flagged 183 of 2,552 patients not already receiving timely antibiotics, at the price of alerts on 18% of hospitalisations.
- Incremental benefit should be reported together with incremental cost, risk, and complexity — MASAI's detection difference sat at p=0.052 while its screen-reading workload fell 44.3%, from 83,231 readings to 46,345.