Model Selection and Error Analysis

Each rung answers a different uncertainty. Only then is the next justified.

This is a shortlist, not an automatic estimator map.

Cross-validation scores guide hyperparameter and family choices. If the same folds are used to report the selected model as though it were fixed in advance, the estimate reflects selection optimism.

Nested cross-validation or a preserved final test separates the inner selection process from outer performance estimation.

The clearest demonstration of what that separation buys came from microarray data. Published analyses were reporting a leave-one-out error of zero once a handful of genes had been picked out. Ambroise and McLachlan showed in 2002 that the zero was an artefact of doing the gene selection outside the cross-validation loop. Their abstract closes on the result: “Using two published data sets, we demonstrate that when correction is made for the selection bias, the cross-validated error is no longer zero for a subset of only a few genes.” Move the selection inside the loop, or use the .632+ bootstrap, and the zero goes away.

Five years later somebody counted how often the field was still making that mistake in print. Dupuy and Simon analysed 42 studies in detail and reported in 2007 that 21 of them — 50% — contained at least one of three basic flaws. One of the three was “in supervised prediction, a biased estimation of the prediction accuracy through an incorrect cross-validation procedure”.

How large is that optimism? It has been measured on data where the right answer is known in advance. Varma and Simon built “null” datasets of 40 samples and 6,000 features, in which the two classes do not differ at all and the true error rate is therefore 50%. They tuned Shrunken Centroids and SVMs on them, then reported the tuned score as if it were an estimate. From their 2006 abstract: “Even though there is no real difference between the two classes for the "null" datasets, the CV error estimate for the Shrunken Centroid with the optimal parameters was less than 30% on 18.5% of simulated training data-sets.” For the SVM the same thing happened on 38% of datasets. A nested procedure, by contrast, “provides an almost unbiased estimate of the true error”. A 50% problem reported as under 30% is selection optimism with a number on it.

How much nesting everyday practice needs is itself contested, and the disagreement is published rather than folkloric. Wainer and Cawley compared nested against flat cross-validation with 12 algorithms on 115 real binary datasets. Their 2021 conclusion: when there are few hyperparameters to tune, flat cross-validation usually selects an algorithm of practically similar quality.

Over-fitting does not stop at the model. It happens in the model-selection criterion too, and Cawley and Talbot measured how large it gets. In such a criterion, they argued in 2010, “a low variance is at least as important” as unbiasedness, because “a non-negligible variance introduces the potential for over-fitting in model selection as well as in training the model”. And “the degradation in performance due to over-fitting the model selection criterion can be surprisingly large”.

Then they put numbers on it. Thirteen benchmark data sets, each repeated across 100 realisations — 20 for the image and splice sets. “The p-value is below 0.05 for five of the thirteen benchmarks, indicating that in each case the median protocol is significantly biased in favour of the RBF KRR classifier”. A protocol that tilts that way, they conclude, “does not provide a reliable protocol for comparing the performance of machine learning algorithms”.

Their general warning is the one that matters here. These effects “are often of comparable magnitude to differences in performance between learning algorithms, and thus cannot be ignored in empirical evaluation”. The gap between two models can be smaller than the wobble in how you picked them.

For every candidate, record where it improves and where it creates new failure. Two published error tables show how much a single aggregate number can hide.

A widely implemented proprietary sepsis alert was measured against one health system's own records. Across 38,455 hospitalizations of 27,697 patients at Michigan Medicine, the Epic Sepsis Model reached a hospitalization-level AUC of 0.63 (95% CI 0.62–0.64). It fired on 6,971 of those hospitalizations, 18%. It did not identify 1,709 of the 2,552 patients who developed sepsis, 67%. Wong and colleagues put the conclusion in one sentence in JAMA Internal Medicine in 2021: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.” In 2024 a second team repeated the exercise on the same vendor model across two county emergency departments, over 145,885 encounters, and reported a sensitivity of 14.7%, a specificity of 95.3% and a positive predictive value of 7.6%.

The second table is about slices. Three commercial gender classifiers were audited against a new 1,270-person benchmark in 2018, and the errors were not spread evenly. Buolamwini and Gebru: “We evaluate 3 commercial gender classification systems using our dataset and show that darker-skinned females are the most misclassified group (with error rates of up to 34.7%). The maximum error rate for lighter-skinned males is 0.8%.” One aggregate accuracy figure covers both of those numbers.

The federal evaluation found the same shape of result at a different scale. NIST put 18.27 million images of 8.49 million people through 189 algorithms from 99 developers and published the result in December 2019. False positive rates across demographic groups, it reported, “often vary by factors of 10 to beyond 100 times”. The same report instructs that both false negative and false positive rates be reported for each demographic group at the operating threshold. Which is to say: the error table is the deliverable.

Keep the candidate set small enough to understand.

Fold results are correlated, datasets are finite, and metric estimates vary across sampling, labels, and thresholds. Report distributions, paired differences, and practical tolerances rather than ranking models to three decimals.

A simpler candidate within uncertainty can be preferable under cost, stability, or governance constraints.

How much distance between two classifiers is real? One comparison ran at a scale that settles the question: 179 classifiers from 17 families, on 121 data sets, published in 2014. The best random forest reached 94.1% of the maximum accuracy. The runner-up was close enough that the ranking does not survive a test. “However, the difference is not statistically significant with the second best, the SVM with Gaussian kernel implemented in C using LibSVM, which achieves 92.3% of the maximum accuracy.” An independent repeat in 2016 used 14 families on 115 real binary datasets and found random forest, GBM and RBF SVM not significantly different from one another. Its author, Wainer, proposed that a change of less than 0.0112 in error rate be treated as irrelevant. That is a published figure for how small a difference is too small to act on.

Which test should compare classifiers across many data sets? In 2006 the question had barely been asked. Demšar noted that comparing two algorithms on a single data set “has been scrutinized for quite some time already”, but that “the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but ignored”. He reviewed practice, examined the candidate tests “theoretically and empirically”, and recommended “a set of simple, yet safe and robust non-parametric tests”. Those are “the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparison of more classifiers over multiple data sets”. Such a comparison is usually reported with a critical-difference diagram: a rank axis with bars joining the candidates that cannot be told apart. The same paper introduced it.

Plot training and validation behavior as the training set grows. A persistent high-bias plateau suggests representation or model limitations. A narrowing gap can indicate value from more independent data.

Curves should preserve time, group, and pipeline boundaries. Otherwise they can repeat the same leakage at several sample sizes.

A model-selection memo should state the problem contract, candidate rationale, data versions, split, metrics, uncertainty, slice results, calibration, cost, known failures, and rollback conditions.

This makes the choice auditable when people, data, and software change.

That list is not an invention of this lesson. TRIPOD+AI is a 27-item reporting checklist for prediction model studies, with a further 13-item checklist for abstracts, published in the BMJ in April 2024. It came out of two Delphi rounds — 170 respondents from 22 countries, then 200 from 27 countries — and a consensus meeting attended by 28 participants, and it explicitly supersedes the TRIPOD checklist of 2015. Its scope is deliberately indifferent to which family you selected: “TRIPOD+AI provides harmonised guidance for reporting prediction model studies, irrespective of whether regression modelling or machine learning methods have been used”.

What happens when the memo is not written has also been measured. A 2025 review examined 140 articles reporting multivariable prognostic models in the three top-indexed general radiology journals. “Only 4% (n = 6) cited TRIPOD, with just 1 including the checklist. TRIPOD adherence was poor overall, with a median score of 57% (interquartile range, IQR 48% to 64%, range 30% to 87%).” In the statistical analysis domain — splits, metrics, uncertainty, the part this lesson is about — adherence was 5%.