Uncertainty, Bootstrap, and Confidence Intervals

If another valid sample had been drawn from the same target population, the measured F1, RMSE, or NDCG would differ. That is not a methodological opinion. It is what a regulator tells manufacturers in writing.

Sensitivity and specificity are estimates, and the FDA says so to device makers in the guidance it issued on 13 March 2007: “These are only estimates for sensitivity and specificity because they are based on only a subset of subjects from the intended use population; if another subset of subjects were tested (or even the same subjects tested at a different time), then the estimates of sensitivity and specificity would probably be numerically different. Confidence intervals and significance levels quantify the statistical uncertainty in these estimates due to the subject/sample selection process.”

How much room that leaves is visible in the guidance's own worked example. From 220 subjects it reports an estimated sensitivity of 86.3% (44/51), with a 95% score interval of (74.3%, 93.2%). Specificity is 99.4% (168/169), with an interval of (96.7%, 99.9%). The 86.3% is the number a dashboard would print. The evidence behind it is equally consistent with 74.3%.

Sampling is only one source. Training seeds and label adjudication add more. Change nothing but the seed of the random number generator and the score still moves: Reimers and Gurevych showed that in 2017, on “the evaluation of 50.000 LSTM-networks for five sequence tagging tasks”. The seed alone produced “statistically significant (p < 10^-4) differences”. For two recent named-entity systems they measured “an absolute difference of one percentage point F1-score depending on the selected seed value”. That is the width of many published wins.

Uncertainty analysis asks which of these sources the evaluation actually represents, and which it leaves outside the interval.

Different sources need different designs, and choosing which ones to sample has a measurable price. In 2021 Bouthillier and sixteen co-authors modelled the whole benchmarking process rather than one training run. They report that “variance due to data sampling, parameter initialization and hyperparameter choice impact markedly” the result. Adding those sources to a deliberately biased estimator brought it closer to the ideal one “at a 51 times reduction in compute cost”. The proceedings version words the same figure as “a 51× reduction in compute cost”.

Sampling the pipeline beat sampling the test set. It was also cheaper by that factor.

An evaluation that resamples only the test rows is silent about the other four rows below.

Resampling should happen at the unit that could plausibly have been redrawn. The method for doing that is old and has one author. Bradley Efron introduced the bootstrap in The Annals of Statistics in 1979, showing the jackknife to be a linear approximation to it.

Two statistical societies later described what the method is for, on the same day and in nearly the same words. The Institute of Mathematical Statistics announced on 12 November 2018 that “The International Prize in Statistics has been awarded to Bradley Efron, professor of statistics and biomedical data science at Stanford University, in recognition of the "bootstrap," a method he developed in 1977 for assessing the uncertainty of scientific results that has had extraordinary impact across many scientific fields.” The Royal Statistical Society's notice that day states the $80,000 prize and words it this way: “Bradley was awarded the prize for the creation of the 'bootstrap', a method he developed in 1977 for assessing the uncertainty of scientific results.” Note what both say the object of the estimate is. The uncertainty of a result, not the quality of a model.

Step 1 is where dependence enters, and where most mistakes are made. When observations are not exchangeable rows, the unit changes shape. Künsch's block bootstrap, published in 1989, resamples “blocks of length l randomly with replacement among the blocks of observations” instead.

Step 5 is not decoration either. The block length l, the choice between percentile and BCa, and the replicate count are all inputs. A reader has to be told each one.

Using the same cases preserves shared difficulty. Philipp Koehn measured what that is worth for machine translation in 2004: “We call this method paired bootstrap resampling, since we compare a pair of systems.” He drew 1,000 resamples and reported the widths that resulted. On 300-sentence test sets the 95% BLEU intervals ran [26.5, 30.7] and [27.5, 33.0]. That is roughly four BLEU points across, wider than most claimed improvements.

Across 100 test sets, paired resampling returned a 95% verdict on 65 of them: “For 65 samples we make a 95% statistically significant conclusion that the Spanish system is better than the Danish”. The abstract puts the optimistic half of that: “Even for small test sizes of only 300 sentences, our methods may give us assurances that test result differences are real.”

The default outlived the paper. sacreBLEU, the reference scorer used for WMT, documents itself as “an efficient implementation of the paper Statistical Significance Tests for Machine Translation Evaluation”. It estimates 95% intervals by bootstrap resampling, and it still defaults to 1,000 resamples — recorded in the signature as bs:1000 and settable with --paired-bs-n. A twenty-year-old default is still the number under most reported MT gains.

A search evaluation contains 100,000 queries from 2,000 users.

The bootstrap's validity is a theorem with counter-examples attached, and the counter-examples were published with the theorem. Bickel and Freedman proved asymptotic validity across a wide class of statistics in The Annals of Statistics in 1981. Their abstract adds: “Some counter-examples are also given, to show that the approximation does not always succeed.”

The plain-language version came later. Horowitz's 2018 survey of bootstrap methods in econometrics puts it this way: “Although the bootstrap is often very accurate, it can be inaccurate and misleading if it is used incorrectly.” The same survey names the settings where it breaks: “There are, however, important settings in which a statistic is neither asymptotically linear nor normal and for which the nonparametric bootstrap is inconsistent.” Manski's maximum score estimator is one of them. The remedies Horowitz points to are subsampling and the m-out-of-n bootstrap. Not more replicates.

For dependent data the fix is the block bootstrap, which Künsch introduced in 1989. Horowitz describes what it does: “The block bootstrap is a method that makes even weaker assumptions about the DGP. This method consists of dividing the data into blocks and sampling the blocks randomly with replacement.” It buys validity at the price of a new tuning decision. Horowitz and colleagues showed that the optimal block length scales as n^r, with r = 1/3 for bias or variance, 1/4 for a one-sided distribution function and 1/5 for a symmetrical one. And Horowitz warns that “The results obtained with the block bootstrap can be sensitive to the choice of block length.” Block sampling also “does not exactly replicate the dependence structure of the original data-generation process”.

So report the method, the unit, the block length or cluster definition, the replicate count, the assumptions and the sensitivity. The block length is a reported assumption, not an implementation detail.

Regulators have already written the reporting checklist. ICH E9 puts it in one sentence: “Estimates of treatment effects should be accompanied by confidence intervals, whenever possible, and the way in which these will be calculated should be identified.” That is two obligations, not one. The interval, and the method, identified in advance. The same FDA guidance of 13 March 2007 makes the parallel recommendation for diagnostic accuracy, asking for “two-sided 95 percent confidence intervals” around sensitivity and specificity.

Applied to a model release, that means the point estimate, the interval, the target estimand and the analysis unit. It also means the sampling frame, the construction used, the replicate count, and whether model-training variation is inside the interval or outside it. For model comparisons, report the interval for the paired difference rather than two overlapping intervals.

If uncertainty crosses a release boundary, follow the predeclared policy: gather data, accept equivalence, run a limited trial, or reject the claim. Koehn's 65 of 100 is the shape of the honest outcome. A third of the comparisons did not resolve, and the report said so.