Confidence Intervals, Hypothesis Tests, and Bootstrap

A product team checks a conversion experiment every hour and stops when p < 0.05. That habit has a price, and the price was worked out numerically long before online experimentation existed. In 1969 Armitage and colleagues published the calculation in the Journal of the Royal Statistical Society, with numerical results for binomial, normal and exponential series. Their abstract states the mechanism without hedging: “If significance tests at a fixed level are repeated at stages during the accumulation of data the probability of obtaining a significant result when the null hypothesis is true rises above the nominal significance level.”

Fifty years later the same result is a regulatory requirement rather than a curiosity.

An interval or a p-value is defined by a procedure — repeated sampling, or a posterior. Its meaning cannot be recovered from the endpoints alone. Confidence intervals, credible intervals, prediction intervals, hypothesis tests and bootstrap intervals answer different questions. Each depends on a model for sampling, dependence, or prior information.

How far the endpoints alone mislead has been measured, not merely suspected. In 2018 Lyu and colleagues put p-value and confidence-interval items to 362 Chinese psychology researchers and students. Ninety-nine per cent got at least one p-value item wrong. Ninety-three per cent got at least one confidence-interval item wrong. Those figures are not a verdict on the participants. They measure how little a pair of endpoints carries once it has been separated from the procedure that produced it.

A responsible analysis states the procedure, the assumptions, the target and the decision rule together.

The nonparametric bootstrap resamples observed units with replacement to approximate an estimator's sampling distribution. It works when the empirical distribution is a reasonable stand-in for the data-generating process. Dependent data need a different resampling unit, or a block, cluster, or time-series bootstrap. Resampling individual rows from repeated users breaks the dependence structure.

The method got its name in 1979, in a paper by Bradley Efron in The Annals of Statistics that took another look at the jackknife. It won him the 2018 International Prize in Statistics, an $80,000 award. The Royal Statistical Society's announcement records the method's spread as a count rather than an adjective: “It has been referenced in more than 200,000 peer-reviewed journal articles since 1980”.

The failure modes are published results too, not cautions written by nervous instructors. Bickel and Freedman proved bootstrap validity in 1981 for t-statistics, empirical and quantile processes and von Mises functionals. The same abstract supplies the other half: “Some counter-examples are also given, to show that the approximation does not always succeed.” A specific broken case arrived in 2010. For the Grenander estimator, a cube-root-n-rate estimator, Sen and colleagues showed that bootstrapping from the empirical distribution function has no weak limit in probability. The same paper gives the repair: the m-out-of-n bootstrap is consistent there. Nonsmooth statistics, boundary parameters, extreme tails and very small samples are the neighbourhood in which that happens.

The words sound similar, but the random objects differ, and the difference is not one that trained readers reliably hold. In 2014 Hoekstra and colleagues put six statements about a 95% confidence interval, ranging from 0.1 to 0.4, to 120 researchers and 442 psychology students. All six statements were false. Their abstract reports what happened: “Although all six statements were false, both researchers and students endorsed, on average, more than three statements, indicating a gross misunderstanding of CIs.” The researchers barely outperformed the students, who had had no training in statistical inference at all. Lyu and colleagues found the same pattern in a separate sample of 362 people in 2018.

The most-endorsed misreading attaches a probability to the parameter given this interval. That is the credible-interval claim, made without a prior, a likelihood, or a posterior. The confidence statement runs the other way: the coverage belongs to the procedure across repeated samples, and the target is fixed. The prediction interval answers a third question entirely, about an outcome not yet observed.

Five hundred and sixty-two people in one study and 362 in another endorsed readings of this kind. Naming which of the three objects an interval describes is part of reporting it, not pedantry about it.

Make inferential choices before inspecting the desired outcome. That sequencing is written into regulation, not left to disposition. The European Medicines Agency's draft guideline on multiplicity issues in clinical trials, released for consultation in December 2016, does one of the sums. Test five independent subgroups, each at a significance level of 2.5% with one-sided directional hypotheses, and the chance of at least one false positive rises to approximately 12%. The conclusion the guideline draws is procedural: “it is therefore necessary that the statistical procedures planned to deal with, or to avoid, multiplicity are fully detailed in the study protocol or in the statistical analysis plan”.

The steps below are that requirement in miniature. The plan is a document written before the data are inspected. Each step names a choice that cannot be made honestly afterwards: the target, the sampling unit, the procedure, the multiplicity budget, the magnitude to be reported, and the assumptions to be stressed.

A test compares an observed statistic with its behavior under a null model. The null model fixes a reference distribution or randomization scheme. The test statistic compresses the data toward the discrepancy of interest. The reference tail identifies outcomes at least as extreme. The p-value is the probability of that tail event under the null model, and nothing more than that — which is what principle 2 of the ASA's 2016 statement spells out.

The decision rule is the step where the other evidence enters: costs, the number of tests run, the monitoring schedule, and prior evidence. The 1969 result and the FDA's 2019 adaptive-designs guidance both act at that last step, not at the p-value itself. The tail probability is computed correctly at each look. It is the rule built on top of the looks that has to absorb them.