Significance, Effect Size, Multiplicity, and the Winner’s Curse

A ranking change improves click-through by 0.01 percentage points across hundreds of millions of impressions. The p-value is tiny. The engineering cost and the user impact can still make the effect irrelevant. Statistical significance concerns compatibility with a null model. Product decisions require effect size, uncertainty, cost, and consequences.

That separation is not a house style. It is the formal position of the profession. On 7 March 2016 the American Statistical Association issued its first-ever position statement on a specific matter of statistical practice, and the matter it chose was statistical significance and p-values. The statement has six principles. The fifth is the one an evaluation report keeps violating: “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.” The statement does not say the p-value is worthless. It says it answers a different question from the one a launch decision asks.

How often is a published finding true at all? A 2005 paper put that question into a model, with the size of the search as one of its terms, alongside study power and bias. A finding is less likely to be true, Ioannidis writes, “when there is a greater number and lesser preselection of tested relationships”. The conclusion arrives without softening: “Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true”. The more things you try, the less the winner means.

A complete comparison answers more than “p < 0.05.” Direction, magnitude, precision and decision relevance are four separate questions. The ASA's fifth principle rules out settling the last two with an answer to the first: a p-value measures neither the size of an effect nor the importance of a result. Each of the four needs its own number and its own unit.

Names matter less than assumptions and units. Two models scored on the same cases produce paired outcomes. What a decision needs is the per-case difference between them, not two independent averages placed side by side. The procedure follows from three things: the resampling unit, the metric, and the stated quantity of interest. That quantity is written down before the comparison runs, as the protocol below shows. It is not something to read off the output afterwards.

Two drug regulators have published the arithmetic that a benchmark table hides. Their numbers transfer directly to a grid of models, metrics and slices.

Failure to reject zero does not show that two systems are equivalent. Equivalence or non-inferiority needs a predeclared margin, one that represents the smallest difference that matters. It also needs a confidence interval precise enough to fall entirely inside that margin.

This is not a thought experiment. It is how generic medicines are licensed. The margin is published in advance, and it is a ratio of 80.00% to 125.00% against the reference product. FDA's guidance on establishing bioequivalence recommends Schuirmann's two one-sided tests procedure and names the limit: “For a broad range of drugs, a BE limit of 80.00 percent to 125.00 percent for the ratio of the product averages has been adopted for use of an average BE criterion.” EMA applies the same acceptance interval. Its 2010 bioequivalence guideline requires the 90% confidence interval for the test/reference ratio of AUC and Cmax to be contained within 80.00-125.00%, with the bounds judged to two decimal places.

Notice the order of operations, and the two decimal places. The margin is fixed and published before the study. The interval must then fit inside it. An interval that spills past a bound fails, however comfortable the point estimate looks. An evaluation that reports a non-significant difference and calls two models interchangeable has done none of this.

Plan the comparison before inspecting the winner. Two of these five steps are not merely good practice. They are written into international guidelines, with numbers, adoption dates and named attributes.

1. Define the estimand. The word comes from a specific document. ICH's E9(R1) addendum, adopted in November 2019, requires an estimand to be built from named attributes: treatment condition, population, variable or endpoint, handling of intercurrent events, and population-level summary. It also fixes when they are chosen: “The clinical questions of interest and associated estimands should be specified at the initial stages of planning any clinical trial.” FDA adopted it as guidance for industry in May 2021. The evaluation analogue is the same list: which unit, which metric difference, which population, which horizon.

2. Set practical margins. A non-inferiority claim has a fixed shape, and EMA's guideline on the choice of the non-inferiority margin states it: “a non-inferiority trial aims to demonstrate that the test product is not worse than the comparator by more than a pre-specified, small amount. This amount is known as the non-inferiority margin, or delta”. That guideline was adopted in July 2005 and came into effect in January 2006. It refuses to let the margin be a statistical convenience: “The choice of delta must always be justified on both clinical and statistical grounds.” And it rules out switching approach retrospectively after a failed result. FDA tests the same way. Its November 2016 final guidance compares the upper bound of the two-sided confidence interval for control-minus-test against “the NI margin, M, which is specified in advance”. M itself comes in two sizes: M1, the entire effect of the active control assumed present in the trial, and the smaller M2, the largest clinically acceptable loss of that effect.

3. Choose the procedure. Use a paired bootstrap, a randomization test, or a justified model, with the resampling unit matching the unit named in step 1.

4. Address multiplicity. Control families, use hierarchical gates, or disclose exploratory scope, remembering that ten independent endpoints carry a Type I error rate of about 22% and five independent subgroup tests about 12%.

5. Confirm selection. Evaluate the chosen model on protected or prospective evidence, gathered after the search stopped.