Multiplicity, Subgroups, and Sequential Monitoring

A product team ran an experiment and watched twenty outcomes across four user segments. One morning the dashboard refreshed and one metric, in one segment, had crossed p < 0.05. They wrote it up as a targeted win: the feature works, for these users. The roadmap turned toward that segment.

Nobody had decided in advance which of those comparisons the experiment was about. Twenty metrics in four segments is eighty chances for something to look striking. With eighty chances, a result of that size is roughly what you should expect when the feature does nothing at all. The team had not found an effect. They had found the largest of eighty noisy readings and given it a name.

The waste was not only the work that followed. The same screen carried segments where the metric had moved the wrong way, and nobody wrote those up. The report had been assembled to explain the win.

That team can always insist the effect was real, because nobody knows what was true underneath. So the rest of this lesson works with cases where the answer was known in advance: a trial split by astrological birth sign, a stroke therapy that was literally a dice roll, one data set handed to 29 teams at once, and simulations run on data built to contain no effect at all.

None of those comparisons came from anywhere exotic. Every one was an ordinary decision someone made while setting the experiment up: which outcomes to track, how many arms to run, where to cut the segments, which covariates and exclusions to fit, how often to look before the end. Each choice multiplies the chances that something compelling appears.

So the question error control has to answer is not how many p-values are on the page. It is how many decisions those numbers were allowed to influence. Correct over the family that matches the decisions, not over whatever list happened to get printed.

It is tempting to believe this only bites when the underlying effect is weak. It does not. ISIS-2 randomised 17,187 patients in 417 hospitals. Aspirin cut five-week vascular mortality from 1016 of 8600 patients (11.8%) to 804 of 8587 (9.4%), 2p<0.00001. The Lancet published that in 1988. The investigators then split their own trial by the patients' astrological birth sign, cuts chosen precisely because they could not possibly matter. Peter Sleight, one of them, reported what came out: “Even in a highly positive trial such as ISIS-2 [3], in which the overall statistical benefit for aspirin over placebo was extreme (P <0.00001), division into only 12 subgroups threw up two (Gemini and Libra) for which aspirin had a nonsignificantly adverse effect (9% ± 13%)”. Twelve slices of one of the most certain results in cardiology made the drug look harmful in two of them. The slicing variable was a birth sign.

Subgroups therefore need more than error control. A claim that the effect differs between two groups is a claim about a difference, and it has to be tested as one. That means an interaction or heterogeneity analysis, run with enough information to detect the difference, and confirmed by replication before anyone builds on it. Significant here and not significant there is two separate readings. It is not evidence that the two differ. Gemini and Libra are what that pattern looks like when the effect is identical everywhere.

The way out is to write down, before the data, what each analysis is for. One or a small set of prespecified decisions carry the primary claim. Key secondary claims support the interpretation: the mechanism, the accompanying benefit. Safety and guardrail measures are watched with deliberately asymmetric priorities, because you want a low bar for noticing harm. Exploratory analyses are allowed and useful, and what they produce is a hypothesis for someone to confirm later. Interim decisions — stop, declare futility, ramp, adapt — get their own rules, fixed before the first look.

There is no single correction to apply, because these are not one job. If a claim is going to be acted on and a false one is expensive, control the chance of making any false claim at all across the family. You will miss some real effects in exchange. If the point of the analysis is to generate candidates worth studying next, control instead the proportion of your announcements that turn out wrong. That tolerates some errors and finds more of what is there. And if the claims are ordered — a primary decision first, then secondary ones that only matter if the primary held — test them in that order and let each one gate the next.

The method follows from the decision you are about to make. It does not follow from the number of rows in the output.

Regulators reason the same way, and they have put it in writing. The FDA finalised its guidance on multiple endpoints in clinical trials on 21 October 2022, five years after the draft it replaced. It refuses to let the family be read off the endpoint list: “Both the issues and methods that apply to multiple endpoints also apply to other sources of multiplicity, including multiple doses, time points, or study population subgroups.” Doses, time points and subgroups are the dashboard's segments and windows under other names.

Two of these control errors, one tests a difference, one budgets error across time. Reports blur them constantly, and almost always in the direction that makes the claim sound stronger than it is. The first two are not folklore either. The split between them was made in a single paper, in 1995.

All of this is done before the data accumulate, because afterwards you cannot separate your own reasoning from what you have already seen.

Name the primary decision and the outcome tied to it. List the whole family of claims that decision belongs to, including the segment cuts and time windows you intend to look at. Mark which analyses are confirmatory and which are exploratory, and then hold to it. Fix the looks: when you will look, what the rule is at each look, and how much of the error budget that look spends. Connect every analysis on the list to a decision and a claim status, so that nothing arrives later without a label.

None of that is a counsel of perfection invented for this lesson. It has been a regulatory requirement, with published wording, since 5 February 1998. That is the date on ICH E9, the guideline that sets the default label for subgroups: “In most cases, however, subgroup or interaction analyses are exploratory and should be clearly identified as such; they should explore the uniformity of any treatment effects found overall.” Exploratory unless prespecified as confirmatory. The burden runs the opposite way from the one most reports assume.

E9 also closes the loophole in the word look. It defines the term so broadly that no glance escapes it: “An interim analysis is any analysis intended to compare treatment arms with respect to efficacy or safety at any time prior to formal completion of a trial.” Any analysis, at any time. Including the one taken because the dashboard happened to be open.

A plan like this takes an afternoon. It is the only thing that lets you point at a result afterwards and say it was not selected.

Counting the metrics on the screen gives you the start of the family, not the family. The team in the opening case counted twenty outcomes and four segments and got to eighty. Even that was generous to themselves, because it left out everything below.

Three psychologists priced these choices instead of deploring them. Simmons and colleagues ran 15,000 simulations per scenario on data containing no real effect, and measured what each ordinary analytic freedom does to a nominal 5% false-positive rate. Psychological Science published the result in 2011. Simply having two dependent variables to choose between takes that rate to 9.5%. All four of the freedoms they tested, used together, take it to 60.7%. Their own summary: “A researcher is more likely than not to falsely detect a significant effect by just using these four common researcher degrees of freedom.”

Lead with the decision you committed to and with how uncertain it still is. That is the one part of the report that was not selected, and it is what a reader is entitled to weigh first.

Everything after it carries its own status. For each secondary and subgroup result, say which family it belongs to, whether the difference was tested as a difference, and how much information there really was in that slice. Exploratory findings get exploratory verbs. They suggest something and they motivate a next test. They do not show or confirm.

How often does that happen? A 2012 review in the BMJ counted. Sun and colleagues went through the randomised controlled trials published in 2007 in the core clinical journals defined by the National Library of Medicine, a representative sample in the authors' description. Of 207 trials reporting subgroup analyses, 64 (31%) made a subgroup claim about the primary outcome. Only 6 of those 64 claims (9%) had a statistically significant test of interaction behind them. Only 26 (41%) had clearly prespecified the hypothesis. And 54 of the 64 (84%) met four or fewer of the review's 10 credibility criteria. Their conclusion is flat: “Authors often claim subgroup effects in their trial report. However, the credibility of subgroup effects, even when claims are strong, is usually low.” When a paper says the effect was concentrated in a subgroup, the base rate says nine times in ten the difference was never tested as a difference.

Safety runs on the opposite logic. A harm signal that has not crossed a multiplicity threshold is still a harm signal. The asymmetry is deliberate, because the cost of acting on a false alarm is not the cost of missing a real one.

The team in the opening case had none of this. Had the primary decision been written down first, the same eighty comparisons would still have existed. The win would simply have arrived as what it actually was: one exploratory result, worth a second experiment.