Confounders, Mediators, Colliders, and Selection

Post-treatment conditioning should be rarest where randomisation is strongest. It is not. Three political scientists coded every article containing a survey, field, laboratory or lab-in-the-field experiment published in the American Political Science Review, the American Journal of Political Science and the Journal of Politics from 2012 to 2014. Seventy-five articles. Every one of them had a randomised treatment.

35 of the 75 — 46.7% — engaged in post-treatment conditioning. 16, or 21.3%, controlled for a post-treatment covariate in a statistical model. 11, or 14.7%, subsetted the data on post-treatment criteria. 8, or 10.7%, did both. Nothing had gone wrong upstream. The randomisation held, the data were clean, the models fit. Then a variable measured after the treatment was added to the right-hand side, or used to decide who stayed in the sample.

What that costs is not a small shading of the estimate in one direction. The authors' own summary of their 2018 paper puts it plainly: “Post-treatment bias can affect our estimates in any direction and can be of any size.” Nothing in any of those 35 outputs said which direction, or how much. A coefficient does not report what question it ended up answering.

The line of code is the same either way. Add the variable to the model, refit, read the coefficient. What differs is nothing the software can see: where that variable sits relative to the treatment and the outcome.

One stratification can carry an answer the whole way across. In US 1991 natality data the unstratified result is the one everyone expects: infants of smokers had higher risks of both low birth weight and infant mortality than infants of nonsmokers. Then the analysis was stratified on birth weight, a variable that lies downstream of smoking rather than beside it. The association flipped sign. Hernández-Díaz and colleagues reported it in the American Journal of Epidemiology in 2006, and their abstract states it in one line: “However, among LBW infants, infant mortality was lower for infants born to smokers (relative rate = 0.79).”

0.79. Same dataset, same exposure, same outcome, and smoking now looks mildly protective among low-birth-weight infants. Nothing was fabricated and no data were dirty. One stratification did it.

Add a common cause and the adjustment closes a path that was carrying association the treatment never produced. The estimate moves toward the effect you wanted. Add something the treatment itself caused and the adjustment closes a path that was carrying the real effect. The estimate moves away from it. Condition on a common effect and you can manufacture an association between two things that had none before you looked.

Three different outcomes, one syntax.

And the diagnostics do not move when the target does. Fit statistics measure how well a model describes the data it was handed. They cannot tell you which question it ended up answering, and they will not warn you that the sign is the wrong way round.

A confounder is a common cause. It sits upstream of both the treatment and the outcome, and it is why the two move together for reasons that have nothing to do with one producing the other. A mediator lies on the path between them: the treatment causes it, and it causes the outcome. A collider is the opposite shape, a variable that the treatment and another cause of the outcome both flow into. Selection is a collider you conditioned on without meaning to, by deciding who got into the sample.

Notice what those definitions are made of. Not the variable's name, not how strongly it correlates with anything. Only arrows, and which way they point.

Which is why the same column can be a confounder in one study and a mediator in another. Move the measurement earlier or later, ask about a different effect, and the role changes while the data sits unchanged. "Control variable" names a position in a regression command. It does not name anything causal, and a long list of control variables is not a defence.

Every nonparametric identification problem reduces to three structurally distinct biases: overcontrol bias, confounding bias and endogenous selection bias. That is the argument of Elwert and Winship's 2014 review in the Annual Review of Sociology, and it makes the taxonomy more than a set of labels laid over the analysis. It is the complete list of ways the arrows can defeat you. The third one also reaches further than the word "selection" suggests. From their abstract: “Endogenous selection bias can result from direct conditioning on the outcome variable, a post-outcome variable, a post-treatment variable, and even a pre-treatment variable.”

Read the end of that sentence again. Even a pre-treatment variable. "I only adjusted for things measured before treatment" is a better rule than most analysts follow. It is still not a proof of safety. The structure decides, and you have to supply it.

This one is a table with a row for each variable you are tempted to include, and columns that decide its fate.

Baseline common causes were measured before treatment and push on both the treatment and the outcome. Those are the ones you want. Treatment descendants are anything the treatment could have changed, measured after it — birth weight in the smoking analysis lives here. Causes of the outcome that the treatment never touched are worth keeping for precision rather than for bias. Selection variables govern who is in the dataset at all; they often appear nowhere as columns, which is what makes them the easiest to forget. Measurement variables record how each of the others was captured, and how badly.

The row with no column has a number on it. UK Biobank invited approximately 9.2 million people. They were aged 40-69 and lived within 25 miles of one of 22 assessment centres in England, Wales and Scotland. 5.5% took part in the baseline assessment. Everything the cohort can ever say about anything passes through that 5.5%, and the number appears in no analysis file. What it did to the cohort is measurable. At ages 70-74, all-cause mortality was 46.2% lower in men and 55.5% lower in women than in the general population of the same age. Total cancer incidence was 11.8% and 18.1% lower respectively. The verdict, from a 2017 abstract in the American Journal of Epidemiology: “UK Biobank is not representative of the sampling population; there is evidence of a “healthy volunteer” selection bias.”

Work across a row twice. Once for the arrows, and once for what you would have to assume for that classification to hold. That second pass is what turns a modelling habit into a decision you can hand to someone who disagrees with you. And the selection row, the one with no column behind it, is the row that most needs the second pass written down.

Domain labels are not roles. What settles the role is when the variable was measured and what produced it — the same two questions the map asks, applied to the cases where intuition tends to fail.

Selection on its own can hand you a result worth writing up. One simulation started from a reported protective association of smoking and assumed no causal effect at all. Griffith and colleagues published it in Nature Communications in 2020: “Calculating over this parameter space, 40% of all possible combinations lead to an artefactual 2-fold protective or risk association operating through this simple model of bias alone.” Under the null, from selection alone, four in ten parameter settings hand you a doubling or a halving to write up.

Classify the candidates before the analysis dataset is finalized, not after. By then the inclusions have already happened, and the table stops being a decision and becomes a justification for one.

That sequencing is not a stylistic preference, and in at least one field it is not optional. ICH E9(R1), the addendum on estimands and sensitivity analysis in clinical trials, was adopted on 20 November 2019. The EMA's CHMP adopted it on 30 January 2020, with effect from 30 July 2020. The FDA issued it as final guidance for industry in the Federal Register of 12 May 2021. It states the requirement in one sentence: “The targets of estimation are to be defined in advance of a clinical trial.” It also requires sponsors to say in advance how each intercurrent event will be handled. An intercurrent event is one that occurs after treatment initiation and affects the interpretation or the existence of the outcome measurement. That is the post-treatment variable problem, written into regulation, and answered before the data exist.

Write one row per variable. Record when it was measured relative to treatment, what you believe causes it, and what you believe it causes. The role follows from those two answers, and the decision follows from the role — adjust for it, leave it out, or handle it with a design built for the purpose. Where a colleague would classify it differently, put their answer in the row as well, and plan to fit that version too.

It is short work, and it does the one thing no model output can. It makes the assumptions visible before they are buried inside a coefficient, and it gives a reviewer something to argue with other than the number.

Two kinds of variable earn a place. Pre-treatment common causes go in because leaving them out leaves a biasing path open. Pre-treatment predictors of the outcome go in because they tighten the estimate, and they are safe only so long as they open nothing new.

Everything downstream of the treatment stays out of a total-effect model: mediators, and anything a mediator caused. So do the variables that decide who entered the sample, unless the design was built to handle them.

The instinct this is arguing with is named in the opening line of the crash course on good and bad controls: “Researchers exposed only to this literature may get the impression that adding “more controls” to a regression model is always better.” Pearl and two co-authors answer it with a catalogue rather than a slogan. Eighteen numbered causal diagrams, Models 1-18, and the same candidate control variable Z classified across all of them as a Good Control, a Neutral Control or a Bad Control. The bad ones have names and numbers: M-bias at Model 7, bias amplification at Model 10, overcontrol bias at Models 11-12, selection bias at Models 16-17, case-control bias at Model 18. The same Z changes category when the assumed causal structure changes, and again when the target shifts from the total effect to the direct effect. Their conclusion is the whole lesson in a clause — “in all cases, structural knowledge is indispensable for deciding whether a variable is a good or bad control”.

Where experts disagree about a role, do not settle it privately. Write down both graphs, fit both adjustment sets, and report the two answers together. If they agree, the dispute never mattered. If they disagree, the disagreement is the finding.

The 35 articles counted at the start of this lesson failed at none of the usual places. The treatment was randomised, the data were clean, the models fit, the code ran. They failed at a decision taken before any of that, when a variable that carried the effect was treated as a variable that contaminated it.