Causal DAGs and Assumption Maps

UK Biobank has 486,967 participants. When the cohort was turned on the question of who catches COVID-19 and who gets severely ill, only 1,410 of them had COVID-19 testing data at the time of the analysis. Everyone else had no test result to model. So the analysis ran on the tested. Being tested looked like a matter of data availability rather than a matter of biology.

It was not data availability. Someone checked 2,556 participant characteristics, one by one, against whether a person had been tested at all. 811 of them — 32 per cent — came back with a false discovery rate below 0.05. Griffith and colleagues ran that check in 2020. Whether a person was inside the analysed sample was itself caused by hundreds of the things anyone would want to study.

That is the structure, and it is worth being slow about. Being tested is a variable that many other variables point into: symptoms, health status, occupation, willingness to present, age. Restricting the analysis to tested people means conditioning on that variable. Conditioning on it changes what the remaining participants look like. Among the tested, someone with no routine reason to be tested was more likely to be there because they were unwell. So the exposures and the outcome arrived in the data linked to each other. Nothing had linked them in the cohort before anyone was selected.

The estimate was not merely noisy. It was produced by the step that was supposed to make the data usable. The paper in Nature Communications opens by saying so: “Collider bias can induce associations between two or more variables which affect the likelihood of an individual being sampled, distorting associations between these variables in the sample.”

What separates a variable that is safe to adjust for from one that wrecks the study is not how useful it looks in a model. It is where it sits in the causal story. There are only a few questions to ask of it. Baseline severity: does it cause both the treatment decision and the outcome? Adherence: is it itself affected by the treatment, and by prognosis? Selection into the sample: do exposure and outcome risk both influence who gets tested, recruited or retained at all? The measurement device: does one shared artifact make the observed variables move together for reasons that have nothing to do with the people?

A causal DAG holds the variables and the direct relations someone is willing to assume between them, with no cycles allowed. What it buys you is the ability to tell a causal path from a backdoor path, and to see in advance whether conditioning on a variable will block an association or manufacture one.

No software builds this graph for you out of a correlation matrix. Correlation is symmetric and the graph is not, so direction has to come from elsewhere: from what practitioners know about the domain, from what happened before what, from how the measurements were actually taken, from how people ended up in the sample, and from the rival explanation a colleague would offer if you asked them to attack the study.

This matters more here than it does in prediction work, because one measured column can demand opposite handling depending only on its position. A confounder is a common cause of treatment and outcome, and leaving it alone leaves a backdoor path open. A mediator carries part of the effect you are trying to measure, and adjusting for it quietly deletes the part it carries. A collider is the common effect of two variables, and adjusting for it opens the path that being tested for COVID-19 opened.

One column has been put through those roles in public. Among US infants born in 1991, infants of smokers had higher risks of both low birth weight and infant mortality. That is the direction anyone would expect. Now stratify on birth weight. Within the low-birth-weight stratum, mortality was lower for infants of smokers: relative rate = 0.79. The sign reverses. Nothing about smoking became protective. Birth weight is affected by the exposure and shares common causes with the outcome, so stratifying on it is conditioning on a collider. The reversal is the artifact of the stratification rather than a finding inside it. Hernández-Díaz and colleagues drew that out in 2006, and concluded: “Under realistic causal diagrams, adjustment for birth weight is unwarranted when the analytical goal is to estimate overall effects of prenatal variables on infant mortality.”

Same column, same summary statistics, opposite instruction.

Name the variables first, including the ones that decide who is in the sample at all, because a variable nobody wrote down cannot be argued about. Then draw the arrows you are prepared to defend, and only those. Then enumerate the paths joining treatment to outcome, forward routes and backward ones alike. Then apply d-separation to each path, to see which are already closed, which are open, and which a candidate adjustment set would close or open.

That much most teams are assumed to manage. Somebody counted instead of assuming. A 2021 review by Tennant and colleagues went through every health research article published between 1999 and 2017 that mentioned directed acyclic graphs or cited DAGitty. 234 of them reported using one. A fifth — 48 papers, 21 per cent — reported the target they were trying to estimate. About half, 115, reported the adjustment set their own graph implied. Only 144, 62 per cent, made any DAG available at all. The graphs themselves averaged 12 nodes (IQR 9-16) and 29 arcs (IQR 19-42), with median saturation 46 per cent. Only 53 of them — 37 per cent — included a single unobserved variable. Most of them quietly asserted that nothing anyone had failed to measure mattered.

So the drawing gets done and the trace gets skipped. The step that gets left off most reliably is the last one: stress the graph. Take the arrow you are least sure of, flip it or delete it, redraw, and run the same trace again.

If the adjustment set survives that, you have a decision. If it does not, what you have is a question, and the honest output of the exercise is the question.

Build more than one plausible DAG, and not as a formality. Build the version a skeptical colleague would defend. Build the version where the arrow you are least confident about points the other way. Build the version where selection into the sample is drawn explicitly instead of assumed away — the version that would have caught the COVID-testing problem before anyone estimated anything.

The bill for skipping that arrives itemised. Decades of observational studies suggested hormone therapy protected postmenopausal women against coronary heart disease. Then the Women's Health Initiative randomised 16,608 women aged 50-79 with an intact uterus, recruited by 40 US clinical centres in 1993-1998. 8,506 took conjugated equine estrogens plus medroxyprogesterone acetate; 8,102 took placebo. The trial found the opposite of protection. CHD hazard ratio 1.29 (nominal 95% CI 1.02-1.63, 286 cases). Invasive breast cancer 1.26 (1.00-1.59, 290 cases). Stroke 1.41 (1.07-1.85, 212 cases). Pulmonary embolism 2.13 (1.39-3.25, 101 cases).

The trial's own report in JAMA states what followed: “On May 31, 2002, after a mean of 5.2 years of follow-up, the data and safety monitoring board recommended stopping the trial of estrogen plus progestin vs placebo because the test statistic for invasive breast cancer exceeded the stopping boundary for this adverse effect and the global index statistic supported risks exceeding benefits.” The NHLBI stopped that arm and announced the decision on 9 July 2002. The rival graph won, and it won expensively.

What repaired the disagreement is the part worth copying. The trial's design was emulated inside the Nurses' Health Study, an observational cohort. The emulation recovered intention-to-treat hazard ratios of 1.42 (0.92-2.20) for the first two years and 0.96 (0.78-1.18) over full follow-up. Hernán and colleagues attributed the original discrepancy to time since menopause and length of follow-up, not to the data being observational. The fix was a re-specified estimand and an eligibility structure drawn out in the open — the things a graph forces you to write down. It was not another column of controls.

So run the trace on each of your graphs. Write down which conclusions hold across all of them and which depend on a single disputed arrow. That list is the real product of a graph review. It says, before an estimate exists, precisely which argument someone has to win for your result to stand.

A graph earns its keep at the moment it forces a room full of people to disagree out loud. What counts as the treatment, and is it the same treatment for everyone who received it. What happened before what. Who entered the sample, and what decided that. Whether the instrument was measuring the patient or partly measuring itself. Which common causes nobody recorded. Those arguments are worth having, and without something drawn on a page they tend not to happen at all.

Which is why the graph belongs with the analysis rather than in a slide someone made once. Version it beside the code. Tie each arrow to the assumption it stands for, so a reader can see which claim a given number is leaning on.

Sometimes the disagreement does not resolve. Two graphs both survive scrutiny and imply different adjustment sets, or different estimands altogether. Report that, rather than picking the graph with the cleaner answer. Then reach for something that can actually help: show how far the conclusion moves as the disputed arrow changes, find a design that does not depend on that arrow, or state the range of effects the data can support instead of one number it cannot.

That last instruction is not this lesson's opinion. Medicines regulators already enforce it, in a numbered guideline. ICH E9(R1) was adopted on 20 November 2019 and came into effect on 30 July 2020. It requires that the target of estimation be fixed before the method of analysis: “The targets of estimation are to be defined in advance of a clinical trial.” Its glossary then defines a sensitivity analysis as “A series of analyses conducted with the intent to explore the robustness of inferences from the main estimator to deviations from its underlying modelling assumptions and limitations in the data.” Assumptions documented and stress-tested, in other words, instead of assumed away. That is the same instruction as stressing the graph. A regulator wrote it, and gave it a document number.