Association, Prediction, Intervention, and Counterfactuals

In March 2012 eBay stopped buying paid search advertising on its brand keywords — the queries that contain the word “ebay” — on Yahoo! and MSN. The equivalent campaign kept running on Google, as a control. Then 99.5 percent of the paid click traffic eBay had stopped paying for came straight back through unpaid natural search. The company had been buying clicks from people who were already on their way to eBay.

A second test in the same study was larger. It ran on non-brand keywords, with the ads switched off in roughly 30 percent of US Nielsen DMAs. Conventional OLS on the pre-experiment data implied a return on investment of 4,173% without time and geographic controls, and 1,632% with them. The experimental estimate was −63%, with a 95% confidence interval of [−124%, −3%]. It was computed against eBay's imputed US paid-search spend of $51 million a year.

Same firm, same channel, same period, same spend. The number was +4,173% when an analyst conditioned on who had clicked. It was −63% when the firm intervened and decided who saw an ad. The three economists who ran the experiment put the mechanism in their abstract: “Because search clicks and purchase behavior are correlated, we show that returns from paid search are a fraction of conventional non-experimental estimates. As an extreme case, we show that brand-keyword ads have no measurable short-term benefits.”

Nobody had to click. The people who did were already shopping, already loyal, already headed for the site. Every one of those things raises purchasing on its own, ad or no ad. The gap in the logs was real. It belonged partly to the advertising and partly to the kind of person who reaches for a link like that. No amount of staring at the recorded data separates the two.

What eBay had before the experiment was one number that four different questions could lay claim to. Only one of those questions was the one the spend was meant to answer.

Filtering is what you do to data you already have. Take the rows where the ad was clicked, take the rows where it was not, compare them. Nothing in the world moved while you did that. You sorted records that were already written. Whatever decided who ended up in which pile is still deciding.

Setting is different. You choose who gets the treatment, by a rule you control, and in doing so you break the old process. Intent no longer selects the treated group, because you selected it. That is precisely what eBay did when it turned the ads off in a random set of markets.

The distinction has a name and a notation. Causal questions sort into three levels — Association, Intervention, Counterfactual — and each level gets its own symbol: P(y|x) for Association, P(y|do(x),z) for Intervention, P(y_x|x′,y′) for Counterfactuals. The typical counterfactual question is “What if I had acted differently?” Judea Pearl set the scheme out in Communications of the ACM in 2019. One sentence carries the ordering: “The second level, Intervention, ranks higher than Association because it involves not just seeing what is but changing what we see.”

Four questions hide inside most claims, and it helps to say them in order. What co-occurred in the data already recorded? What can be forecast for a case nobody has seen yet? What changes when a policy decides who gets treated? And what would have differed for the same people under a different strategy? The first two draw on the same recorded rows and sit at the first level. The third is the do-operator. The fourth is the last rung.

That the rungs do not quietly collapse into one another is a proved theorem, not a preference. The formal Causal Hierarchy Theorem measures the models where they do collapse, with respect to the Lebesgue measure over a suitable encoding of L3-equivalence classes of structural causal models, and finds that “the subset in which any PCH collapse occurs is measure zero”. Four researchers at Columbia's Causal AI Lab published it. Corollary 1 states the consequence in one line: “To answer questions at Layer i, one needs knowledge at Layer i or higher.”

So the first two questions you can answer with the data on hand. The last two you cannot, unless the treatment was assigned in a way you are willing to defend out loud.

Hormone therapy protects the heart. Four people can say that sentence and mean four incompatible things by it. This particular sentence has had its readings pulled apart in public, on one cohort, with both sets of numbers printed.

The observational Nurses' Health Study appeared to show that women who took hormones were protected from coronary heart disease. Hernán and colleagues re-analysed that same cohort as a sequence of emulated trials, matching the protocol of the randomised Women's Health Initiative. The protection went away: “The ITT hazard ratios (HRs) (95% confidence intervals) of CHD for initiators versus noninitiators were 1.42 (0.92-2.20) for the first 2 years, and 0.96 (0.78-1.18) for the entire follow-up.” Nothing new was collected. The same women, the same records, a different question — and an answer close to the randomised WHI result of 1.29 (1.02–1.63) for coronary heart disease.

The analyst means the pattern recorded among the nurses. Read as a description of who took what, it is right. The forecaster means that hormone-use history carries signal about later coronary events. Also right, and it needs nothing further to become right.

The clinician means that prescribing hormones will lower a woman's risk. That claim leans entirely on how hormone use came to be initiated. When the initiation was re-cast as a trial protocol, the estimate moved to 1.42 (0.92-2.20) in the first two years and 0.96 (0.78-1.18) overall.

The fourth reader means something narrower and much heavier: that a particular woman who had a heart attack would not have had it, had she been prescribed hormones. Neither the cohort nor the trial speaks to that. The other version of that woman was never recorded.

Advertising has no monopoly on this. The same four questions sit inside a sentence about medicine, and there the gap between the first reading and the third was large enough to stop a trial early.

The Women's Health Initiative estrogen-plus-progestin trial randomised 16,608 postmenopausal women aged 50–79, 8,506 to hormones and 8,102 to placebo. It was halted on 31 May 2002, after a mean 5.2 years. The hazard ratios came back the wrong way. With nominal 95% confidence intervals: 1.29 (1.02–1.63) for coronary heart disease, 1.26 (1.00–1.59) for invasive breast cancer, 1.41 (1.07–1.85) for stroke, 2.13 (1.39–3.25) for pulmonary embolism. Decades of observational evidence had pointed the other way. The conclusion in JAMA is one line long: “Overall health risks exceeded benefits from use of combined estrogen plus progestin for an average 5.2-year follow-up among healthy postmenopausal US women.”

The words that separate the four readings of a drug sentence are small enough to slide past a reader in a hurry. A tense here, a verb there, and the evidence required has quietly multiplied.

The exercise takes a minute. It is worth running on any claim that shows up wearing a verb like improves, drives, identifies, or prevents.

Write the sentence down. Then write it four more times: once as a statement about data already collected, once as a forecast about cases not yet seen, once as a claim about what happens when someone acts, and once as a claim about what would have happened to the same people under a different action. Change nothing but the question being asked.

Take a real one. The claim is that a widely used risk algorithm identifies the patients who need extra care. The algorithm scored need by predicting future health-care cost. Obermeyer and colleagues examined it across millions of US patients. As a forecast of spending it worked. As a statement about need it did not: at any given score, Black patients were substantially sicker than White patients. Correcting the target would have raised the share of Black patients flagged for extra help from 17.7% to 46.5%. Two New York State regulators wrote jointly to the CEO of UnitedHealth Group about Optum's Impact Pro on 25 October 2019: “By relying on historic spending to triage and diagnose current patients, your algorithm appears to inherently prioritize white patients who have had greater access to healthcare than black patients.” The letter demanded the company prove the algorithm was not discriminatory or stop using it.

That is the exercise done in public, by a regulator. The sentence held at the second level and broke at the third. Three of the four versions will usually turn out to be things your evidence does not reach. That is the useful part of the exercise, not a failure of it. The version you can defend is the version to say out loud, and it is often the dullest one on the page.

The check that catches all of this is a language check, and it costs a few seconds. Words like because, effect, identifies, prevented, and would have are promises. Each commits you to naming an intervention and naming the comparison it is measured against. If you cannot name both, the sentence is not ready. The repair is to say something smaller: what was observed, or what can be forecast.

In drug regulation this is no longer advice. A trial has to write its estimand down in advance. The rule is ICH E9(R1), the addendum on estimands and sensitivity analysis in clinical trials, adopted on 20 November 2019 and in effect in the EU since 30 July 2020. Its glossary defines the estimand: “A precise description of the treatment effect reflecting the clinical question posed by the trial objective. It summarises at a population-level what the outcomes would be in the same patients under different treatment conditions being compared.” With it goes a declared strategy for every intercurrent event — treatment policy, composite, while on treatment, hypothetical, or principal stratum. The verbs have to be settled before the data are analysed, in two jurisdictions, in writing.

Done habitually, the check guards two ends at once. A prediction that works stops being sold as a treatment, which is the mistake eBay's pre-experiment arithmetic walked into and the mistake the pneumonia rule encoded. And an average that holds across a population stops being retold as the story of one particular person, which is the mistake waiting at the far end of the same lesson.

No model catches either one. The sentence does.