Prediction, Causation, and Counterfactual Questions

The wording of the question determines which assumptions and evaluation designs are relevant. Optum's Impact Pro answered the first of the three well enough to be deployed at national scale. It was used as though it had answered the second.

A variable can influence both the action and the outcome. Sicker patients may receive more treatment and also have worse outcomes, which makes treatment appear harmful in raw observational data.

A predictive model can use that association effectively without resolving the causal story. Causal estimation cannot. It requires assumptions about which variables block or create bias, and those assumptions have a price that has been measured on data where the right answer was already known.

The National Supported Work Demonstration was randomized, so its answer was known. In 1982 dollars the experimental estimates were $851 (SE $317) of extra 1979 earnings for AFDC female participants and $886 (SE $476) of extra 1978 earnings for male participants. Then LaLonde threw the randomized control group away. Writing in the American Economic Review in September 1986, he rebuilt the evaluation with the observational comparison groups and the econometric adjustments then standard, and scored the results against those known benchmarks. For the women the non-experimental estimates were usually positive and larger than the experimental one. For the men they were negative and smaller. Same data, same programme, and the error running in opposite directions by subgroup. His abstract draws the conclusion: “This comparison shows that many of the econometric procedures do not replicate the experimentally determined results, and it suggests that researchers should be aware of the potential for specification errors in other nonexperimental evaluations.”

The patients below are invented, with counts chosen to make the arithmetic easy. A ward treats 200 patients. Doctors give the treatment mostly to severe cases, which is the confounding the section above describes.

The numbers below are invented to make the arithmetic easy. Splitting by severity helped in the table above because severity caused both the treatment and the deaths. Splitting by a variable that the causes themselves produce does the opposite: it manufactures an association. Step 4 of the design review below calls such a variable a collider.

Take one severe patient from the invented ward above: treated, and alive at the end. A counterfactual question asks whether this patient would have lived without the treatment. The table cannot say, and neither can any model trained on it.

Every method leaves one requirement in place. Treatment, outcome, population, timing, and interference still have to be defined.

External variation is the rung that is easy to describe and hard to find. Card and Krueger found some. On 1 April 1992 New Jersey raised its minimum wage from $4.25 to $5.05 an hour, and Pennsylvania's stayed at $4.25. They surveyed 410 fast-food restaurants on both sides of that line, before and after the change, and reported it in 1993: “Relative to stores in Pennsylvania, fast food restaurants in New Jersey increased employment by 13 percent.” The standard model predicted the opposite. In 2021 the Royal Swedish Academy of Sciences awarded David Card a share of the Nobel Prize in Economic Sciences “for his empirical contributions to labour economics”, much of it built on natural experiments like this one.

The randomized rung is bounded by ethics and operations, and occasionally those bounds hand you the design. Oregon reopened its OHP Standard Medicaid programme in 2008 with about 10,000 slots and roughly 140,000 eligible adults. The state's own final report on what it did next gives the reason: “When the department decided to open enrollment for OHP Standard, an estimated 140,000 Oregon adults would have been eligible for the program, so the department determined the most equitable way to select enrollees was to create a reservation list from which names would be randomly drawn.” More than 90,000 people put their names on the list. Eight staggered draws from March to October 2008 sent applications to 35,476 people, and 10,031 enrolled.

Because assignment was random, a rationing decision produced a genuine randomized trial. After about two years it found no statistically significant effect of Medicaid on measured blood pressure, cholesterol or glycated hemoglobin. It cut depression by 30 percent, 9.2 percentage points from a base of 30, and it virtually eliminated catastrophic out-of-pocket spending, down 4.5 points from a base of 5.5. Randomization did not make every hoped-for effect appear. It made the answer legible in both directions.

One product can need both predictive and causal models, and the second is a separate piece of work with its own design and its own bill. The tutoring version of the question below was settled by assignment rather than by forecasting. Twelve authors reported two randomized controlled trials of Saga Education's high-dosage tutoring in Chicago public high schools in the American Economic Review in March 2023: 2,633 ninth- and tenth-graders randomized in the first, 2,710 students in the replication. Their abstract: “Participating in math tutoring increases math test scores by 0.18 to 0.40 standard deviations and increases math and non-math course grades.” An earlier working paper, from March 2021, reported the same two trials as 0.16 SD and 0.37 SD, at a cost of $3,500 to $4,300 per participant per year. No improvement in the accuracy of the first bullet in each pair below would have produced those numbers.

Before modeling, make the hypothetical comparison explicit.

Step 5 is not a matter of taste, and at least one regulator has written the ordering into public documents. The 21st Century Cures Act, enacted on 13 December 2016, defines real world evidence as data on a drug's usage, benefits or risks “derived from sources other than traditional clinical trials”, and it required a framework within two years. The FDA published that framework in December 2018. It records that randomization has been considered a critical element in establishing a causal relationship, and then says what the alternative costs: “Although observational studies may provide credible evidence, there is a stronger scientific justification for deriving evidence of a drug effect from randomized controlled trials as compared to observational studies.” The statute defines a whole category of evidence by what it is not: not a traditional clinical trial. The framework then prices the difference.

Words such as “drives,” “prevents,” “because,” and “impact” imply more than predictive association. Feature importance, correlation and model explanations do not establish intervention effects.

How often the wording outruns the design has been counted, not merely deplored. Haber and his co-authors, in the American Journal of Epidemiology in 2022, screened 1,170 articles on non-randomized studies from 18 high-profile journals published 2010–2019 and had three reviewers rate each abstract. The exposure–outcome linking language carried no causal implication in 13.8% of abstracts, weak in 34.2%, moderate in 33.2% and strong in 18.7%. The commonest linking word was “associate”, at 45.7%. The recommendations ran ahead of the findings they rested on: “The implied causality of action recommendations was higher than the implied causality of linking sentences for 44.5% or commensurate for 40.3% of articles.” At larger scale, Wang and Yu classified 91,933 observational-study abstracts in PLOS ONE on 12 August 2026 and found 31.7% making causal claims. That is close to the 31% Cofield and colleagues had found by hand in 525 obesity and nutrition papers.

Use descriptive language when the evidence is predictive. If the product acts on causal claims, involve domain experts and appropriate experimental or causal-inference methods.