Heterogeneous Treatment Effects and CATE

Anyone with a modest average effect and enough attributes to slice by will find a slice that looks spectacular. The cleanest demonstration of what such a slice is worth was staged by trialists on their own data. They ran it deliberately, and printed it beside the result it was meant to protect.

ISIS-2 randomised 17,187 patients entering 417 hospitals. Aspirin cut 5-week vascular mortality from 1016/8600 (11.8%) to 804/8587 (9.4%). That is a 23% (SD 4) odds reduction, with 2p<0.00001. As treatment effects go, that is about as unambiguous as evidence gets.

Then the trialists divided those same patients by astrological birth sign. Patients born under Gemini or Libra showed a 9% (SD 13) increase in mortality on aspirin. Every other sign showed a 28% (SD 5) reduction. The trial’s own report in The Lancet, in 1988, put it in print: “For example, subdivision of the patients in ISIS-2 with respect to their astrological birth signs appears to indicate that for patients born under Gemini or Libra there was a slightly adverse effect of aspirin on mortality”.

Nothing distinguished those patients from the rest except the month they were born in. The birth-sign number described the shape of the search, not the shape of the response. The trialists printed it because the next reader would run the same search over variables that sound serious, and believe the answer.

Targeting a segment is the last stage of five. A search like the birth-sign one performs the middle stage first.

The sequence starts with the base estimand: the average effect the trial or experiment was built to measure, before anyone slices anything. In ISIS-2 that was the 23% (SD 4) odds reduction in 5-week vascular mortality. Then modifiers are selected, and selected is the honest word. Someone decides in advance which baseline variables could plausibly change the response, and writes that list down where it can be counted later. A birth sign fails at this rung, and fails visibly. That is the whole reason it was chosen for the demonstration. Only then is a CATE estimated. The fourth stage validates the heterogeneity: does the pattern survive data that had no hand in finding it? Translating to policy comes last, and it is a separate question with its own arithmetic.

Read downward, each rung carries an assumption the rung below it depends on. Read upward, each rung is a decision that can be got wrong on its own. A result can be perfectly computed at one rung and reported as though it came from another.

Heterogeneous treatment-effect analysis asks one question. Does the average causal contrast differ across covariates measured before treatment? When the variation is real, and the data genuinely contains the groups being compared, the answer supports targeting, interpretation and scientific learning.

The difficulty underneath is not a matter of technique. Each unit reveals one outcome: the one that happened. The effect on that unit is the gap between what happened and what would have happened otherwise. The second half of that gap is unavailable for everyone, forever. None of the 17,187 patients in ISIS-2 supplied both halves. The 23% (SD 4) odds reduction became visible only because 8600 and 8587 patients were compared as groups. Averages over groups are the only place a causal effect becomes visible at all. A conditional average is not a blurry approximation of the individual effect. It is a different quantity, and the only one on offer.

Rainfall across climate zones behaves the same way. Regional averages differ enough to plan around without telling you the path of a single droplet. The difference is that climate zones sit there waiting to be measured. Treatment groups — and the comparability that makes them mean anything — have to be built.

That building is where a flexible search does its damage. Hundreds of candidate splits over thin slices of data will return a large number somewhere. A large number is precisely what the search was constructed to find.

Four claims sound almost the same in a meeting and demand entirely different evidence.

The mildest is that the average effect is positive overall. Next comes the claim that the effect is larger in a group named before the data were seen: one prespecified split, one comparison, and a reader can count the tests that were run. Third is the claim that a model has discovered structure in the response. There, many variables offered many possible splits, and the winning slice was chosen by the same data now being used to support it. Last is the claim about a person, which no design delivers, because the comparison it names was never observable in the first place.

How often the third level is presented as the second has been counted. Wallach and colleagues went through 64 randomised trials making 117 subgroup claims in their abstracts, and published the tally in JAMA Internal Medicine in 2017. Only 46 (39.3%) had a significant interaction test behind them at all. Of those 46, only 13 (28.3%) had been prespecified. Exactly 1 (2.2%) adjusted for multiple testing. Then comes the part that settles the argument: “Only 5 (10.9%) of the 46 subgroup findings had at least 1 subsequent pure corroboration attempt by a meta-analysis or an RCT. In all 5 cases, the corroboration attempts found no evidence of a statistically significant subgroup effect.”

Five out of five is not a failure of arithmetic. Nothing in those estimates was necessarily computed incorrectly. The claims were simply one rung above what the evidence could carry, and the rung is invisible in the printed number.

The levels blur in conversation because the words blur first, and two of these four are routinely used for one another. They carry the rest of this lesson, so they are worth pinning down before the validation plan arrives.

Treat CATE as a prediction target that can never be checked against its own truth. No row in the data holds the number the model is predicting, so the usual reflex — hold out some data, compare predictions to outcomes — does not apply directly. Everything in the plan below is a substitute for the comparison you cannot make.

Fix the base estimand and the list of candidate modifiers before touching the data, so the number of comparisons is known rather than discovered afterwards. This is not house advice. It is the regulatory test. The European Medicines Agency’s guideline on the investigation of subgroups in confirmatory clinical trials was adopted in January 2019 and took effect on 1 August 2019. It makes prespecification, biological plausibility and replication the test of whether a subgroup finding is credible. On the problem with running many subgroup analyses, the guideline is plain: “Considering multiple subgroups increases the probability of false positive findings, defined here as subgroups where the effect is concluded to differ from the effect seen in the primary analysis population when in fact it does not.”

Split the sample honestly, letting one part find structure and the other measure what was found. Check support before believing any local estimate. A slice holding few treated units, or few untreated ones, contains no real comparison, and it will still return a confident number.

Then evaluate the targeting rule the model implies, rather than the model that implied it. There is a named estimator for exactly that. Rank-weighted average treatment effects (RATE) score the rule. The Qini coefficient falls out as a special case, a central limit theorem comes attached for inference, and the worked application is aspirin targeting after stroke. Yadlowsky and colleagues defined them in the Journal of the American Statistical Association in 2024. Their opening sentence is why the object of evaluation has to be named out loud: “There are a number of available methods for selecting whom to prioritize for treatment, including ones based on treatment effect estimation, risk scoring, and hand-crafted rules.” Those methods produce different rules. It is the rule that gets deployed.

Written afterwards, that plan is a defence. Written beforehand, it is a test the result is allowed to fail.

Credible effect modification meets two conditions, and a birth sign meets neither. The variable is measured before treatment, because anything measured afterwards may have been changed by the treatment itself, so grouping on it compares groups the treatment created. And the variable connects to a mechanism or to a policy constraint, meaning someone could have said in advance why the response should differ along it.

That second condition is what separates a modifier from a coincidence with a name. It is necessary, and it is not sufficient. The first bullet below is a mechanism everyone can state out loud, and when someone finally measured it, it pointed the other way.

Heterogeneity earns its way into a policy in two situations. It produces differences repeatable enough to raise expected policy value once costs and capacity are counted. Or it identifies a group the average policy would harm, which is worth acting on even when the aggregate barely moves.

Everything else is a finding, not a policy. When the variation will not reproduce — as it did not in any of the 5 corroboration attempts Wallach and colleagues could locate — or when it sits in a region with too little support to measure, the simpler average-effect rule is the better rule, and collecting targeted evidence is the better next step.

What makes this a decision problem rather than an estimation problem is that the constraints are part of the object. Athey and Wager formalise that in Econometrica in 2021: “In many areas, practitioners seek to use observational data to learn a treatment assignment policy that satisfies application-specific constraints, such as budget, fairness, simplicity, or other functional form constraints.” They prove regret guarantees for the rule that is chosen. So the thing carrying the guarantee is the policy, not the effect model behind it. Budget and capacity belong inside the arithmetic, not in a caveat printed after it.

Report the uncertainty around the boundaries as well as the effects inside them. A subgroup edge is an estimate with error attached, and a person sitting just on either side of it is essentially the same person. Converting a continuous estimate into a rigid personal label discards that fact and accepts the consequences of being wrong about it.

The error the ISIS-2 trialists staged for their readers was never really statistical. A number was allowed to choose an action before anyone had asked what the action would be worth if the number were right.