Outcome Regression and Counterfactual Model Diagnostics

Some subgroups in a study rarely or never receive one of the treatments. The failure has a name — positivity — and a stated mechanism. A review published in 2012 puts it in two sentences: “Positivity violations occur when certain subgroups in a sample rarely or never receive some treatments of interest. The resulting sparsity in the data may increase bias with or without an increase in variance and can threaten valid inference.”

Take the second sentence slowly, because it removes the defence most teams reach for. The sparsity may raise bias “with or without an increase in variance”. A subgroup effect can arrive with a narrow interval and a stable-looking value and still be wrong. What has gone missing is not precision. It is comparison.

Set that against the model a team is proud of. A gradient-boosted learner, trained on the whole study population, predicts observed outcomes accurately. Its test error is low enough that nobody argues with it. So they ask it the question the study existed to answer: does treatment help the rare high-risk patients? The model returns an effect for that subgroup, and the number looks exactly as reliable as every other number it has produced.

The subgroup contains twelve untreated patients. Every claim about what would have happened to a high-risk patient left alone rests on those twelve. Mostly it rests on the region past them, where the model has stopped comparing anyone to anyone and is continuing a curve it learned elsewhere. This is precisely a subgroup that rarely receives one of the treatments of interest. By the review's own account, the estimate can hold steady while the bias grows.

The validation report cannot see this. Test error averages over everybody, and everybody is mostly ordinary patients who received both treatments in workable quantities. A subgroup thin enough to break the estimate is far too small to move the average that was supposed to guard it.

What would have caught it are checks that refuse to average. Calibrate predictions against observed outcomes separately within each arm, not across the pooled data. Read residuals by slice — risk level, site, time, subgroup — instead of as a single figure. Flag counterfactual predictions that sit where propensity or neighbour density says no comparable unit lives. Then refit under different interactions, different learners and a restricted target population, and see whether the effect survives.

An outcome regression is asked for the outcome under each treatment strategy, for every unit in the study. The effect is the gap between the two. One of those predictions describes something that happened and can be held against a recorded outcome. The other describes a world that was never run.

That asymmetry is the whole difficulty. Predictive validation grades the first prediction and is silent about the second. The second is where the estimate for the rare subgroup came from.

What goes into the model follows from this. It has to carry baseline prognosis — everything known about a unit before treatment that shapes the outcome. It has to let treatment interact with those baseline traits, or it has no way of expressing that treatment works differently for different people. And it must not contain anything measured after treatment. Adjust for something the treatment itself caused and the effect you are trying to measure gets absorbed into the covariate.

The choice of learner is then a choice about what you are willing to give up. A parametric model fixes the functional form in advance. That makes it readable, and makes its extrapolation an assumption the analyst stated rather than one the fit invented. A flexible learner — trees, boosting, forests, neural models — finds shapes nobody specified, and extrapolates in ways nobody specified either.

That last sentence has been measured in public. A competition at the 2016 Atlantic Causal Inference Conference, “Is Your SATT Where It's At?”, drew thirty competitors across two tracks. The black-box track alone ran 7,700 simulated realisations — 77 scenarios, 100 replications each. Every one was built on 4,802 complete-case observations with 58 covariates from the Collaborative Perinatal Project. Lack of overlap was one of the knobs the organisers turned deliberately. Dorie and Hill say what that does to a model, in their 2019 report on the competition: “This type of lack of overlap can be particularly challenging for methods that rely on models since many methods will attempt to extrapolate beyond the range of common support.”

The results split cleanly along the line this lesson is about. Methods that flexibly modelled the response surface performed best overall, so flexibility earned its place on point estimation. Interval coverage stayed broken anyway. The nine worst methods covered below 75% of the time. CBPS and h2o Ensemble bought their coverage with average interval lengths of 0.78 and 6.1. BART's own intervals averaged about 82% coverage until a TMLE augmentation lifted them to near-nominal. Winning on error and telling the truth about uncertainty were not the same achievement. No amount of the first supplied the second.

Read the audit as a route, with a stop at each decision, where any stop can invalidate the ones before it.

State the target first: which treatment strategies, in which population, compared how. That stop is not a matter of analytical taste, and it has a date on it. ICH's E9(R1) addendum on estimands was adopted at Step 4 on 20 November 2019, and the FDA adopted it as guidance for industry on 12 May 2021. It defines what the first stop produces: “An estimand is a precise description of the treatment effect reflecting the clinical question posed by a given clinical trial objective. It summarises at a population level what the outcomes would be in the same patients under different treatment conditions being compared.” The estimand is assembled from four attributes — treatment condition, population, variable and population-level summary. That is the map's first stop written as a checklist.

The guideline also fixes when the stop happens: “The targets of estimation are to be defined in advance of a clinical trial.” In advance. Not at analysis time, and not after a flexible learner has shown you which subgroup looks interesting.

Then choose baseline covariates, taking nothing measured after treatment. Decide how treatment is allowed to interact with those covariates, because that decision is what lets the effect vary at all. Validate the fit against observed outcomes, which establishes that the model is competent at the half of its job that can be graded.

Then audit support, and expect this last stop to be the one that hurts. It is where the twelve untreated patients would finally have shown up, long after the model had passed everything else.

A specification ladder is one model after another, each more flexible than the last, all of them aimed at the same causal contrast: same target, same population, same baseline covariates, and no post-treatment variable on any rung.

You are not climbing it to find the rung that fits best. You are climbing it to see whether the estimated effect moves as the flexibility increases. An effect that holds steady is resting on the data. An effect that swings is resting on the functional form. The honest report says which one you have.

Somebody has built the exhaustive version of that ladder, and published what it found in 2015. For each of 417 NHANES variables tested against all-cause mortality, Patel and Ioannidis fitted 8,192 Cox models — every combination of 13 adjustment covariates. Then they read the spread of the estimates that came back. They call that spread vibration of effects. The result: “For 31% of the 417 variables, we observed a Janus effect, with the effect being in opposite direction in the 99th versus the 1st percentile of analyses.”

For nearly a third of the questions, then, the estimate did not merely swing. It reversed sign, and both signs were available to an analyst who fitted one model and reported it. Name “a Janus effect” and you are naming a result that exists in the specification space rather than in the data.

The practical reading for a hand-built ladder of four or five rungs is not that 8,192 is the target. It is that your four or five are a sample from a space that large. A single reported specification is a sample of size one from it. The range across your rungs — not the estimate on your favourite rung — is the quantity worth putting in the report.

There are conditions under which modelling the outcome is the right move. Prognosis is measured well. Both treatment strategies were genuinely available across the covariate range. The diagnostics hold up separately within each arm rather than only in aggregate. Under those conditions it uses what you know efficiently.

Even then, run something beside it. Weighting leans on a model of how treatment was assigned instead of a model of the outcome; doubly robust methods use both. When the estimates agree, the answer is not hostage to a single modelling choice. When they disagree, you have found where the assumptions are carrying the result, which is more useful than the agreement would have been.

And where the data is thin — where the twelve untreated patients were — no estimator rescues you. The alternative is not a better fit but a smaller question, and that move has been worked out and numbered. The optimal subsample for estimating an average treatment effect under limited overlap was derived by Crump and Imbens in 2009. Their abstract reduces it to something you can apply on a Tuesday: “For a wide range of distributions, a good approximation to the optimal rule is provided by the simple rule of thumb to discard all units with estimated propensity scores outside the range [0.1,0.9].”

The trade-off is in the title of their earlier working paper rather than in a footnote: Moving the Goalposts. To “discard all units with estimated propensity scores outside the range [0.1,0.9]” is to move them. The estimand changes. You are no longer reporting an effect for the study population; you are reporting one for the population where both strategies were actually tried. That is a real cost and it belongs in the report. But it is an honest, stated restriction, which is more than can be said for letting a flexible model manufacture a comparison nobody in the study ever supplied.