Inverse Probability Weighting and Pseudo-Populations

For years the Nurses' Health Study was read as showing hormone therapy protecting the heart. Then the same cohort was re-analysed to emulate the Women's Health Initiative trial. No new women, no new records — only a different design imposed on data already collected. The finding did not survive in the shape it had been carrying.

The re-analysis appeared in Epidemiology in 2008. It compared estrogen/progestin initiators against non-initiators. The intention-to-treat hazard ratio of coronary heart disease was 1.42 (95% CI 0.92–2.20) for the first 2 years, and 0.96 (0.78–1.18) over the whole follow-up. The early figure points the opposite way from protection. The whole-follow-up figure sits close to nothing at all.

Intention-to-treat asks about the decision to start, not about the treatment. Women stop, switch and restart. The label attached to a person at baseline gradually stops describing what that person is taking. Hernán and colleagues put the reason in one line of the abstract: “Because the ITT approach causes severe treatment misclassification, we also estimated adherence-adjusted effects by inverse probability weighting.” The adherence-adjusted, inverse-probability-weighted hazard ratios were 1.61 (0.97–2.66) for the first 2 years and 0.98 (0.66–1.49) overall.

Nothing had been coded wrong in the older reading of the cohort. The arithmetic was fine, as arithmetic generally is. What changed was the thing underneath it: which comparison the number claimed to describe, and which units were being asked to stand for whom while it was computed. Weighting is where that claim gets made explicit. That is why it is worth learning to read weights before learning to trust them.

A weight is an instruction about representation. Give a record a large weight and you are telling the estimator to count that one person many times over. Do it to everyone, using the reciprocal of the probability of the treatment each of them actually received, and you get a reweighted version of the sample. That is the pseudo-population. In it, the measured baseline covariates are balanced across treatment groups. Treatment there looks as if it had been assigned without regard to those covariates, so the marginal outcomes can be compared directly on whatever scale the question is asked in.

Two assumptions carry that claim, and they fail in different ways. Conditional exchangeability says the measured covariates are enough: hold them fixed and the treated and untreated were comparable. Positivity says every kind of unit in the target population had some real chance of either treatment.

The second one has a methodological literature of its own. Petersen and colleagues reviewed it in 2012 and stated the failure mode plainly: “Positivity violations occur when certain subgroups in a sample rarely or never receive some treatments of interest. The resulting sparsity in the data may increase bias with or without an increase in variance and can threaten valid inference.” Read the middle clause twice. Bias with or without an increase in variance means the damage does not have to announce itself in a confidence interval. An estimate can be both tight and wrong. The same paper frames every standard remedy — restricting the adjustment set, restricting the sample, changing the projection function or the target intervention — as changing the question rather than cleaning the data.

Positivity is the assumption the weights themselves report on, which is the useful thing about them. As the probability of the observed treatment approaches zero, its reciprocal explodes. The exploding weight is the method saying out loud that this particular comparison has almost no data behind it. Read that way, a weight distribution is a diagnostic before it is a correction.

Weights are not raw facts either. They come out of a model of treatment assignment. So they inherit every error in that model, every error in how treatment was recorded in the first place, and every stretch of the covariate space where support is thin. Kang and Schafer measured that inheritance in 2007. They simulated 1,000 samples from a population with true mean μ = 210. With a correctly specified propensity model at n = 1,000, the IPW-POP estimator had bias −0.01 and RMSE 1.81 — essentially unbiased. With a misspecified one, the same estimator had bias 5.05 and RMSE 12.10. It was worse at n = 1,000 than at n = 200, where bias was 1.58 and RMSE 8.35. The error grew as the sample grew. More data means more chances to estimate a small propensity and hand its reciprocal to the average. Their abstract puts the condition in one line: “Methods that use inverse-probabilities as weights, whether they are DR or not, are sensitive to misspecification of the propensity model when some estimated propensities are small.”

Which weights you build should follow from the question, not from the default in whatever routine you called. ATE weights describe the full eligible population. ATT weights describe the treated population, a narrower group that is usually far better supported. Overlap weights give each unit a weight proportional to its probability of being assigned to the opposite group; Li and colleagues introduced them in 2018. They lean deliberately toward the units whose treatment was genuinely uncertain, and away from those whose treatment was a foregone conclusion. What makes them more than a preference is stated in the abstract: “The overlap weights are bounded, and minimize the asymptotic variance of the weighted average treatment effect among the class of balancing weights.” They also carry an exact small-sample mean-balance property. Different questions, different answers. The software will not ask which one you meant.

An analyst can define every one of these correctly and still be blindsided. The vocabulary of weighting mostly describes the design, and only rarely describes the evidence. Read them in order. The early ones name what you built. The last one names what survived building it.

It runs in the order the work has to be done, and the order is not decoration. Choose the target first, because ATE, ATT and overlap answer different questions and everything downstream inherits that choice. Estimate the propensities, the probability of the treatment each unit received. Construct the weights from them. Then audit the pseudo-population: check that the covariates balanced, and check what the weight distribution looks like while you are in there. Estimate outcomes last.

Every arrow is also a place to write down an assumption. Then, when the estimate turns out to rest on a few rows, you can point at the step that put it there instead of arguing about the result.

Balance tables are the usual evidence that weighting worked: after weighting, the measured covariates line up across the groups. Balance is necessary. It is not sufficient, because it says nothing about how the balance was bought.

The other half of the evidence is what a weight distribution looks like when somebody actually prints it. The EFFECT study covered 9,107 patients discharged alive after acute myocardial infarction from 102 Ontario hospitals, between 1 April 1999 and 31 March 2001. Austin and Stuart used it in 2015 to demonstrate IPTW diagnostics. They reported the full weight distribution for two propensity-score specifications rather than a reassuring summary of it. For the simple specification: “the mean stabilized weight was equal to 1.002, while the standard deviation of the stabilized weights was equal to 0.29. The minimum and maximum weights were 0.353 and 4.656, respectively.” The complex specification gave a mean stabilized weight of 1.001, SD 0.30, range 0.361–3.793. A mean far from one, or very extreme values, they treat as a signal of non-positivity or model misspecification.

That is the shape of a weight distribution that is behaving. What decides how much you actually know is concentration — whether the weight is spread across the sample or piled onto a few rows. These make it visible, and not one of them requires looking at the outcome.

The order is the safeguard. Once the treatment effect is on the screen, every decision that follows is being made by someone who knows which way each choice pushes the answer: what counts as an extreme weight, where to trim, whether balance is good enough. Nobody is disinterested enough for that to be safe.

This is not only a private discipline. It has been codified. ICH E9(R1), the addendum on estimands and sensitivity analysis in clinical trials, was adopted under Step 4 on 20 November 2019. It took legal effect in the EU on 30 July 2020. It defines the object it requires: “An estimand is a precise description of the treatment effect reflecting the clinical question posed by a given clinical trial objective.” The targeted population is one of the named attributes that must be specified when the estimand is constructed — in advance of the trial. A regulator already treats "which population is this number about" as something you write down before you can see what the answer would be.

So assemble the dashboard first, and judge the design on it while the outcome stays covered. The weight distribution and its maximum. The effective sample size beside the nominal one. Balance overall, and balance in the tails. The influence of the heaviest units. If the design fails there, it fails whatever the effect turns out to be. That is the only moment at which you can say so and be believed.

Weighting earns its place when the target population has credible support in the data and no small set of units dominates the estimate. Both conditions, not one. A design can balance beautifully and still hand the answer to a few rows.

When instability survives every reasonable diagnostic, the honest moves are all forms of narrowing. Redefine the target to a population the data actually covers. Switch to the bounded overlap weights of Li and colleagues, and answer for the units whose treatment was genuinely in doubt. Match, and accept the smaller comparable set. Or report the region where support was missing and decline to extrapolate into it. Petersen and colleagues are explicit that each of these is a change to the question, not a repair to the data. Which is precisely why each of them has to be declared.

What you cannot do is take a heavily trimmed estimate and keep calling it the ATE. The trimming changed the population underneath the label, and a label that hides that is a claim about people the study never observed.

The cohort that opened this lesson still had a finding available to it. Not hormone therapy protecting the heart — that reading did not survive the trial emulation. What it had was a stated contrast under a stated protocol: estrogen/progestin initiators against non-initiators, adherence-adjusted by inverse probability weighting, at 1.61 (0.97–2.66) for the first 2 years and 0.98 (0.66–1.49) overall. Intervals that wide are not a failure of the analysis. They are the size of the answer the data could support, published at that size instead of at a more attractive one.