Identification Versus Estimation

When the same machinery is applied thirty-two times and agrees with the truth in one half of the cases and not the other, the machinery is not the thing that varied.

Thirty-two randomised trials, each one reproduced from insurance claims. That was RCT-DUPLICATE, funded by the FDA, and the results ran in JAMA in 2023. Every emulation used the same design template — a propensity-score-matched new-user cohort study — drawn from three US claims databases: Optum Clinformatics, MarketScan and Medicare. The estimator family was held fixed across all 32. What varied was whether the design elements that define the question — population, intervention, comparator, outcome, time — could be emulated in claims data at all.

Overall agreement with the randomised results was Pearson r = 0.82 (95% CI, 0.64-0.91). That is the figure that fits on a slide. Split by design fidelity, it comes apart. The 16 emulations that could reproduce the trial's design elements reached r = 0.93 (0.79-0.97). The 16 where those elements could not be emulated reached r = 0.53 (0.00-0.83). Same databases, same estimator, same investigators. The lower end of that second interval is zero.

The authors put the finding in one sentence: “Real-world evidence studies can reach similar conclusions as RCTs when design and measurements can be closely emulated, but this may be difficult to achieve.” Two different questions were on the table in each of those 32 attempts. Only one of them is a question an estimator can answer.

Identification means your assumptions leave a causal quantity exactly one possible value, given the distribution of data you can actually observe. That is a property of the question and the design around it. It is settled before anyone picks a model, and picking a different model does not revisit it.

This is not the house style of one research school. It is a regulatory requirement. The harmonised addendum on estimands and sensitivity analysis, ICH E9(R1), was adopted in 2019, and it makes the separation explicit: “This framework enables proper trial planning that clearly distinguishes between the target of estimation (trial objective, estimand), the method of estimation (estimator), the numerical result (“estimate”, see Glossary), and a sensitivity analysis.” Europe adopted it with legal effect from 30 July 2020. The FDA issued it as final guidance in 2021. Sponsors in the EU, the US and Japan are held to it.

Estimation is the separate job of approximating that identified quantity from a finite sample. This is where flexible methods earn their reputation. They fit the auxiliary pieces better. They pick up differences between subgroups. They lose less to a badly guessed functional form. All of that is real, and all of it happens downstream of the target being fixed.

What no learner can do is manufacture information that was never recorded. If treatment was assigned partly on a judgement nobody wrote down, that judgement is absent from every row, in every fold, at every sample size. If no untreated person resembles the treated ones, fitting harder does not invent one. A finer ruler gives you a more precise reading of whatever point you decided to measure. It has nothing to say about whether the point belongs where you put it. The comparison is unfair in one direction: a ruler is fixed, while causal assumptions can be argued with, probed against a design, sometimes falsified outright. But that argument is made in sentences, not inside the fit.

An unidentified effect and a badly estimated one arrive looking alike: a number, an interval, a plot with a few odd points. The remedies have nothing in common.

The cleanest documented pair uses one dataset twice. The Nurses' Health Study data on postmenopausal hormone therapy was re-analysed in 2008 by Hernán and colleagues, in Epidemiology. No new data was collected. No exotic estimator was reached for. They re-specified the question as a sequence of emulated trials, with a defined eligibility, a defined time zero and an intention-to-treat contrast, and reported that “The ITT hazard ratios (HRs) (95% confidence intervals) of CHD for initiators versus noninitiators were 1.42 (0.92-2.20) for the first 2 years, and 0.96 (0.78-1.18) for the entire follow-up.” That 1.42 points the same way as the hazard ratio of 1.29 the randomised Women's Health Initiative had found in 2002 — out of a cohort whose earlier analyses had shown a strong protective association. What moved the answer was eligibility, time zero and the contrast. Three sentences of design, not one line of code.

Estimation error responds to work you can do at your desk: a better nuisance model, a more robust estimator, more data of the kind you already have. Design failure responds to none of it. It asks for a different question, a different population, a different way of generating the data, or a frank statement of what cannot be recovered. Reaching for the first set of tools when the fault lies in the second is how a team ends up steadily improving a number that was never measuring the treatment.

The 16 emulations that landed at r = 0.53 have a family. In each of the following the work looks like a modelling task, the software runs without complaint, and what is missing is causal information that no estimator supplies.

Any causal result handed to you can be pulled apart into four layers. ICH E9(R1) gives three of them names.

At the top is the estimand, defined in the addendum as “A precise description of the treatment effect reflecting the clinical question posed by the trial objective”: for whom, under which intervention, compared with what. Below it is the argument that turns that quantity into something computable from observed data, together with the assumptions it leans on. Below that is the estimator, “A method of analysis to compute an estimate of the estimand using clinical trial data”. At the bottom is the estimate, “A numerical value computed by an estimator”, and its reported uncertainty.

Read a result from the bottom and the printed number is all you ever see. Read it from the top and you find out whether the quantity was reachable at all. The 16 RCT-DUPLICATE emulations that reached r = 0.53 did not fail at the estimator layer. They failed one layer above it, where the population, intervention, comparator, outcome and time could not be reconstructed.

The addendum is explicit about the order in which the layers are built: “The targets of estimation are to be defined in advance of a clinical trial.”

Take an analysis someone is proposing and force the two halves apart on paper, in that order.

The first half is prose and mentions no software. It names the population, the intervention and the comparison. It states the assumptions that would let observed data stand in for outcomes nobody got to see. It says where those assumptions would break. If this paragraph cannot be written, there is nothing for an estimator to estimate.

That has been turned into a published checklist. In 2025 Imbens and Xu revisited LaLonde's 1986 experimental-benchmark critique, running modern estimators on the same data. Their finding is a warning about the second half: “We show that modern methods, when applied in contexts with sufficient covariate overlap, yield robust estimates for the adjusted differences between the treatment and control groups. However, this does not imply that these estimates are causally interpretable.” Robustness across methods is not evidence of identification. What they recommend instead is examining the assignment process, inspecting overlap and running placebo tests. Goodness-of-fit tests they judge inadequate for the job.

The second half is the numerical procedure. It becomes worth arguing about only once the first half survives being read by someone hostile. Written in this order, a missing comparison group turns up in a sentence, before anyone fits anything.

The whole distinction rests on these, so it is worth being exact about what each one means.

If the identification argument rests on assumptions you would not defend to a hostile reader, or on a population where the comparison does not exist, more estimation is not a remedy. It is camouflage. Complexity buys the appearance of rigour and moves the weak joint further from view.

Four honest moves remain, and none of them is a better estimator. Narrow the target to the part of the population where a comparison genuinely exists. Collect the data that would record whatever is driving treatment. Find or build a design that carries the argument on its own — the route that took the Nurses' Health Study data from a strong protective association to 1.42 (0.92-2.20) in the first 2 years. Or report the set of values the assumptions actually support, and say what it would take to shrink it.

The fourth move is not a theoretical option. In the spring of 2020 nobody could say how many people had been infected, because test data was missing non-randomly. Manski and Molinari published the range instead of a number: “We find that the cumulative infection rates as of April 24, 2020, for Illinois, New York, and Italy are, respectively, bounded in the intervals [0.004, 0.525], [0.017, 0.618], and [0.006, 0.471].” The corresponding infection-fatality rates were bounded to [0, 0.033], [0.001, 0.049] and [0.001, 0.077], against reported death rates among confirmed cases of 0.045, 0.059 and 0.134. Those intervals are wide enough to be uncomfortable. They are also what the data and the stated assumptions support. A bounded claim over the region with real comparisons, plus a stated blank for the rest, is a smaller result than a point estimate. It is a true one.

Once identification is credible, the choice of estimator becomes worth arguing about on its own terms: robustness, efficiency, the diagnostics it exposes, and whether it fits the design that made the argument work in the first place.