Instrumental Variables Beyond Randomized Encouragement

Nobody was going to randomize which elderly heart-attack patients got a catheter. So a 1994 study in JAMA reached for something outside the clinical decision that still moved it: how far each patient lived from one type of hospital rather than another. The data were Medicare claims from 1987 through 1991. The abstract makes the case for the instrument in one sentence: “Patients' differential distances to alternative types of hospitals are strong independent predictors of how intensively an AMI patient will be treated and appear uncorrelated with health status.” The first half of that sentence is what the data can settle. The second half is a claim about the world.

Distance does other things.

It changes how easily a patient comes back for follow-up. It decides how much of a day, and how much money, one appointment costs. It shifts which alternative services are within reach at all. None of those reach the patient through the procedure. An effect estimated this way would carry the whole of what it means to live near medical infrastructure, packaged and labelled as the effect of one operation. The study could see the edge of this in its own results. After controlling for access, rural residence still added 0.6 percentage points of acute mortality — geography touching death by a route of its own. The headline estimate was correspondingly modest. The marginal effect of invasive procedures on mortality between one and four years was “at most 5 percentage points”. Admission to a high-volume hospital was worth less than 1 percentage point at four years.

How often does that second pathway exist? Somebody counted. A 2014 review in the Annals of Internal Medicine went through 187 published IV comparative-effectiveness studies. Of those, 114 used one of the four commonest instrument families: distance to facility, regional variation, facility variation, physician variation. Sixty-five of the 114 used mortality as the outcome. In every one of the 114 the reviewers could identify potential unadjusted instrument-outcome confounders. Almost nobody had gone looking: “Only 4 (6%) instrumental variable CER studies considered potential instrument-outcome confounders outside the study data.”

The usual checks cannot close that gap. A first-stage F statistic confirms that the instrument really does move treatment. That is useful, and completely silent about the other pathways. Covariate balance asks whether patients at different distances look alike on the characteristics somebody measured, leaving the unmeasured ones exactly where they were. Placebo outcomes probe whether distance reaches results it should have no business touching. Swapping in a different instrument shows whether a different set of responders gives a different answer. Each of these can expose a broken instrument. None of them can certify a working one. And each of them runs inside the study data, which is the whole point of that 6%.

Strip the machinery away and this is what an instrumental-variable design does. Something external nudges people into or out of treatment, for reasons unconnected to how they would have fared either way. The analysis keeps only the part of treatment that the nudge explains. The rest is discarded, including the part contaminated by people's own reasons for seeking treatment.

That much on its own is not enough, and there is a proof. Imbens and Angrist opened their 1994 Econometrica paper by announcing the negative result: “First we show that the existence of valid instruments is not sufficient to identify any meaningful average treatment effect.” What rescues it is an extra condition on the relation between the instrument and participation status — monotonicity. Add it, and what is identified is an average effect only for those who can be induced to change their participation status by changing the instrument. The standard IV estimator is a weighted average of such local average treatment effects. Not the sickest. Not the average patient. The movable ones. That restriction is what the word local means.

Two years later the conditions got numbers. A 1996 paper in the Journal of the American Statistical Association set them out as a numbered list, its fifth assumption labelled, in its own text, “Assumption 5: Monotonicity (Imbens and Angrist 1994)”. Its contribution was to split the econometrician's single “uncorrelated with the error” condition into two things that fail differently: an exclusion restriction and an ignorable-assignment condition. And it priced the failure. Without exclusion, the IV estimand equals the local average treatment effect plus the average direct effect of the instrument on noncompliers, multiplied by the odds of noncompliance. Drop the assumptions altogether and there is no effect left to read: “Without these assumptions, the IV estimand is simply the ratio of intention-to-treat causal estimands with no interpretation as an average causal effect.”

That ratio is worth seeing with real numbers. The illustration is the Vietnam draft lottery. Take white men born in 1950, with the draft-eligibility cutoff at random sequence number 195. The reduced-form difference in the probability of civilian death was .0009 (SE .0006). The first stage of draft eligibility on veteran status was .1593 (SE .0401). The IV estimate was .0056 (SE .0040). Divide the first by the second and you have the third. Everything the design is arguing about lives in whether that division is allowed.

Only the first condition is a property of the data. The other three are claims about the world — about this instrument in this setting, not instruments in general. A correlation with treatment strong enough to satisfy any referee still says nothing about the other pathways. That is exactly how a differential-distance design gets as far as it does.

Laid out below, a design becomes a sequence of defences rather than a sequence of computations. Name the instrument and the mechanism you claim it works by — differential distance to a catheterization hospital, draft eligibility at random sequence number 195. Then establish the first stage. That is the single step the data can settle on its own, and it comes with a number: .1593 (SE .0401) for draft eligibility on veteran status. Defend independence, which means saying why nothing that shapes outcomes also shapes who gets nudged. Defend exclusion. That is the step the 1996 paper separated out, and the step the 2014 review found unguarded in all 114 studies it examined. Read the effect last, and read it as local.

The decision points sit between the stages, not at the end. Every arrow marks a place where a design that has already passed the previous test can still fail this one.

What is missing from the studies that never looked is not a statistic. It is a document.

Give each assumption its own page. Relevance is the easy page: show the first stage and say how strong it is — .1593 with a standard error of .0401, and the F statistic beside it. Independence needs the story of how the instrument came to vary at all. Who or what set it, and could anything that shapes outcomes have shaped that? A lottery number drawn for men born in 1950 has a very short version of this story. A patient's distance from a catheterization hospital has a long one. Exclusion needs the hardest page: every route from instrument to outcome that does not pass through treatment, listed, with an argument for why each one is small or absent. An assumption whose violations you cannot enumerate is an assumption you cannot defend. Monotonicity needs you to name whoever might respond backwards and say why they are rare or impossible — the condition numbered fifth in 1996 and credited back to 1994.

Then try to break it, and break it from outside. The most damning number in the 2014 review is not that potential instrument-outcome confounders existed in all 114 designs. It is that only 4 of 187 studies, 6%, looked for them anywhere but in their own data. Covariate balance, placebo outcomes and overidentification tests all run on the same rows that produced the estimate. So for each page of the dossier, ask what evidence would show it wrong. Go and find that evidence where the study is not looking. Write down what happened when it did not appear. A dossier that collects only supporting evidence is advocacy with footnotes.

There is a way out, and it costs nothing except ambition: report what the instrument itself did, and say so.

The Oregon Health Insurance Experiment did both in public. Oregon reopened OHP Standard, its Medicaid programme for low-income adults, in January 2008, with a budget for about 10,000 additional enrollees. The window stayed open five weeks, 28 January to 29 February 2008. Sign-ups reached 89,824. Eight roughly equal lottery drawings, staggered from March through September 2008, selected 35,169 individuals, representing 29,664 unique households. The lottery is the instrument, and its own effect is reported first. The abstract of the 2011 paper states it: “In the year after random assignment, the treatment group selected by the lottery was about 25 percentage points more likely to have insurance than the control group that was not selected.” That is a first stage printed as a result, not buried as a diagnostic.

The rescaled number sits beside it rather than replacing it. The clinical results appeared in the New England Journal of Medicine in 2013: 6,387 lottery-selected adults compared with 5,842 not selected, and Medicaid coverage cutting positive depression screening by 9.15 percentage points (95% CI -16.70 to -1.60; P=0.02). A reader can see the 25 points of coverage the lottery moved and the 9.15 points attributed to coverage. A reader can then decide whether the division was earned. Virtually all of it was prespecified. The analysis plan was archived on 3 December 2010, before the outcomes were known.

Now take the harder case. A differential-distance design has a credible assignment variable and a contested exclusion restriction. Those are different failures. The effect of distance itself on patient outcomes remains estimable, and remains worth knowing, because distance is something policy can act on by moving services or by paying for travel. That study's own 0.6 percentage points of extra acute mortality for rural residence, net of access, is precisely such a number. Reporting that reduced-form effect, and saying plainly that it bundles several pathways, is a defensible result. Rescaling it into an effect of the procedure is a further claim that has to be argued for.

The same restraint applies when the assumptions do hold. State the estimand you actually identified, the effect among the people the instrument moved, and resist carrying it to populations the instrument never touched.