Causal Inference Capstone: Design and Defend an Intervention Study

You will want to hand in a number. The number is the last thing a reviewer looks at, and it is the easiest part of the work.

Look at what a defensible one costs. Remote monitoring after a heart-failure hospitalization has already been through a randomized trial. BEAT-HF, published in JAMA Internal Medicine in 2016, reports an adjusted hazard ratio of 1.03 for 180-day all-cause readmission, 95% CI, 0.88-1.20, P = .74. That line took 1,437 patients hospitalized for heart failure. They were randomized at six California academic medical centers between 12 October 2011 and 30 September 2013. Monitoring of weight, blood pressure, heart rate and symptoms plus nurse health-coaching calls went to 715 of them; usual care to 722. The estimate is one sentence of the abstract. Everything that entitles anyone to read it is the other years.

What you actually hand in is a chain. At one end sits a decision somebody has to make. At the other sits a program that runs, gets watched, and one day gets switched off. In between are the versions of the intervention you actually mean, the quantity that would answer the decision, how people were assigned to it, and how the outcome was measured. Then why this data can identify that quantity at all, how you estimated it, and what the diagnostics said. Then what policy will and will not permit, and who keeps the program honest afterwards. Any one of those links can cap what you are allowed to claim. How careful the rest of it was does not matter.

Which is why honest scope is rewarded here and precision is not. A capstone that ends in redesign the study, we can only bound this, randomize the next round, or do not deploy is stronger work than a tight interval resting on an assumption nobody defended.

A hospital wants to send recently discharged patients home with remote monitoring, hoping to keep them out of the emergency department for the next thirty days. The idea is a good one. The first question is not how to analyse it. It is whether somebody has already run it.

Somebody has. BEAT-HF randomized 1,437 patients hospitalized for heart failure between 12 October 2011 and 30 September 2013, at six California academic medical centers. Monitoring of weight, blood pressure, heart rate and symptoms plus nurse health-coaching calls went to 715 of them; usual care to 722. The result section of the abstract says: “The intervention and usual care groups did not differ significantly in readmissions for any cause 180 days after discharge, which occurred in 50.8% (363 of 715) and 49.2% (355 of 722) of patients, respectively (adjusted hazard ratio, 1.03; 95% CI, 0.88-1.20; P = .74).” There was no significant difference in 30-day readmission either, or in 180-day mortality.

Now read the observational analysis the hospital was about to commission against that. Monitoring has been handed out before, and never at random: the sickest patients and the best-connected ones got it. Monitoring also changes how often a nurse picks up the phone. In BEAT-HF the health-coaching calls were part of the intervention, not a nuisance to be adjusted away. Patients wear the thing diligently at first and less as the weeks pass. And when a patient does turn up at an emergency department across town, the hospital may never hear about it. The outcome goes quietly missing for exactly the people it matters most for. On top of that, there are only enough devices for one-third of the patients who qualify.

Suppose you defeated every one of those threats. You would arrive, expensively, at a quantity that a randomized comparison of 715 patients against 722 has already put at 50.8% versus 49.2%. The estimator was never the binding constraint. What is left to decide is which version of monitoring you mean, which patients you are talking about, whether you can randomize or must emulate a trial, and how the outcome gets measured. Then which patients have comparable counterparts at all, whether the effect differs between them, who gets the scarce devices, and whether that allocation is fair. Then what you watch after launch, and what would make you stop.

The dossier has five stages, and they run in the order the work runs: the question and protocol, the causal structure, the design portfolio, the estimation and diagnostics, then the policy and lifecycle.

A reviewer reads them the other way. They start at the recommendation, ask which decision it changes, and walk back down the chain until they reach a link that will not take their weight. Walk BEAT-HF backwards and every step holds. The conclusion rests on 1.03 (95% CI, 0.88-1.20). That rests on a randomized comparison of 715 patients against 722. That rests on a named intervention and a stated 180-day window, in a population defined by hospitalization for heart failure at six named-in-advance sites.

Build the map so it can be walked in that direction. Write each assumption where it is actually used, and mark each decision point. Do that and you find the weak link before the reviewer does. That is the only version of this that ends well.

Four terms get confused with each other constantly, and they do different jobs.

Each one should be short enough that somebody reads it to the end, and complete enough that somebody else can replay what you did from it. Two of the four have published specifications behind them, which is worth knowing before somebody dismisses them as house style.

The first is one sentence long. Hernán and Robins wrote it in 2016: “Causal inference from large observational databases (big data) can be viewed as an attempt to emulate a randomized experiment—the target experiment or target trial—that would answer the question of interest.” Specify the trial you would run in full protocol detail. Then judge your observational analysis by how well it emulates that trial.

The capstone is not graded on whether the hospital deploys anything. It is graded on whether the ending matches the evidence that was gathered.

Recommend the program as written when three things hold: the quantity you estimated is the one the decision needs, the design identifies it, and the value survives an honest push on the assumptions. Recommend it narrowed when it holds only for part of the population. Deploy where the support exists, and say plainly where it does not. Recommend more evidence first when the weak link is fixable. That usually means randomizing the next expansion, measuring the outcome you have been losing, or reporting bounds instead of a point. Recommend against deploying when the chain breaks somewhere you cannot repair — or when the chain holds perfectly and the answer is simply no.

That fourth ending is in print. Telephone-based interactive-voice-response telemonitoring was tested in Tele-HF, which randomized 1,653 recently hospitalized heart-failure patients: 826 to telemonitoring, 827 to usual care. The primary end point was readmission for any reason or death from any cause within 180 days. It occurred in 52.3% versus 51.5%: a difference of 0.8 percentage points, 95% CI, -4.0 to 5.6, P=0.75. Readmission for any reason was 49.3% versus 47.4%, and death 11.1% versus 11.4%. The authors closed their abstract in the New England Journal of Medicine in 2010 with the ending itself: “Among patients recently hospitalized for heart failure, telemonitoring did not improve outcomes. The results indicate the importance of a thorough, independent evaluation of disease-management strategies before their adoption.”

Only the last of those feels like failure. It is the one the hospital most needs to hear. It is also the hardest to write, because it means having done the work and still saying no.

Give a second team one instruction: attack the link you are proudest of.

Their job is to tell the story in which your effect is not real, using nothing but your own dossier. The sickest patients were the ones who got devices. The nurse calls did the work and the device was along for the ride. The patients who abandoned the device are the ones who ended up in an emergency department you never saw.

Notice what randomizing does to those three. It closes the first outright. It turns the second into a question about what your intervention is defined to include, rather than a bias to be adjusted away — BEAT-HF simply put the calls inside the intervention and said so. It does nothing at all to the third. An emergency visit that never reaches your records is missing from the outcome whether you randomized or not. If the second team can tell that story and your evidence cannot rule it out, that is your result, and you found it in time.

Submit one package, built so that each of its readers can work through it without you in the room and without needing each other.

A scientific reviewer should be able to find the estimand, the assumptions and the diagnostics, and reproduce what you did from them. The standard to aim at is a BEAT-HF-style report, where the intervention, the arms, the sites, the enrollment window and the endpoint can all be read off the first page. A policy owner should be able to find the recommendation, what it costs, who it reaches and what would stop it, without wading through the estimator. A patient, or somebody speaking for them, should be able to find out what was done with their data, what the program does to someone like them, and how to object to it.

The Tele-HF authors asked for thorough, independent evaluation of disease-management strategies before their adoption. Independent evaluation is only possible against a package somebody outside the team can actually read. A section that only makes sense to the person who wrote it is not finished.

Recommend action when all of this holds at once: the quantity you estimated is the one the decision needs, the design credibly identifies it, comparable patients exist across the range you would deploy to, the measurement would survive being checked, and the value still looks worth it after the assumptions are pushed as far as a critic plausibly would push them.

All of it. Not most of it.

That last push has a unit, and refusing to report it is a choice. VanderWeele and Ding introduced the E-value in 2017 for exactly this: “The E-value is defined as the minimum strength of association, on the risk ratio scale, that an unmeasured confounder would need to have with both the treatment and the outcome to fully explain away a specific treatment-outcome association, conditional on the measured covariates.” They propose that in all observational studies intended to produce evidence for causality the E-value be reported, or some other sensitivity analysis be used. They suggest calculating it twice: once for the observed association estimate, after adjustment for measured confounders, and once for the limit of the confidence interval closest to the null. One number for the estimate, one for the edge of the interval. That is the whole cost of showing how hard your finding is to break.

When one of the conditions fails you still have something to hand in, and it is not a hedge. Narrow the target to where the evidence actually lives. Ask for the stronger experiment. Ask for the measurement you were missing. Report bounds rather than a point. Or state that there is no causal claim to be made here yet.

Then write down the part that is almost always left out: who owns the next decision, and what specific evidence would change this one. A recommendation without those can never be wrong, which is another way of saying it can never be checked.