Distributional Effects, Equity, and Causal Reporting

The Moving to Opportunity housing-voucher experiment raised earnings on average. Inside that average, the sign of the effect flips. Children who moved below age 13 earned $3,477 more per year in their mid-twenties — 31% more, against a control mean of $11,270. Children who moved after age 13 had, if anything, negative long-term impacts. Chetty and two colleagues reported the split in the American Economic Review in 2016. Age at move was a group that existed before anyone was assigned anything.

An average taken across all of those children would have been arithmetically correct. It would have carried the younger movers' gain while absorbing the adolescents' loss without a trace. The number is not false. It is answering a narrower question than the one on the table, and nobody reading a one-line recommendation could tell that from the recommendation.

What a release needs is enough structure that a reader can rebuild the claim and find its weakest link without asking the analyst. That is not house style. Clinical trials have had a rule for it since 2019. ICH E9(R1), the addendum on estimands and sensitivity analysis, requires a trial's target of estimation to be fixed in advance. Five named attributes do the fixing: treatment condition, population, variable, intercurrent-event strategy, and population-level summary. Its glossary defines an estimand as “A precise description of the treatment effect reflecting the clinical question posed by the trial objective. It summarises at a population-level what the outcomes would be in the same patients under different treatment conditions being compared.”

An estimand card carries those five attributes. An assumption register lists each identification claim, the evidence behind it, the person who owns it, and how far the answer moves when it fails. A support report says who the data could not speak for: the excluded groups, the tails of the weights, the places where the estimate is extrapolation rather than measurement. A decision memo carries the recommendation, the alternatives that lost, the uncertainty that remains, the monitoring, and the conditions under which the policy stops. Four documents, one job. Let a stranger find the weak link.

The adolescent movers did not vanish from the estimate. They were averaged into it. The abstract puts it in one sentence: “Moving as an adolescent has slightly negative impacts, perhaps because of disruption effects.”

Any mean effect pools people who differ in four ways: how much risk they carried to begin with, how strongly they respond to the treatment, what it costs them to take it up, and whether it reaches them at all. Once those are pooled, the mean reports all four at once and distinguishes none of them.

Distributional analysis is the habit of asking the estimate to come apart again. What happened at the bottom of the outcome distribution rather than at its middle. What happened in the tail, where the losses live. What the effect looked like inside groups fixed before anyone saw a result — age at move, in the case above. And, since the policy spent something real, where the resources moved.

That work is still estimation. It produces more numbers, and none of them says what to do. Deciding that a small decline for one group is acceptable because the total went up is a value judgment. Deciding it is unacceptable is also one. Whose outcomes count, which harms are ruled out whatever they buy, how gains should be spread across people who did not start level: somebody has to make those choices and defend them. Causal inference tells you what each choice would cost. It does not tell you which cost to accept.

This is why a single fairness number never closes the argument. At least three questions hide inside the word, and each has its own documented failure.

Are the groups equally offered the treatment, and equally likely to receive it once offered? A commercial risk algorithm affecting millions of US patients assigned Black and White patients the same scores at different levels of sickness. Obermeyer and colleagues reported it in Science in 2019. Their abstract gives the size of it: “Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.” The failing quantity there was access to the programme, not the effect of the programme.

Do the causal gains differ across the groups that do receive it? In the RECOVERY trial, what dexamethasone did depended on how much respiratory support the patient already needed. Preliminary results were announced on 16 June 2020: 2,104 patients randomised to dexamethasone against 4,321 randomised to usual care. The Chief Investigators' statement reported: “Dexamethasone reduced deaths by one-third in ventilated patients (rate ratio 0.65 [95% confidence interval 0.48 to 0.88]; p=0.0003) and by one fifth in other patients receiving oxygen only (0.80 [0.67 to 0.96]; p=0.0021). There was no benefit among those patients who did not require respiratory support (1.22 [0.86 to 1.75]; p=0.14).” Treat those brackets as the announcement's own wording, not as the intervals to reuse. Six days later the same team's preliminary report gave 0.65, 0.51–0.82; 0.80, 0.70–0.92; and 1.22, 0.93–1.61. The final report in the New England Journal of Medicine gave 0.64 (95% CI 0.51–0.81), 0.82 (0.72–0.94) and 1.19 (0.91–1.55). Same drug, same trial, three pre-specified strata. In the third, the point estimate runs the wrong way.

And is any group carrying a loss you have already said you will not accept? That is the question the adolescent movers pose. No fairness figure quoted alone will show you which of the three you passed and which you failed.

The order of a causal report is not decoration. Decision first, estimate second. A report built the other way round has no way to tell a finding from a rationalization written after the fact. ICH E9(R1) puts the estimand ahead of the analysis for exactly that reason. The target of estimation is fixed in advance, through its five attributes, so the question cannot quietly follow the answer.

So the map starts where the analyst should have started, by naming the decision that has to be made and the estimands that would inform it. Design is documented next: who was compared with whom, and under what identification claim. Then the estimates are reported, the average and the distribution it came from together — the pooled figure and the split by age at move on the same page. Then they are stressed, so a reader can see which conclusions survive when the assumptions are bent and which do not. Only at the end is the decision itself recorded, with its owners and the conditions that would end it.

Read the map at its joints rather than its boxes. Every arrow is a place where an assumption was accepted or a choice was made. Those are the places a reader should be able to push.

The release document is read by someone checking whether the estimate identifies anything, and by someone who has to sign for the consequences. The first wants the design, the claims, and the places they break. The second wants to know who gains, who loses, and what happens when it goes wrong. A memo written only for the first gets waved through by people who could not have caught the problem. A memo written only for the second cannot be proved wrong.

Build it from four pieces, in the order the map sets out. The estimand card leads, and it is not a free-form paragraph. Use the five attributes ICH E9(R1) names: treatment condition, population, variable, intercurrent-event strategy, population-level summary. Everything after the card is an answer to that question and to no other. The assumption register follows, each claim carrying an owner, so that disagreement has somewhere to go. Then the estimates, average and distribution together, never the average alone. A Moving to Opportunity memo that reported only the pooled gain would have been true and useless. Then the support report, the honest list of everyone the data could not speak for. Then the decision memo, where the recommendation finally appears next to the alternatives it beat.

Write the equity part as an argument rather than a metric. Say which of the three questions the policy is being judged against: access, as in the gap from 17.7% to 46.5%; differential gain, as in RECOVERY's 0.65, 0.80 and 1.22; or a loss you have ruled out, as in the adolescent movers. Then say who decided that it would be judged that way.

The instinct on release is to close the gaps and let the result look settled. That is the wrong target. A report nobody can argue with has not resolved the disagreement. It has only moved it somewhere that leaves no record.

Useful disagreement needs the seams left showing. Keep the empirical estimate, the value judgment and the implementation assumption in separate sentences. Then a reader who accepts the estimate can still reject the recommendation, and point to the sentence where the two of you parted. Then show the movement. Which conclusions flip when the target population changes, when the harm constraint is tightened, when a sensitivity parameter is pushed as far as anyone would defend. A conclusion that survives all of that has earned something. A conclusion that flips under a value many people hold is the most useful line in the report.

The decision has to be a record rather than an announcement. For some deployers that is now law rather than taste. Article 27 of the EU's 2024 AI Act obliges public-sector and certain private deployers of high-risk AI systems to perform a fundamental rights impact assessment before deployment. It must name the categories of persons and groups likely to be affected, the specific risks of harm to them, and the human-oversight measures. It must also name, in the words of Article 27(1)(f), “the measures to be taken in the case of the materialisation of those risks, including the arrangements for internal governance and complaint mechanisms.” The result is notified to the market surveillance authority.

So: name the owners. Name the affected groups, the ones the estimate ran against among them, and the channel through which they can come back. Say what data is kept and for how long. Say what is being monitored, and state in advance which reading sends the policy back for redesign or takes it out of service.