Staggered Adoption and Event-Study Designs

The design is the one everyone runs. Units adopt a policy in different years. You fit a two-way fixed-effects regression with a full set of leads and lags. Then you read one coefficient off the output and call it the effect of the policy.

That coefficient is a weighted average of every possible 2x2 difference-in-differences comparison in the data. A decomposition theorem circulated in 2018 and published in 2021 showed what it is made of. Goodman-Bacon replicated Stevenson and Wolfers' study of unilateral divorce and female suicide. Behind the single number that gets reported sit 156 distinct 2x2 components.

Those components do not agree with each other. The reported coefficient is -3.08 suicides per million women (s.e. 1.27). The comparisons that put treated states against untreated states average -5.33 and -7.04. The comparisons that use later-treated states as the treated group and earlier-treated states as the control average +3.51. That is the opposite sign. Strip out those contaminated later-versus-earlier terms and the estimate becomes -5.44. Drop the untreated states instead, so only already-treated states are left to serve as controls, and it flips to +2.42 (s.e. 1.81). Same data, same regression, opposite conclusion.

The paper states the mechanism plainly: “I also explain why the negative weights occur: when already-treated units act as controls, changes in their treatment effects over time get subtracted from the DD estimate.”

This is not a rare pathology. A 2022 survey in the Journal of Financial Economics counted 744 difference-in-differences papers in five finance and five accounting journals, published between 2000 and 2019. Of those, 407 (54.7%) use staggered designs. And 394 of the 407 (97%) appeared from 2010 onward. The abstract does not hedge: “We explain when and how staggered difference-in-differences regression estimators, commonly applied to assess the impact of policy changes, are biased.”

The same authors re-estimated a 2010 bank-branching-deregulation result on 1,519 state-year observations. The two-way fixed-effects coefficient on log Gini is -0.022 (s.e. 0.008), significant at 1%. The Callaway–Sant'Anna estimator gives 0.001 (s.e. 0.007). A stacked regression using not-yet-treated controls gives 0.000 (s.e. 0.005). A published, significant finding becomes a zero once the comparisons are cleaned. Nothing in the original output said so.

Everything that went wrong in the divorce-and-suicide regression and in the bank-deregulation regression went wrong before any plot existed. The order of the work matters more than any individual step.

Define the cohorts first, by adoption date, so you know who is being compared with whom. Choose the controls second: period by period, decide which units are still untreated at that moment. Never-treated or not-yet-treated, never already-treated. Estimate the group-time effects third, one number per cohort per period, nothing averaged yet. Inspect the dynamics fourth, watching how those numbers move as time since adoption grows. Aggregate last, and deliberately: pick the rule, say what it is, show what it absorbed.

Worked in that order, the assumptions and the decision points stay where a reader can see them. Worked in the usual order, they are buried inside a coefficient. In the bank-deregulation re-estimation, that was the difference between reporting -0.022 at the 1% level and reporting 0.001.

When units adopt at different times, there is no single number waiting to be recovered. The effect can differ by cohort, because the units that moved first were not the units that moved last. It can differ by duration: a policy in its first year is not the same policy in its fifth. A single two-way fixed-effects coefficient stirs the clean comparisons together with the contaminated ones and reports the mixture.

The mixture has a formal description. The coefficient is a weighted sum of group-period average treatment effects, and those weights can be negative. That was proved in the American Economic Review in 2020, by de Chaisemartin and D'Haultfœuille. They name the reason: “However, the “control group” in some of those comparisons may be treated at both periods. Then, its treatment effect at the second period gets differenced out by the DID, hence the negative weights.”

The negative weights are countable. In the newspaper data from 2011, 40% of the weights attached to the fixed-effects coefficient are negative, and 46% of those attached to the first-difference coefficient are negative. In the union data from 1998, the two-way fixed-effects union wage premium is 0.107 (s.e. 0.030). Their heterogeneity-robust DID_M estimator gives 0.041 (s.e. 0.034), a difference significant at t = 2.60. The headline is roughly two and a half times the estimate that uses only admissible comparisons.

The repair is to stop averaging first. Estimate the effect for each cohort in each period against a comparison group that is genuinely untreated at that moment. Then combine those estimates by event time, by cohort, by calendar time, or by whatever weights the policy question actually calls for. Three different things get drawn as the same picture. One regression with relative-time indicators. Relative-time effects estimated separately by cohort, so the cohorts at least stay visible. And ATT(g,t) estimated directly, cell by cell, using only eligible controls. The display can look nearly identical in all three cases. The comparisons underneath are not the same comparisons at all.

Much of the confusion in a staggered design comes from the word time doing two jobs at once: the calendar, and the distance from a unit's own adoption date. Four terms hold those apart. The 156 components and the ATT(g,t) cells are both built out of them.

A finished event-study line looks like a measurement, drawn at constant thickness from end to end. It is an aggregation of many cohort and calendar comparisons. A handful of choices made along the way decide what it says.

One of those choices is not even visible as a choice. When a panel contains no never-treated units, the entire path of fully dynamic event-study coefficients is not point-identified. For any constant κ, the shifted path {τ_h + κ(h+1)} fits the data exactly as well once the fixed effects are adjusted. That is Proposition 1 of a 2024 paper in the Review of Economic Studies, and Borusyak and his co-authors state it flatly: “We show that those specifications are under-identified if there is no never-treated group”. Any straight-line tilt of the curve on the page is observationally equivalent to the curve on the page. A rising line and a flat line can be the same estimate.

Start from the group-time cells rather than one omnibus regression. Lay out a table with a row for each cohort and a column for each period. Fill a cell only where an admissible untreated comparison genuinely exists. Callaway and Sant'Anna's minimum-wage application reports seven such cells. The decomposition of a single divorce-law coefficient unpacks 156 comparisons, most of them of a kind you would never have chosen deliberately.

Look at that table before plotting anything. The empty cells are information. They mark the places where a two-way fixed-effects regression goes looking for a control and settles for an early adopter. That substitution is what turned -3.08 into +2.42 once the untreated states were removed. If the table is mostly holes, the curve you were about to draw would have been filling them in for you.

Report the cohort-time estimates and say which rule combines them. Do it before presenting one average curve, not in a footnote after it. The paper that defined ATT(g,t) treats the summary as a separate, stated act. Callaway and Sant'Anna: “We also propose different aggregation schemes that can be used to highlight treatment effect heterogeneity across different dimensions as well as to summarize the overall effect of participating in the treatment.”

Their own application makes the stakes arithmetic rather than rhetorical: state minimum-wage increases and county teen employment, 2,284 counties across 29 states, 2001–2007. The seven ATT(g,t) estimates range from 2.3% to 13.6% lower teen employment. From those identical cells, a simple group-size average returns -5.2%. Their overall treatment-participation aggregation returns -3.9%. A two-way fixed-effects post-treatment dummy returns -3.7%. Nothing about the data changed between those three numbers. Only the rule did. A reader who knows the rule can tell whether an average is answering their question or a different one.

Then separate two things that a single line runs together. An effect that genuinely changes as a policy matures is one. A curve that changes because the cohorts contributing to each event-time point changed is the other. The 2004 cohort in that minimum-wage data is Illinois, and it shows -3.4% in 2004, -7.1% in 2005, -12.5% in 2006 and -13.6% in 2007. That is real growth within one cohort. It is also exactly the pattern that, averaged carelessly across cohorts with different exposure lengths, can be manufactured out of nothing.

Where no clean comparison group exists for the later periods, stop the curve there. A line that ends early is a finding. A line extended to look complete is a story.