Regression Discontinuity Designs

The scholarship cutoff is not a teaching example. It is where this design came from. An award rule drew a hard line through an otherwise ordinary group of students. The line sat on a test score — a number that varies continuously, for reasons that have nothing to do with the rule. Thistlethwaite and Campbell wrote that up in 1960, in the Journal of Educational Psychology. The method has been cited 1,511 times since per OpenAlex, 1,557 per Semantic Scholar.

A later survey of the method opens with exactly that origin: “The RD design was first introduced by Thistlethwaite and Campbell (1960) in their study of the impact of merit awards on the future academic outcomes (career aspirations, enrollment in post-graduate programs, etc.) of students.” That is Lee and Lemieux. Nobody randomised anything. The administrative rule did the work a randomiser would have done, but only for the students sitting close to the line.

That is also where the design can die, and it dies before any estimate is computed. Suppose the students knew where the line sat. Or the people recording the scores knew. Then a score's position relative to the cutoff can depend on how hard someone pushed. The students just above the line would not be the same kind of people as the students just below. They would be the ones who could push. A jump in later outcomes would be measuring that difference and calling it the award.

Whether it happened is not a matter of opinion. It leaves a signature in the distribution of the score itself. There is a named statistical test for that signature, and a United States federal regulation now requires the test to be run. Both appear later in this lesson.

The design rests on one idea, and it is a modest one. A student who scored a hair below the threshold and a student who scored a hair above it are the same student in everything that matters to their later outcome. Ability, background, ambition, the job market they will graduate into — all of it should drift smoothly as the score rises. Nothing has a reason to lurch at one arbitrary number. The award rule does lurch there. So if the outcome lurches too, the rule is the only candidate left standing.

That argument buys a real effect and a narrow one. It holds for people at the line. It says nothing directly about the applicant who scored far above it, or far below.

When the rule decides outright, the jump in outcomes is the effect of the rule. When the rule only shifts the odds, that jump has to be scaled by the jump in the probability of actually being treated. The number that comes out belongs to a smaller group still: the students whose treatment genuinely turned on which side of the line they landed.

A zoning boundary makes the same argument in physical form. Two houses face each other across a line on a map. One falls under a rule the other escapes, and the difference between them is what the rule did. That comparison is worth exactly as much as the boundary is arbitrary. Real boundaries rarely change one thing at a time. It is the same worry a manipulated test score raises, arriving from a different direction.

The work runs in a fixed order, and the order matters, because each stage can kill the one after it.

First, learn the rule as it was actually applied: who set the threshold, who knew about it, and whether anyone had reason or opportunity to move a score across it. Second, look at the data before modelling it — the distribution of the score, the outcome plotted against it, and the characteristics that were fixed before assignment and should not jump. Third, choose the window around the cutoff you are willing to defend. Wide enough to contain data, narrow enough that the smoothness claim is still believable. Fourth, estimate the size of the discontinuity inside that window. Fifth, try to break it. Move the window. Move the cutoff to a point where no rule exists, and see whether the jump survives.

In the United States that sequence is not a private habit, and not even advice. It is regulation. Since 2024 the Department of Education's rules — 34 CFR 77.2 — have incorporated by reference the What Works Clearinghouse Procedures and Standards Handbook. The handbook applies five RDD standards to any study of this kind. Standard 1 alone demands institutional, statistical and graphical integrity of the forcing variable. The required statistical test must “fail to reject the null hypothesis of continuity in the density of the forcing variable at the 5 percent significance level”, citing McCrary, 2008 — the same test the next sections use. A study is not even eligible for review unless the forcing variable takes at least four unique values on each side of the cutoff. And on the institutional half of the standard, the handbook is blunter than most textbooks: “The study will not satisfy this standard if there is a clear opportunity to manipulate scores AND a clear incentive.”

Four of the five stages are about whether the comparison is allowed. Only one of them produces a number.

Before any estimate is taken seriously, the design has to survive three questions. All three are answered with pictures rather than with a model.

Does the score behave itself around the cutoff? A distribution that passes smoothly through the threshold is consistent with nobody having steered. McCrary turned that into a formal test in 2008 and ran it on two running variables at once, one clean and one dirty: “The methodology is applied to popular elections to the House of Representatives, where sorting is neither expected nor found, and to roll-call voting in the House, where sorting is both expected and found.” In the popular-election vote shares the test found nothing. In roll-call votes from 1857 to 2004 it put the estimated log discontinuity in the density at 50 percent at 52 percent, with a t-ratio of 6.6. The same procedure clears one running variable and convicts the other. That is the only way to know it is testing anything.

Do the characteristics fixed before assignment jump at the cutoff? They should not. If something measured before the award shifts sharply at the line, the two groups already differed before the rule ever applied. The outcome gap then has an explanation the design cannot separate from the treatment.

Does a jump appear where no rule exists? Run the same estimate at a placebo cutoff set a little away from the real one. Barreca and colleagues did this at the 100-g cutoffs between 1,000 g and 3,000 g in birth-weight records, and reported what came back: “Using a bandwidth of 85 g, 37 of 41 placebo estimates indicate that mortality is lower to the left of the cutoff. With a bandwidth of 30 g, 41 of 41 placebo estimates indicate that mortality is better to the left of the cutoff.” In the working-paper version of the same study the count at 30 g is 42 of 42 point estimates below zero.

Forty-one out of forty-one is not a design finding effects at fake thresholds now and then. It is a design that finds one everywhere it looks. And note which way the verdict moved as the window narrowed: from 37 of 41 at 85 g to 41 of 41 at 30 g. It got worse, not better.

The gap between the sharp and the fuzzy case is not bookkeeping. It changes the population the estimate describes.

Admission to a flagship state university is decided by a cutoff, and Hoekstra's 2009 study shows the arithmetic with the intermediate numbers left in. Enrollment did not go from 0 to 1 at the admission cutoff. It jumped by 0.388 (t = 10.57), because some admitted students went elsewhere and some rejected ones did not vanish. The raw earnings discontinuity was 0.095 (z = 3.01). Rescaled by that first-stage jump, it becomes an intent-to-treat effect of 0.135 and an enrollment effect of 0.223 (SE 0.079). Across functional forms and bandwidths the discontinuity ranged from 7.4% to 11.1% — the sensitivity check reported rather than suppressed. The conclusion Hoekstra states is narrower than the headline anyone would write from it: “The results indicate that attending the most selective state university causes earnings to be approximately 20% higher for white men.”

Read that sentence for what it does not say. It does not say 0.095, the number the data show directly. It does not describe applicants far from the cutoff. It does not describe anyone outside the population named in it. Where the rule only nudges, the estimate belongs to the students whose enrollment was actually settled by their side of the line — not the ones who would have attended regardless, not the ones who would have declined either way. That group cannot be listed by name.

The assumptions grow as well. A fuzzy design needs the cutoff to touch the outcome only by changing who gets treated. That is a claim about how the world works, and the data cannot confirm it.

Each of these leaves a visible mark, and each costs the design something different. None of them is caught by the estimate itself, which returns a confident number in all four cases.

The number carries its conditions with it. It is the effect for people near this threshold, under this rule, during the period the data cover, for the population named in the sentence. Change any one of the four and it stops being evidence. Hoekstra's result is not that selective universities raise earnings by 20%. It is that attending the most selective state university raises earnings by approximately 20% for white men, at that admission cutoff, in that period.

Extrapolation is the tempting failure. What a merit award did for students at the line — the question asked in 1960 — says little about the strongest applicants, who may have had other options. It says as little about the weakest, who may have needed more than money. Claiming the effect for them requires an argument about how the effect changes as the score rises. That argument has to come from somewhere other than the discontinuity, because the discontinuity only ever looked at one point.

And if the density test rejects, or the covariates jump, or 41 of 41 placebo cutoffs produce an effect, then the honest description is that outcomes differ across the threshold. That is a fact about the data. It is not yet a fact about the rule. The distance between those two sentences is the entire design.