Interference, Spillovers, and Network Experiments

A platform that wants to know what a fee change does to its hosts can only change the fee for some of them. That is the constraint. The usual response is to assign listings one at a time and compare. On Airbnb, Holtz and four co-authors did something less usual. They randomized the design itself. The same platform fee increase was evaluated twice over: once under Bernoulli randomization, which assigns individual listings, and once under cluster randomization, which assigns whole clusters at once. The two designs could then be set against each other instead of argued about. Management Science published the result in 2024.

The comparison is the finding. The abstract puts it in one sentence: "Results from our meta-experiment indicate that at least 20% of the TATE estimate produced by an individual-level randomized evaluation of the platform fee increase we study is attributable to interference bias and eliminated through the use of cluster randomization." At least a fifth of the individual-level total average treatment effect was not the fee increase. The 2020 preprint of the same meta-experiment put the share at 32.60%. The published number is the conservative one — a floor, not a point estimate.

The mechanism is the ordinary one. A booking that a treated listing loses does not disappear from the platform. It can land on an untreated listing competing for the same guest, and that listing sits in the control group. A loss on one side and a matching arrival on the other push the two arms apart together. The measured gap holds the fee increase plus a transfer. Cluster randomization does not abolish the transfer. It moves most of it inside a cluster, where it no longer stands between the arms being compared. The ≥20% is what that move removed.

Four ideas do the work of pulling such a comparison apart. The rest of the lesson lives on them.

The usual way of writing a causal effect gives each unit two possible outcomes, one under treatment and one under control. That notation has already ruled the Airbnb problem out of existence. There is nowhere in it to record what happened to a listing because of somebody else's fee.

Widen it and a unit's outcome may depend on how many of its competitors were treated. It may depend on how close the treated ones sit in the network, or on how much of the market the treated group holds between them. Treatment becomes the whole pattern of assignments, seen from where that unit stands. The exposure mapping is the short summary of that pattern you are willing to commit to. Committing to one is a design decision, and it comes first.

Three designs commit differently. Cluster randomization assigns whole groups at once, so most of the spilling happens inside a group rather than between the arms being compared. That was the arm of the Airbnb meta-experiment against which the interference bias was measured. Graph-cluster designs cut a network into pieces chosen so that few connections cross between arms. Two-stage saturation randomizes what fraction of each group gets treated, then which members do. Coverage itself becomes something you can estimate an effect against, instead of something you have to assume away.

The saturation design is not a thought experiment. Job placement assistance in France was evaluated with one, and the Quarterly Journal of Economics published it in 2013. Crépon and four co-authors randomized coverage before they randomized people. The abstract states the first stage exactly: "In the first step, the proportions of job seekers to be assigned to treatment (0%, 25%, 50%, 75%, or 100%) were randomly drawn for each of the 235 labor markets (e.g., cities) participating in the experiment." Job seekers within each market were randomized second. The extra stage bought a result no individual-level design could have produced. The eight-month gains for treated youths came partly at the expense of untreated eligible workers, leaving very little net benefit overall. A design that only compared treated to untreated job seekers inside those markets would have reported the gain and never seen where it came from.

Randomizing individuals is still worth doing. What changes is that the result no longer reads as a unit effect unless you also know who nearby was assigned what.

On Airbnb the spilling worked by subtraction. Demand moved between listings, so one arm's gain was the other arm's loss and the measured gap opened wider than the truth. That is one flavor, and not the only one. Where a treatment does people good, it often does some good to the people standing near them as well. The control group improves, the gap narrows, and the experiment understates what a full rollout would deliver. Which way you are wrong follows from how the effect travels. Name the mechanism before you guess at the sign.

The narrowing kind has been measured. Mass deworming in Kenya was randomized across 75 schools rather than across pupils. Miguel and Kremer published the result in Econometrica in 2004. Over 1998–1999 the total effect on school participation in treatment schools was 7.5 percentage points (standard error 2.7). Pupils in untreated comparison schools inside the spillover area gained about 2.0 percentage points (standard error 1.3). Their abstract draws the consequence: "Deworming substantially improved health and school participation among untreated children in both treatment schools and neighboring schools, and these externalities are large enough to justify fully subsidizing treatment." A pupil-level randomization would have differenced a treated child against a classmate who was already benefiting. It would have understated the program by construction.

The spillover is not always the smaller term. On 2 November 2010, the day of the US congressional elections, political mobilization messages were randomized to 61 million Facebook users. Seven researchers reported in Nature in 2012 that "The effect of social transmission on real-world voting was greater than the direct effect of the messages themselves, and nearly all the transmission occurred between 'close friends' who were more likely to have a face-to-face relationship." The quantity most experiments are built to estimate was the smaller half of what the treatment did. The larger half travelled along a specific and narrow part of the graph.

A suspected spillover is a story about a mechanism. Building the experiment means turning that story into an assignment you can actually make and a quantity you can actually estimate — in that order. The order is not the lesson's advice. It is the stated structure of the framework the field uses.

The framework has three parts, and their sequence is part of it. Aronow and Samii set them out in 2017: "The framework integrates three components: (i) an experimental design that defines the probability distribution of treatment assignments, (ii) a mapping that relates experimental treatment assignments to exposures received by units in the experiment, and (iii) estimands that make use of the experiment to answer questions of substantive interest." Design, then exposure mapping, then estimands. The estimand is component (iii). It is the last thing defined, and in practice the last thing anyone can honestly change.

The same paper supplies the machinery that makes the sequence usable: inverse-probability-weighted estimators built on exposure probabilities, and randomization-based variance estimators that account for the clustering interference creates. They apply it to a field experiment on the spread of anti-conflict norms among school students. There the effect travelling between units is the object of study rather than a nuisance.

So start from the mechanism, not from the data you already hold. If you believe the effect travels because units compete for the same customer, the grouping that matters is the set of units that compete, whatever tables happen to exist. Fix that grouping and the exposure mapping follows from it. The design follows from the mapping. Then say in advance which quantity you will report. A direct effect, a spillover effect, and a contrast between two allocation policies are three different numbers. Picking among them after the results are in is how a spillover turns into whichever story is most convenient.

Define the network or groups. Choose the exposure mapping. Select the assignment design. Define the estimands. Use design-aware inference.

They run in that order because each one is written in the vocabulary of the one before it. You cannot say who counts as a neighbor until you have said what a connection is. You cannot choose between cluster, saturation, and graph-cluster assignment until you know which exposure you are trying to vary. And the standard errors at the end have to respect the dependence the design created. Units inside the same cluster are not independent observations, however the software treats them.

The third decision has a published price. A randomized saturation design is a two-stage experiment: randomize a group's treatment saturation, then individual assignment, with partial interference mapped onto a regression model with clustered errors. Baird and three co-authors formalized that in the Review of Economics and Statistics in 2018. They prove the trade-off rather than warning about it. "We show that the power to detect average treatment effects declines precisely with the ability to identify novel treatment and spillover effects." Not roughly, not on average — precisely. Every increment of ability to see the spillover is bought with power on the average effect. A design chosen to answer both questions answers each of them less well than a design chosen to answer one.

The clustered-errors part of that same formalization is where the fifth decision comes from. Partial interference is what makes the errors clustered, and design-aware inference is what respects it. The second decision carries the condition from the graph-cluster work: the estimate is unbiased "provided that the exposure model has been properly specified." Set the five decisions side by side with the assumption each one carries. Then you can see which of them your result is resting on.

Keeping these four apart is most of what it takes to say clearly which quantity you have estimated and which one you have not. They are not the lesson's coinages. Hudgens and Halloran formalized the direct, indirect, total and overall vocabulary in 2008. Their abstract opens by naming the assumption the whole field had been making: "A fundamental assumption usually made in causal inference is that of no interference between individuals (or units); that is, the potential outcomes of one individual are assumed to be unaffected by the treatment assignment of other individuals."

Dropping it, they define the four estimands for a population of groups with within-group interference. They prove that the total causal effect equals the sum of the direct and indirect effects. And they give unbiased estimators under a two-stage randomization that assigns groups first and individuals within groups second. That arithmetic result is what makes the separation more than a naming convention. Total = direct + indirect. So reporting one component alone is not a partial answer. It is a mislabelled one.

What a platform wants to know is what happens if the fee changes for everybody. That is a comparison between two allocation policies, not between a treated listing and an untreated one. It is the usual case once interference is real, and it is why the meta-experiment was worth running. The individual-level design was answering a question about a world in which only some listings had the new fee, and at least 20% of its answer was "attributable to interference bias and eliminated through the use of cluster randomization."

Report the direct and spillover components wherever the design separates them, and then say which coverage level they were measured at. The components only add up to a policy answer at the saturation you actually ran. That is exactly why Crépon and colleagues drew saturations of 0%, 25%, 50%, 75%, or 100% rather than settling on one. The answer at 25% coverage and the answer at 100% coverage are different quantities. Their finding — that gains for treated youths came partly at the expense of untreated eligible workers — is a statement about the range of coverage levels they actually assigned.

And leave them there. A spillover measured in a dense network at low coverage says very little about a sparse one at high coverage. A displacement effect measured in one market will not survive being carried into another. Moving an estimate across that distance takes a model of how the mechanism itself changes with density, size, and coverage. Without one, what you have is a result about the setting you ran it in. That is worth more than a wrong result about everywhere else.