Online Experiments, Guardrails, and Long-Term Value

At LinkedIn, about one experiment in ten used to come back with a number the team could not read as the effect of the change. LinkedIn published that in 2018, in a paper on catching biased experiments automatically. The sentence is theirs: "At LinkedIn, about 10% of our triggered experiments used to suffer from bias." One in ten, on a platform built on recommendation.

Two other numbers explain why detection had to become automatic. Metrics computed per experiment had more than quadrupled, from 1,000 to 4,500. Nobody reads 4,500 metrics off a dashboard. The paper also names where novelty bites hardest. Recommendation algorithms with a limited candidate pool, such as People You May Know, are among the three experiment types most prone to novelty effect. A recommender with few candidates to give is exactly that surface. The first week measures curiosity. The sixth measures habit.

Report effects by user state, market, provider, inventory, activity, and cold-start condition. A positive average can hide a large loss for the users who rely most on the surface. At eBay, the heavy users who account for most of the ad spend were the ones dragging the true return down. The aggregate still looked healthy. Keep a decision record as well: expected mechanism, power analysis, exclusions, result stability, guardrail tradeoffs, follow-up obligations. Experiments generate evidence. They do not make the release decision.

Randomization can identify incremental effects when assignment, exposure, outcome, and interference are handled correctly. It also needs the treatment defined as everything that changed with it: the model, the candidates, the rules, the layout, the latency, and any change in logging or fallback. Recommendation experiments often violate the simple assumptions — shared inventory, seller behavior, social effects, repeated users, learning policies. The unit and the horizon must match the mechanism being tested.

eBay ran a four-week email experiment in July and August 2013, "Bid Item Ending Soon", across 4.9 million qualifying U.S. users and 10,425,390 auctions. Randomized and read at the user, the campaign showed roughly 0.74% extra revenue. Read at the auction, the estimate implied about 0.35%. The auction is the unit where treated and control bidders were competing against each other. Blake and Coey drew the conclusion themselves: "This suggests that the user level estimate of 0.74% is an overstatement of the true effect by a factor of over two."

Nothing was broken. The 0.74% is a clean number about a well-run experiment. It is just an answer to a question about users, in a market where the effect lives in auctions.

Recommender experiments break in ways an ordinary A/B test does not. The randomization unit may not match the unit that experiences the effect. The label may mature after the test has stopped. The five stages below — randomization unit, treatment integrity, metric hierarchy, power and duration, decision and rollback — are where that goes wrong. The first of them has a measured price.

Airbnb increased a platform fee, and a meta-experiment was built around it: the same evaluation run by randomizing individuals, then by randomizing clusters of similar listings. Management Science published the result in 2024. "Results from our meta-experiment indicate that at least 20% of the TATE estimate produced by an individual-level randomized evaluation of the platform fee increase we study is attributable to interference bias and eliminated through the use of cluster randomization." At least a fifth of the headline number was interference, not effect. Clustering is not a methodological nicety on that surface. It moves the number by at least 20%, and the individual-level design gives no sign that it should have.

Predeclaring the decision and the rollback is what keeps the readout from being negotiated afterwards.

Two of these failures have been counted rather than warned about. Approximately 6% of experiments at Microsoft exhibit a sample ratio mismatch: the arms did not get the share of users the design called for. That figure was published in 2019 by Microsoft, Booking.com and Outreach.io, drawing on over 10,000 historical online controlled experiments across four companies and more than 25 products. A product running ten thousand experiments a year can expect at least one SRM per day. They treat it as fatal rather than cosmetic for a stated reason: "In particular, SRMs cause a selection bias that invalidates any causal inference that could be drawn from the experiment." The threshold is tighter than intuition allows. Their worked example is a 50.2/49.8 split, 821,588 users against 815,482. The probability that it happened by chance is less than 1 in 500k.

Peeking has a number too. Optimizely's Stats Engine is built on a 2015 paper on always-valid inference, and that paper's introduction prices the habit: "Even with 10,000 samples (a sample size which is quite common in online A/B testing), Type I error can easily increase fivefold." A nominal 5% false-positive rate behaving like roughly 25%. That is not a discipline problem to be solved with a reminder not to look. The authors' remedy shipped: the methodology "has been implemented in a large scale commercial A/B testing platform to analyze hundreds of thousands of experiments to date".

Every changed component belongs in the treatment definition, fallbacks included, rather than the headline model swap alone. The five steps run in order. Define the treatment, choose the unit and the population, verify the instrumentation, monitor a metric hierarchy. Then extend the horizon: persistence, adaptation, retention, supply response.

Step five is the one teams say they cannot afford. There is a named method for buying it cheaper, and it arrives with a named assumption. A multi-site California job training experiment tracked employment for nine years. Athey and colleagues showed the answer had been available much earlier: "Rather than waiting a full nine years to directly observe the long-term impact, we show that it is possible to use short-term (the first six quarters) outcomes as surrogates." Those first six quarters, combined into a surrogate index, could have estimated the nine-year employment effect, with a 35% reduction in standard errors. That holds if the Prentice surrogacy assumption does.

That proviso is the whole trade. Short proxies substitute for a long outcome only if they carry all of the treatment's effect on it. That is an assumption about the mechanism, not a property of the data. Extending the horizon means either waiting, or defending that assumption in writing.