Online Controlled Experiments and A/B Testing

An A/B test does not ask whether one model predicts labels better. It asks what happens when eligible units are randomly assigned to different product experiences.

For one week, from 11 to 18 January 2012, Facebook randomly assigned 689,003 users to News Feeds with either less positive or less negative content. The result appeared in PNAS in 2014. “We show, via a massive (N = 689,003) experiment on Facebook, that emotional states can be transferred to others via emotional contagion, leading people to experience the same emotions without their awareness,” the study's three authors wrote in its significance statement. Nobody was surveyed. Nobody was asked. The feed itself was the instrument. The difference between the arms was the difference between two versions of the product.

The same paper is the standard warning about what randomization does not license. It drew a formal editorial expression of concern over informed consent. Cornell, confirming its two co-authors, Jeffrey Hancock and Jamie Guillory, stated that the researchers never saw the content of actual posts. Randomization supports a counterfactual comparison only when assignment, exposure, measurement and analysis follow the design. And only within what the assigned units have agreed to.

Eligibility, random assignment, exposure, outcome measurement, analysis. A break in any link invalidates the comparison, and link two is the one that fails without a visible symptom.

Assignment is not a fresh start. Microsoft ran a 7-day A/A check, then a 47-day A/B experiment, then kept watching the same buckets after the experiment was over. The carryover effect only died out around the third week after the experiment ended. In a second case the buckets had not recovered after three months.

Google measured the same shape from the other side. Jake Brutlag injected server-side delay into search and reported the aftermath in 2009: “Users exposed to the 400 ms delay for six weeks did 0.21% fewer searches on average during the five week period after we stopped injecting the delay.” The delay was gone. The users were not the same users.

That is what an A/A test is for. Microsoft's mitigation was re-randomizing the bucket system between experiments. A unit carries the last experience it was given into its next assignment. The chain has to be rebuilt, not merely re-pointed, before the next comparison is read.

Choosing a convenient row can violate independence, and the size of that violation is measurable rather than rhetorical.

LinkedIn ran a cluster-randomized network A/B test on the Netherlands sub-network: about 6.1 million users, average internal degree 93.0. It split that network into 600 shards, 300 assigned to treatment and 300 to control. They ran for two weeks, after an A/A check. The average treatment effect on social gestures was 0.168 under the SUTVA assumption, which holds that each user's outcome depends only on that user's own assignment. Under exposure models that allow spillover, the same experiment gave 0.264 to 0.714. “The ATE estimates with consideration of network effect are all larger than the estimate under SUTVA,” the 2015 paper reports. The user-level number was not noisy. It was low, and by how much depended entirely on which assumption was being made about the network.

Facebook built its own cluster-randomized framework for the same reason: one “that accounts for interference between experimental units”, with two case studies that “capture significant network effects”. Cluster randomization costs independent units and demands cluster-aware variance. User randomization costs an unknown amount of the effect whenever the users talk to each other.

A two-week read is not a small version of the long-term result. It is the first few percent of a curve whose timescale has been measured.

Sample-size planning requires the baseline variance, randomization unit, expected traffic, target effect, significance policy, and desired power. Designing only for statistical detectability can create tests large enough to find effects too small to justify action.

Traffic is not the only lever, and it is rarely the cheapest. Interleaved evaluation was validated on commercial search logs and published in 2012, measured against the best absolute click metric, Clicks@1. “There is only one pair for which interleaving was less sensitive than Clicks@1. In all others, interleaving was much more sensitive, significantly reducing the number of queries needed by over an order of magnitude,” Chapelle and Joachims report. Airbnb said in 2025 that its interleaving and counterfactual evaluation “increased the sensitivity of experiments by a factor of up to 100 (depending on the approach and metrics) compared to traditional A/B testing”. Same decision, an order of magnitude to a hundredfold less data. The comparison was designed differently.

Variance reduction is the second lever. It has to be specified before the result is read, not chosen after it. Microsoft introduced CUPED in 2013, and the abstract opens with the number: “The results on Bing’s experimentation system are very successful: we can reduce variance by about 50%, effectively achieving the same statistical power with only half of the users, or half the duration.” The conclusion gives the individual experiments — “variance reductions of 45%, 52% and 49% with one week of experiment and one week of pre-experiment data”. ShareChat deployed the same family of adjustments and reported in 2024 that “we can either improve A/B-test confidence in 77% of cases, or can retain the same level of confidence with 30% fewer data points”. Neither lever needs one extra user.

Precommit the design and monitor integrity from launch. Step one is where most of the trouble is avoided: naming the mechanism, the primary outcome, the guardrails and the practical threshold. The obvious metric can move the wrong way and still be correctly measured.

Microsoft published one such case in 2012. “When Bing had a bug in an experiment, which resulted in very poor results being shown to users, two key organizational metrics improved significantly: distinct queries per user went up over 10%, and revenue per user went up over 30%!” Degrading the product raised both headline numbers, because a user who cannot find an answer searches again. Google's mobile ad-load cut ran the other way. Its own 2015 paper reports that “Reducing the mobile ad load strongly improved the user experience but was a substantially short-term revenue negative change; with our work, the long-term revenue impact was shown to be neutral.” One change looked good and was not. The other looked bad and was not.

Neither case is settled by better statistics on the metric that moved. Both are settled by choosing, before launch, which quantity the team has agreed to be steered by. Then validate instrumentation with A/A tests, invariant metrics and sample-ratio monitoring. Analyze effects with intervals and heterogeneity. Decide under a rule written down in advance.