Product A/B Testing and Instrumentation

Two checkout buttons went into a test and the new one won: more of the people who saw it finished the purchase. The team wrote the color into the design guidelines, closed the ticket, and shipped it everywhere.

The variant had also been built on top of a caching change that had nothing to do with buttons. It loaded about 200 ms faster. That rider is not a rounding error, and its price has been measured on purpose. Bing ran a deliberate slowdown and published the result in 2013: "We recently ran a slowdown experiment where we slowed 10% of users by 100msec (milliseconds) and another 10% by 250msec for two weeks. The results showed that performance absolutely matters a lot today: every 100msec improves revenue by 0.6%." At that rate the 200 ms the button rode in on is worth roughly 1.2% of revenue by itself. Google ran its own delay experiment in 2009 and found the same shape: 100–400 ms of added latency cost −0.2% to −0.6% searches per user.

So the comparison that actually ran was between the old checkout and the new checkout — button, cache and all. Nowhere in the output is there a line that separates the two. The color may well have helped. The experiment never asked about the color. It asked about the bundle, and it answered honestly about the bundle.

That gap between what a report says was tested and what was actually shipped is the subject of this lesson. It opens at four named places.

The button test went wrong before any statistics ran. It went wrong at the point where somebody decided what would ship together.

The most repeated small-change-huge-effect story in the field is itself an instance of that mistake. Google raised its search results from 10 to 30. Traffic and revenue in the experimental group dropped by 20%. Marissa Mayer blamed the extra half-second the longer page took to generate. Bing's experimenters took the attribution apart in 2014: "Slowdown experiments that ran at Bing [11] isolated just the performance factor. Numbers showed that a 250msec delay at the server impacts revenue at about 1.5% and clickthrough-rate by 0.25%. While this is a massive impact, 500msec would impact revenue about 3% not 20%, and clickthrough-rate would drop by 0.50%, not 20%". Google's own figures point the same way: 100–400 ms bought only −0.2% to −0.6% searches per user. The 20% was a real number from a real experiment. The half-second was a story told about a bundle — more results and more latency shipped together. Two independent isolated slowdown experiments say the latency cannot carry it.

A trustworthy test settles eight things in writing before traffic moves. What gets randomized — a user, a session, a device, an entire account. Which event makes that unit eligible. How long its arm persists. What counts as having been exposed. Which single outcome decides the question. How long after the trigger that outcome still counts. Which guardrails must not degrade whatever happens. And who ends up in the analysis at the end.

Instrumentation is not a layer underneath those definitions. It is those definitions. Assigned means whatever the bucketing code writes down. Converted means whatever the logging code fires on. If the document and the code disagree, the code wins, because the code is what produced the rows.

No estimator repairs that layer from above. Bucketing that drifts between releases. Exposures that never log. Bots counted as customers. One person arriving as three devices. A metric computed one way in one arm and another way in the other. Put a correct statistical test on top of any of that and you get a number with a confidence interval and no meaning.

Everyone assigned. Everyone exposed. Everyone who engaged. Those are three different groups of people, and analysts slide between them without announcing it, almost always in the direction that makes the numbers look cleaner.

Randomization guarantees the first one and nothing else. Exposure is narrower and can be defended, but only if the treatment has no way of changing who gets recorded as exposed. A variant that renders faster, or later, or not at all on old devices changes exactly that.

Engaged users are the trap. Clicking, opening and scrolling deeper are not filters. They are outcomes, and the variant is allowed to move them. Cut the analysis down to engaged users and you are comparing people who chose to engage under one experience against people who chose to engage under a different one. The random assignment is gone at that point. What is left is an observational comparison of two self-selected crowds, presented in the review as an experiment.

MSN has published what that costs when nobody chooses it on purpose. An experiment there raised a rotating carousel from 12 cards to 16. It came back with a statistically significant DECREASE in card engagement: on its face, a feature that made things worse. The platform's sample-ratio-mismatch warning is what stopped the shipping decision. Seven researchers published the diagnosis in 2019: "A deep-dive analysis by experienced analysts revealed that during the data processing phase, the most engaged users in the treatment group were being classified as computer bots and removed from the experiment analysis by an algorithm that scans for bot activity. Some users in the treatment engaged with the carousel cards so much that they crossed the engagement threshold used in the bot detection algorithm. After accounting for the bot classification results, the results were flipped and the correct decision to ship the feature was made."

Nobody in that room decided to filter on engagement. A bot detector thresholding on engagement did it for them, downstream, in the data processing phase. The mechanism is identical to the engaged-user cut: a filter keyed on something the treatment itself moves removes treatment users preferentially. Here it was strong enough to reverse the sign of the answer.

They come up in every launch review, and a good share of the disagreements in that room are two people using one word for two different things.

Trigger, assign, expose, measure, decide. The map lays those five in one line so the joints between them are visible. Every failure in this lesson lives at a joint rather than inside a stage.

The first joint has been documented down to the line of code. LinkedIn published the base rate in 2018: "At LinkedIn, about 10% of our triggered experiments used to suffer from bias." One of its cross-promotion tests shows the shape of it. The variant-evaluation tracking event fired AFTER the cool-off check rather than before it. Treatment users who had been cooled off were never enrolled at all. The arms reached the analysis in the wrong proportion — a severe sample size ratio mismatch produced by nothing but the order of two statements. The repair is to "exchange lines (1) and (3)" so the trigger fires before the branch.

The checkout button leaked between assign and expose: the arm carried more than the arm was supposed to carry. The MSN carousel leaked between measure and decide, when a bot filter chose the population after the fact and replaced the one that was randomized. Read the line in order. At each arrow, ask what would have to be true for the next stage to inherit a clean comparison — and what, in the logs, would show you that it did not.

None of these tell you whether the change worked. They tell you whether the machine that answers that question is currently working. That is why they run first, and why an unresolved one stops the analysis rather than annotating it.

They also have a price, because an experiment does not stop when you stop it. Bing reported a case in 2012: a 7-day A/A period, then a 47-day A/B, then three-plus weeks of continued monitoring. The carryover effect on the OEC — sessions per user — only died out around the third week after the experiment had ended. In a second case a bug had exposed users to a very bad experience. Three months later those buckets had still not recovered to their pre-experiment levels. Google measured the same persistence from the other side: users given a 400 ms delay for six weeks went on doing 0.21% fewer searches for the five weeks after it was removed. A bucket that has been through an experiment is not clean again on the day the flag flips. That is what a failed A/A really costs you — the capacity of those buckets until you re-randomize.

Everything the contract fixes is cheap to fix before traffic moves and expensive to argue about afterwards. Afterwards each choice has a number attached to it, and everyone can already see which choice pays.

Medicine has measured how strong that pressure is. It is the one field where freezing the primary outcome is mandatory, pre-registration is required, and the CONSORT standard is endorsed by all five journals involved. The COMPare study checked every trial published over six weeks in the NEJM, Lancet, JAMA, BMJ and Annals of Internal Medicine. Sixty-seven trials. Its 2019 report: "Outcome reporting was poor overall and there was wide variation between journals on pre-specified primary outcomes (mean 76% correctly reported, journal range 25-96%), secondary outcomes (mean 55%, range 31-72%), and number of undeclared additional outcomes per trial (mean 5.4, range 2.9-8.3). Fifty-eight trials had discrepancies requiring a correction letter (87%, journal range 67-100%)." Only 43% of those trials even had a pre-trial protocol publicly available. If a registered, mandated primary outcome still drifts by a mean of 5.4 undeclared extra outcomes per trial, a primary metric that lives only in a slide deck is not fixed at all.

So write it down first. The unit of randomization. The trigger. How long the arm persists. What counts as exposure. The one primary outcome. The window in which that outcome still counts. The guardrails. The population that will be analyzed. Write it somewhere timestamped that you cannot quietly revise. Then write down the two lines that get skipped: the result that would make you ship, and the result that would make you stop.

Those last two are why the other eight are worth the meeting.

Start with the sentence the experiment can actually support: assigning the complete shipped variant to the eligible population, under the traffic and the time period that were tested, moved the primary metric by this much. Every clause in that sentence is load-bearing. Drop the word complete and you are back with the checkout button, or with 10 results against 30.

Make the product decision on assignment, not on who turned out to be interested. That is not a house preference. It is written into ICH E9, the statistical guideline every major drug regulator adopted, in force since 1998. Its §5.2.1 says that "The intention-to-treat (see Glossary) principle implies that the primary analysis should include all randomised subjects." Then §5.2.2 turns to the narrower per-protocol set and names the reason the tempting population is dangerous: "The use of the per protocol set may maximise the opportunity for a new treatment to show additional efficacy in the analysis, and most closely reflects the scientific model underlying the protocol. However, the corresponding test of the hypothesis and estimate of the treatment effect may or may not be conservative depending on the trial; the bias, which may be severe, arises from the fact that adherence to the study protocol may be related to treatment and outcome." Read adherence as engagement. That is the engaged-user filter described exactly, in a numbered regulatory instrument from 1998.

If exposure is genuinely required, define it operationally and then check whether the treatment could have influenced who gets observed as exposed. If it could, exposure is an outcome wearing a filter's clothes.

Breaking the result down by latency, by reach, by exposure is how you explain the mechanism, and it is worth doing. It is not permitted to become the new headline. A primary metric redefined after the outcome is visible is not a primary metric.

And when instrumentation differs between the arms, or an SRM has not been explained, the causal conclusion waits. The MSN carousel was a shippable feature that the pipeline reported as a significant decrease. Repair the experiment and run it again. A result from a broken pipeline is not a weak result. It is not a result.