Causal Questions, Incrementality, and Policy Effects

A relevance model estimates likely behavior from exposure patterns in the data. A causal or uplift model compares potential outcomes under recommendation against an alternative policy. The highest-propensity user is not necessarily the user with the largest treatment effect.

Causal claims require assumptions, randomization, natural experiments, instruments, or carefully justified observational methods. Feature importance and association are not substitutes, and no quantity of data promotes one into the other. That comparison has been run at a scale which removes the usual excuse. Fifteen U.S. advertising experiments at Facebook — 500 million user-experiment observations, 1.6 billion ad impressions — were set against multiple observational models. The result ran in Marketing Science in 2019: “The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables.” The sample was enormous. The conditioning was extensive. The two answers still did not meet.

A 2015 study put a number on the gap using a real recommender rather than a hypothetical one. It read browsing logs covering 2.1 million users on Amazon.com over a 9 month period. Over 4,000 products hit traffic shocks, and those shocks supplied variation the logs alone cannot give — an instrumental-variable design. The finding, in the authors' own words: “We find that although recommendation click-throughs do account for a large fraction of traffic among these products, at least 75% of this activity would likely occur in the absence of recommendations.”

Both halves matter. The recommender really did carry a large fraction of the traffic, so every dashboard fed by click attribution was reporting something true. It was not reporting persuasion. Most of that activity survives the removal of the recommendation.

A causal question is well posed only once all five rows are filled: treatment, comparator, outcome, assignment mechanism, target population. The comparator is the row that gets skipped. Recommended instead of what — nothing at all, another item, another slate, or the same item a day later?

Treatment, comparator, population, outcome, and horizon: the estimand is written in those five parts before any data is touched. Policy value is confirmed at the end against capacity, displacement, guardrails, and long-term effects. Estimating the effect is the part in the middle. It is also the part that fails quietly when support and balance were never checked.

Step two is more physical than it sounds. Criteo released a public uplift benchmark in 2018 and describes how it was built in plain operational terms: “This dataset is constructed by assembling data resulting from several incrementality tests, a particular randomized trial procedure where a random part of the population is prevented from being targeted by advertising.” The current release holds 13,979,592 rows: 12 anonymized features, a binary treatment indicator, visit and conversion labels, a treatment ratio of 0.85. That is 210 times the size of the datasets available before it. Creating variation means deliberately withholding the treatment from part of the population, at roughly 14 million rows of it.

Step four has a name and a price. Honest estimation splits the sample: one part builds the partition, another estimates the treatment effects. Athey and Imbens introduced it in 2016 and reported what the split buys: “Through a simulation study, we show that for our preferred method honest estimation results in nominal coverage for 90% confidence intervals, whereas coverage ranges between 74% and 84% for nonhonest approaches.” Splitting the sample costs 7-22% in mean squared error. Not splitting it costs the interval its meaning. Meaning was the only thing the interval was there to provide.

Association as effect is a mistake about what the data showed. Short-horizon uplift is a mistake about when to stop looking. High purchase propensity gets labelled persuasion. An immediate gain gets reported before regret, churn or displacement has had time to appear.

The second of those has been measured and acted on. Google built an experiment methodology for long-term user learning in its search ads, and said why: “However, the short-term effect is not always predictive of the long-term effect, i.e., the final impact once the product has fully launched and users have changed their behavior in response.” The 2015 paper applies the method to two launches. One of them was a 50% reduction of the ad load on Google's mobile search interface. A short-window uplift number would have forbidden that decision.

Usefulness and incrementality are different questions. A recommendation may be useful even if it does not create a new purchase: it can reduce search cost, prevent mistakes, improve timing, or support informed choice. Choose the estimand that matches the intended product benefit. When evidence is weak, state the result as predictive or associational. Downgrading the claim is better than laundering uncertainty through causal vocabulary.

Weak has a size, and it is larger than most teams assume. Twenty-five large field experiments, representing $2.8 million in digital advertising expenditure, were analysed by Lewis and Rao. Their finding ran in the Quarterly Journal of Economics in 2015: “The median confidence interval on return on investment is over 100 percentage points wide.” Individual-level sales are volatile — a coefficient of variation of 10 is common. An informative experiment can therefore require more than 10 million person-weeks. A team that has not bought that much data is not in a position to report a return. It is in a position to report an association. Saying so is the accurate move, not the modest one.