Counterfactual Evaluation, Propensity, and Position Bias

Off-policy evaluation needs five facts about every past decision. The context available before the choice. The action or slate actually displayed. The probability the logging policy gave that choice. The observed reward under a defined window. And the probability the proposed policy would assign to the same action. Two of them are usually missing. They are the last two.

The logs that do carry them carry them because someone paid for them in advance. The data behind the replay method, published in 2011, came from a "random bucket" on the Yahoo! Front Page Today Module, running between 1 and 10 November 2009. Articles were drawn uniformly from the pool. So the logging propensity was fixed by the design, not reconstructed afterwards from serving code. That is the shape of a record that supports counterfactual estimation. It is also a decision that has to be taken before the data exists, not after.

Off-policy estimators ask how a target policy might have performed using data generated by another policy. Inverse propensity scoring reweights observed rewards by target-to-logging probability ratios. Direct methods predict rewards; doubly robust methods combine both ideas. The methods require overlap, trustworthy propensities, stable outcome definitions, and controlled variance. They do not recover evidence for actions the logging policy never took.

That last sentence is a theorem, not a caution. Under a support-deficient logger, the bias of the IPS estimator equals the expected reward the target policy places on the actions the logger gave zero probability. That was proved in 2020. The gap between what the estimate says and what the policy would do is exactly the value sitting in the region the logger stayed out of.

What logging propensities cost is documented, in impressions. The replay method was validated on a log of the Yahoo! Front Page Today Module gathered between 1 and 10 November 2009. The paper describes the source plainly: “In the random bucket, articles are randomly selected from the article pool to serve users. There are about 40 million events in the offline evaluation data set, and about 20 articles available in the pool at every moment.”

The evaluator is a rejection rule. It walks the log and keeps an event only when the algorithm under test chooses the same arm that was actually served. Otherwise the event is discarded, and the algorithm never sees it. Under uniform logging an event survives with probability exactly 1/K. A pool of K articles therefore costs a factor-K shrinkage of usable data. With roughly 20 articles live at any moment, about one logged impression in twenty reaches the estimator. The other nineteen were served, paid for and thrown away by the evaluation. Some forty million events is what the unbiased number is made of before that shrinkage, not after it.

This is the honest version of the sentence a lesson usually asserts. Propensities are not metadata that a careful engineer remembers to write down. Here they were bought: ten days of deliberately randomised serving, and a large multiple of the impressions the estimate finally rests on.

Propensity omission is decided by the logging code and position confusion by the analysis, months apart and usually by different people. One stores the chosen action without its selection probability. The other reads a click as relevance without modelling where on the page it happened.

The slate failure has a price tag. Treat a whole page as a single action and IPS needs m^Ω(l) samples for a slate of length l drawn from m items. A 2017 paper measured that and offered a cheaper route. Its pseudoinverse estimator needs about O(l·m/ε²) samples for error ε, under an additive-reward assumption. The paper states the trade directly: “Inspired by work in combinatorial and linear bandits [7, 31, 11], we propose an estimator that makes only a weak assumption about the evaluated metric, while exponentially reducing the data requirements in comparison with IPS.” They tested it on MSLR-WEB30K, over 31K queries. They tested it again on logs from a popular search engine, whose logging policy randomly chose slates of size l = 5 from at most 8 pre-filtered documents: 77 unique queries and 22K examples. The scale is deliberately small. The exponent is what decides whether the page-level question is answerable at all.

Randomized swaps, interleaving, examination models, or carefully designed propensities can help separate attention and relevance. A correction is only ever as credible as the exposure model and the logging behind it. Report intervals and show where the estimators disagree, rather than one point estimate. Large variance or poor overlap is itself a result: it says the historical data cannot safely answer the question being asked of it.

The reason to distrust a raw click was established by watching eyes, not by argument. In 2005 Joachims and four colleagues sat subjects in front of a proxy that silently manipulated Google's result list, and eye-tracked them while they searched. Phase I recruited 34 subjects, 29 of whom gave usable gaze data. Phase II recruited 22, of whom 16 were usable: 6 "normal", 5 "swapped", 5 "reversed". In the swapped condition the top two results traded places, so the second abstract was frequently the more relevant one. Subjects went on clicking link one. The gaze data is what makes this a measurement rather than an interpretation: “It is very interesting that users click substantially more often on the first than on the second link, while they view the corresponding abstract with almost equal frequency.” Reversing the whole list moved the average rank of a clicked document from 2.66 in the "normal" condition to 4.03 in the "reversed" condition. Users saw both abstracts. Rank decided which one they clicked. Anything that reads those clicks as a relevance signal is reading position and calling it preference.

Validate the logger before estimating anything from it: action probabilities, candidate sets, randomization and exclusions all have to be reconstructible after the fact. Then measure overlap, comparing target and logging support before estimating any value. Then inspect the weights. Then triangulate estimators — IPS, self-normalized, direct, doubly robust. Then confirm online under a guarded experiment, because no weighting scheme is by itself evidence of deployment value.

The weight-inspection step has a published test rather than a feeling. The failure it looks for was named in 2015, by Swaminathan and Joachims: “We show that this conventional estimator suffers from a propensity overfitting problem when used for learning over complex hypothesis spaces.” The diagnostic follows from the same conditions that make IPS unbiased. The empirical mean of the propensity weights has expected value exactly 1, so its deviation from 1 flags a bad estimate. For the hypothesis selected by IPS-based POEM with positive losses, that mean weight collapsed to 0.274 on Scene, 0.028 on Yeast, 0.000 on TMC and 0.175 on LYRL. The self-normalised Norm-POEM stayed at 0.981, 0.840, 0.941 and 0.945 on the same four. On TMC, POEM's Hamming loss of 17.305 was worse than random guessing. A mean weight of 0.000 against 0.941 on one dataset is what a failing weight distribution looks like when someone bothers to print it.

Steps three to five have an industrial precedent as well. In 2018 Criteo checked offline estimators against ground truth it actually held: “We have access to a proprietary dataset of 39 online A/B tests, representing a total of few hundreds of billions of recommendations.” They set the capping parameter to c = 100 and dropped the uncapped estimators outright. With IS or NIS, they reported, the confidence interval on the uplift would never support a positive or negative decision — it would always be neutral. A few hundred billion recommendations were not enough to make raw IPS decide anything. That is the lesson's "large variance is itself a result", observed at a scale most teams will never reach.