Interaction Logs, Exposure, and Missing-Not-at-Random Data

A bookseller trained on purchases and treated every unpurchased title as a negative. Most of those titles had never appeared to the user at all. So the model learned the old recommender's exposure pattern as if it were preference.

The way out of that is expensive, and the largest published example prices it. In May 2009 Yahoo!'s Front Page Today Module ran a random bucket. Instead of ranking articles, it drew them at random from the article pool. The point was to make the logged events usable for unbiased offline evaluation. About 4.7 million events came from 1 May alone, for tuning. The evaluation set ran from 3 to 9 May. Over 33 million events in all, and a 12.5% click lift for LinUCB over a context-free bandit.

The 2010 paper reporting that traffic makes the claim this lesson turns on: “we argue that any bandit algorithm can be reliably evaluated offline using previously recorded random traffic.”

Previously recorded random traffic is exactly what the bookseller's log did not contain. No amount of care in splitting that log creates it after the fact.

A log that starts at the click has already started too late. This record starts at the decision context and ends at the logging probability. The fields in between are the ones teams find they needed only after an analysis has failed.

The last box is the one most often left empty, and one public production log filled it. In 2016 Criteo released the first public dataset carrying accurately logged propensities for batch learning from bandit feedback. Over 103 million banner-ad impressions, 35GB gzipped, about 250GB raw on the dataset page and 256GB by the paper's own count. The logging policy sampled banners without replacement from a Plackett-Luce model. For every impression it recorded the probability with which the displayed banner had been sampled.

It also wrote down a second exposure decision that most teams would have taken silently: “Importantly, non-clicked examples were sub-sampled aggressively to reduce the dataset size and only a random 10% sub-sample of non-clicked impressions are logged.”

Because the sub-sampling rate is on the record, an analyst can reason about it. Apply the same thinning without a record and the file simply looks like an ad system whose banners were rarely ignored.

Implicit-feedback logs are not an independent sample of preferences. They record behavior after eligibility, retrieval, ranking, layout, and user choice have already filtered the world. The missing entries are therefore missing-not-at-random. A team that treats them all as equal negatives rewards whatever was already popular and punishes anything the old system never showed. It also scores any policy that resembles the logger too kindly. The log is not a picture of what users want. It is a picture of what the last system chose to show them.

Two of these are worth auditing before the others. When only clicks are stored, non-clicks and non-exposure become the same row forever. When late outcomes overwrite history, an old example quietly acquires information nobody had at recommendation time.

How long that second window stays open is a decision somebody has to write down, and Criteo wrote it down in public. Its attribution dataset covers 30 days of live traffic: 16.5 million impressions, 45,000 conversions, 700 campaigns. Each impression carries a timestamp, a click flag, a conversion flag defined as a conversion within 30 days, and whether the conversion was attributed to Criteo. The 2017 paper releasing it states the reason: “The industry standard for conversion attribution is to credit the advertizing platform which owns the last click in a 30 days window before the conversion.”

A label attached to an impression served today can still change a month from now. A training set assembled with no stated window is one whose labels depend on the day somebody happened to read them.

Three cases, and most logs store only the first. A positive interaction and an item nobody ever saw look identical in a table that never recorded what was on the screen. Between them sits the case carrying most of the information: exposed but unselected, shown and passed over.

Separating the three cannot be done to a log afterwards. It has to be built, and the cost is visible in the datasets built for it. Schnabel and Joachims, with three colleagues, constructed the distinction by hand in 2016. Their Coat Shopping dataset paid 290 Mechanical Turk shoppers to browse an inventory of 300 items: “They were asked to find the coat in the store that they wanted to buy the most. Afterwards, they had to rate 24 of the coats they explored (self-selected) and 16 randomly picked ones on a five-point scale.”

The 24 are exposure the shopper chose. The 16 are exposure nobody chose, and they exist only because the experimenters paid for them. The same paper's other dataset, Yahoo! R3, pairs over 300,000 self-selected ratings from 15,400 users with a test set of 5,400 users each rating 10 randomly chosen songs. That is the going rate for a test set the old policy did not select.

The opportunity is logged before the click is. That ordering is the whole point of the contract, which is why by step five the team is auditing what never appeared at all.

Step three is the one whose payoff is easiest to underrate, because it looks like bookkeeping. A team that recorded which ranker served which impression need not degrade live results to learn its own propensities. Joachims and five colleagues showed that in 2019: “First, we show how to harvest a specific type of intervention data from historic feedback logs of multiple different ranking functions, and show that this data is sufficient for consistent propensity estimation in the position-based model.” They evaluated the estimator on two production systems, ArXiv Full-text Search and Google Drive Search.

The interventions were already in the history. Versioning is what makes them legible. A log that mixed several ranking functions without naming them has thrown that history away.

Step four is where a window of the Criteo kind gets stated explicitly, so that a past example is not quietly updated with a fact that arrived after it was served.