Offline Evaluation Protocols and Temporal Splits

Two leaks matter here, and only one of them is obvious.

The obvious one has been put on a scoreboard. A 2023 study ran BPR, NeuMF, SASRec and LightGCN under a leave-last-one-out split on MovieLens-25M, Yelp, Amazon-music and Amazon-electronic. Its finding: “Through carefully designed experiments, we show that all models indeed recommend future items that are not available at the time point of a test instance, as the result of data leakage.” The quantity is not fixed. It grows with how much of the future the training set is allowed to see. The paper reports “an increasing trend of ‘future items’ recommended by all models on all datasets” as more future data enters training. NeuMF on Yelp, tested at year Y5, returned 602 future items when trained to Y6. Trained to Y10, it returned 5,205. Every one of them is catchable by a date check, because every one carries a date.

The second leak carries nothing. Schnabel and Joachims gave the reason in 2016: “Most data for evaluating and training recommender systems is subject to selection biases, either through self-selection by the users or through the actions of the recommendation system itself.” A relevance set built only from what the logger exposed cannot be repaired by inspecting timestamps. The item that was never shown produced no row at all.

So their remedy changes the data rather than the audit: a test set where exposure is randomised. The Yahoo! R3 data pairs a self-selected training set of over 300K ratings from 15,400 users with a test set in which 5,400 users rated 10 randomly chosen songs. Their own Coat dataset has 290 Mechanical Turk shoppers rating 24 self-selected coats plus 16 uniformly picked ones from a 300-item inventory. Against a randomised test set the distortion becomes readable. Propensity-scored matrix factorisation (MF-IPS) reached test MSE 0.989 where naive MF reached 1.891, and MAE 0.810 against 1.154, on a paired t-test at p < 0.001. One leak you can audit for. The other you have to design out in advance.

The hundred-sampled-negative protocol is not a hypothetical bad habit. It has an origin, an author and a stated reason. A 2017 paper, Neural Collaborative Filtering, set it out. Hold out each user's latest interaction. Rank the test item against 100 randomly sampled items the user had not interacted with. Report HR and NDCG truncated at 10. The paper gives the motive without dressing it up: “Since it is too time-consuming to rank all items for every user during evaluation, we followed the common strategy [6, 21] that randomly samples 100 items that are not interacted by the user, ranking the test item among the 100 items.” It is a compute decision, adopted for speed. It also defines the ranking task that every score built on it describes.

What that decision does to the answer was measured in 2020. Krichene and Rendle showed that sampled ranking metrics are inconsistent with their exact counterparts. Their worked toy example has three recommenders — A, B and C — over a catalogue of n = 10,000 items. Sample m = 99 irrelevant items and the order reverses under sampled Average Precision, NDCG and Recall@10. The exact metrics rank C best. The sampled ones rank it worst. Only AUC preserves the exact ordering, and in the small-sample limit every metric collapses toward AUC. Their own summary ends the argument: “The worst recommender would be found to be the best one.”

This is not confined to constructed examples. On instances of real recommender algorithms run on MovieLens, they report, a study comparing them only on sampled metrics “would draw the wrong conclusion for top-heavy metrics such as Recall, NDCG and AP, even with unlimited repetitions of the experiment”. Keep the last clause. More runs do not fix it. The error is in the task the protocol posed, not in the noise around it.

Choosing the split is choosing which histories, users, items, catalog states and labels exist at each evaluation point. A random row split often breaks the chronology, and with it the dependency structure that recommendation data actually has. A useful protocol copies the thing it stands in for: how often the model is retrained, what was eligible, what candidate generation would have returned, what state the user was in, how long the outcome took to arrive, and which decision production actually made. Warm, cold, session and long-term questions may each need a protocol of their own.

Each of the failures above is a place where the simulation quietly stopped copying deployment. A training cutoff that admitted later events. A candidate set of 99 sampled items standing in for a catalogue of 10,000. A relevance set assembled from whatever the old policy chose to show. The protocol is not the paperwork around the experiment. It is the experiment.

The evaluation timeline runs from the training cutoff to repeated windows across time. The order of those cuts is what makes an offline number mean anything. Move one of them and the number answers a different question. Moving the training cutoff from Y6 to Y10 is exactly what took NeuMF on Yelp from 602 recommended future items to 5,205, at the same test year.

Each protocol asks a different question, and only one of them resembles deployment. A random interaction split asks whether the model can fill gaps in a history it has already seen most of. A leave-last or temporal split asks what happens next. A cold-start split asks about users, items, markets or domains the model has never met.

The disagreement between protocols has been measured. Seven recommenders — NMF, BPR, NeuMF, VAECF, NGCF, Triple2vec and VBCAR — were run on the Ta Feng and Dunnhumby datasets under three splits: leave-one-last-item, leave-one-last-basket and global temporal. The work was published at RecSys 2020. Across 230 models generated by varying the hyperparameters of NeuMF, VBCAR and Triple2Vec, Kendall's tau between the system rankings produced by pairs of splitting strategies ranged only between 0.5284 and 0.7630. Same models, same data, different split, and the leaderboard reorders.

Its authors draw the conclusion carefully: “This does not indicate that one strategy is better than another, but is evidence that these splitting strategies are in effect evaluating very different aspects of recommendation.” That is the reason to name the deployment question first. No split is right in general. A split is right when it matches the decision production makes.

The deployment question is named before training, not after it: warm ranking, cold start, next item, discovery, or policy value. Then the chronology is reconstructed, so that no candidate post-dates the decision it is offered for. Then the candidate set is fixed — all eligible items, or a sampled evaluation documented precisely, since a score won against 99 sampled items is a score on a different task. Then tuning is separated from reporting. Slices and baselines arrive last, and they decide whether the protocol answered the question it began with.