Bayesian Personalized Ranking and Pairwise Learning

What makes a sampled negative look hard is the same thing that makes it look like an item the user actually wants. That is a measurement, not a worry. A 2020 paper by Ding and colleagues puts it in one line: “Both false negative instances and hard negative instances have large scores (Figure 1(b)), making it hard to discriminate them (harder negative samples are more likely be false negative, Figure 1(c)).”

To put a cost on it, they built synthetic sets. A controlled share of held-out true positives was fed back into training as negatives. Movielens-100k has 942 users, 1,447 items and 44,140 training records; 5,509 false negatives went in, taken from 50% of the test ground-truth records. Ecommerce-toy has 1,000 users and 2,000 items; 3,246 went in. Their method, SRNS, separates the two cases by prediction variance rather than by score alone. Across three real datasets it improved NDCG@1 by 4.71-8.40% over seven baselines, uniform sampling and AOBPR among them.

A hard sampler is not a difficulty dial. It is a dial that trades difficulty against mislabelling. The paper prices the trade.

For a user, BPR constructs a positive item and an unobserved item, then increases the score difference between them under a regularized pairwise objective. It targets personalized order directly, rather than rating reconstruction.

The step that does the work is an assumption, and the 2009 paper states it as one: “If an item i has been viewed by user u – i.e. (u, i) ∈ S – then we assume that the user prefers this item over all other non-observed items.” Formally the training data is the triple set D_S := {(u,i,j) | i in I+_u and j in I \ I+_u}. Every item the user has not been observed to consume is eligible to be the j that loses the comparison.

The evaluation ran on two datasets. Rossmann: 10,000 users, 4,000 items, 426,612 purchases. A Netflix subsample: 10,000 users, 5,000 items, 565,738 rating actions. Hyperparameters were grid-searched in the first of 10 repeated train/test splits, then held constant.

The pair is an assumption created by the sampler. Draw the negative uniformly, by popularity, from the batch, adaptively, or from the hard cases. Each choice hands the model a different training distribution, and a different risk of calling a future purchase a negative.

Time defines the positives. The sampler is declared rather than inherited from a default, and false negatives are controlled deliberately. A BPR baseline built that way can be compared with whatever claims to beat it.

Step 3 has a published mechanism behind it rather than a wish. Exposure can be modelled as a latent variable, and in 2016 four researchers did exactly that. Their abstract gives the reason: “In implicit feedback settings, all the items, including the ones that a user did not consume, are taken into consideration. But this assumption does not accord with the common sense understanding that users have a limited scope and awareness of items.” Their model, ExpoMF, ran on four datasets: Taste Profile (221,830 users, 22,781 items, 14.0M interactions), Mendeley (45,293 / 76,237 / 2.4M), Gowalla (57,629 / 47,198 / 2.3M) and arXiv (37,893 / 44,715 / 2.5M). It beat weighted matrix factorisation on MAP@100 on three of the four: 0.109 against 0.092 on Taste Profile, 0.055 against 0.048 on Mendeley, 0.054 against 0.051 on arXiv. Precision@k the same authors declined to report at all, on the grounds that an item absent from a user's test set may still be consumed later. The false negative had moved out of training and into the metric.

Step 5 is not a formality either, because the candidate protocol can decide the winner on its own. In Rendle's worked example there are n = 10,000 items and m = 99 sampled irrelevant items. Exact average precision ranked algorithm C best, at 0.101 against 0.010 for both A and B. The sampled version put A on top at 0.630 ± 0.129, with B at 0.336 ± 0.073 and C at 0.325 ± 0.050. His verdict: “Compared to the exact metrics in Section 3.2, even the relative ordering of metrics completely changed.” Sampled recall needed about m = 5,000 samples out of 10,000 items before it became reliable, and only AUC stayed consistent under sampling.

So where the arms must differ, freeze everything else and say so. A 2020 paper of Rendle's puts the rule in a footnote: “However, we wanted to use the same loss and sampling strategy for all competing methods to ensure that this is a meaningful comparison, which will allow us to attribute differences in quality to the choice of similarity functions.” Whatever you leave free is the only thing the comparison is about.

One pass of BPR is three draws and a comparison: a user, an item that user interacted with, an item they did not. That third draw is the j in the triple set D_S := {(u,i,j) | i in I+_u and j in I \ I+_u}. It is there because it was not observed. Nothing in the data says it was rejected. The loss lifts the first score above the second. The closing step watches for behavior that belongs to the sampler rather than to the data.

Eleven of eighteen neural recommenders published at top-level conferences could not be reproduced with reasonable effort. The counts went on the record in 2019, in a paper by Ferrari Dacrema and two colleagues: “Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort.” Of those 7, 6 “can often be outperformed with comparably simple heuristic methods” — nearest-neighbour or graph-based. The single method left standing “did not consistently outperform a well-tuned non-neural linear ranking method”.

What a tuned baseline is worth was then priced. In 2020 Rendle and three colleagues revisited the NCF experiments on the original authors' published dataset splits and evaluation code. They trained a dot-product matrix factorisation and chose its hyperparameters, the stopping iteration included, on a validation set. At d=192 that MF beat the MLP-based NeuMF on both datasets. On Movielens 1M: HR@10 0.7294 against 0.7093, NDCG@10 0.4523 against 0.4349. On Pinterest: HR@10 0.8895 against 0.8777, NDCG@10 0.5794 against 0.5576. The NeuMF numbers were not theirs to retrain; they are the non-cherry-picked results Ferrari Dacrema and colleagues had reproduced. Loss and negative sampler were held identical across the competing models, so the remaining difference belonged to the similarity function alone.

So keep a transparent, tuned baseline, and have it before claiming that a deeper architecture has learned superior personalization. Evaluate top-K relevance, popularity distribution, coverage, false-negative slices, and online value. A harder sampler that improves offline recall can still reduce discovery or new-item exposure.