Sequential Recommendation and Next-Item Prediction

Everything here turns on one line. Only events from the historical prefix, before the target timestamp, may be used. Representation, model and scoring all follow from it. The temporal evaluation at the end is what proves it held.

The line is easier to state than to hold. A 2023 study ran four recommenders — BPR, NeuMF, SASRec and LightGCN — on MovieLens-25M, Yelp, Amazon-music and Amazon-electronic. The split was leave-last-one-out, the one the field reaches for by default. All four recommended items that did not yet exist at the test timestamp. And the prefix is never the whole history anyway. The SASRec timings everyone quotes were measured on sequences cut to the most recent 200 actions.

An e-commerce sequence model predicted the path from product view to cart page to checkout, and predicted it accurately. It looked like the strongest model the team had built. What it had memorized was interface transitions, not product preference. Each decision below has a published counterpart, and the published counterparts are sharper than the intuition.

Each carries a different assumption about which part of a history matters. GRU-style recurrence compresses everything into one evolving state. SASRec-style attention lets each position reach back to the items it needs. The simple transition baseline assumes only the last item counts — and it wins often enough that a comparison without it means very little.

That last sentence is a measurement, not a caution. In 2018 a sixty-page journal study set GRU4REC and factorized Markov models such as FISM and FOSSIL against simple nearest-neighbour schemes, across several datasets. Ludewig and Jannach reported of the neighbourhood methods: “Our experiments reveal that algorithms of this latter class, despite their sometimes almost trivial nature, often perform equally well or significantly better than today's more complex approaches based on deep neural networks.” That is why the grey column stays in the table.

Recurrent networks, convolutional sequence models and self-attention architectures such as SASRec all estimate the next relevant item from prior actions. SASRec's authors describe their model as one that “seeks to identify which items are ‘relevant’ from a user's action history, and use them to predict the next item”. That framing belongs to the whole family. Recency, repetition, transitions and long-range dependencies become representable in a way an unordered profile cannot express.

The target is still a proxy. Predicting the next item may favour predictable navigation, replenishment or plain habit, even when the product is trying to produce discovery, satisfaction or long-term value. Note what the reported gains are gains in. SASRec's headline improvement is 6.9% Hit Rate and 9.6% NDCG against the strongest baseline. Both measure how well the next item was guessed. Neither measures whether guessing it served the user.

Chronology leakage inflates the number. Parameter-budget unfairness inflates the comparison. Random splits or future-derived features let later behaviour into training; an untuned small baseline lets a large attention model win without earning it. Each of the five below has been measured on named models.

An event vocabulary that admits page navigation will produce a model of the checkout funnel. Defining it is step one. Steps three, four and five decide whether the eventual comparison means anything. Each of them rests on a published measurement rather than a rule of thumb.

Step three, tuning the simple baselines. The literature does not agree with itself about which of two well-known sequential models is stronger. A 2022 review read 370 papers citing BERT4Rec. It found 40 peer-reviewed publications making 134 BERT4Rec-versus-SASRec comparisons on 46 datasets. BERT4Rec won 86 of them (64%), SASRec won 32 (24%), and 16 tied (12%). Petrov and Macdonald then ran the replication themselves. At default settings the original BERT4Rec code did not reproduce its published results: “However, we are able to replicate the reported results with the original code if training for a much longer amount of time (up to 30x) compared to the default configuration.” That is roughly 42 minutes of training stretched to about 21 hours. A baseline's number can be an artefact of its training budget rather than of its architecture.

It can also be an artefact of whose code you ran. Hidasi and Czapp compared six unofficial reimplementations of GRU4Rec against the official code, on five public datasets. One implements a different and flawed algorithm. The remaining five lack accuracy-critical features, and four contain bugs. “In the most extreme case, out-of-the-box performance is 99.63% lower compared to the official version.” The GRU row in your comparison table may not be GRU4Rec at all.

Step four, standardizing negatives, is the sampled-metrics result applied. Sampled metrics do not persist relative statements, even in expectation, and they collapse toward AUC as the sample shrinks. The sequential-model version of that experiment used four models and five datasets. It set uniform-random and popularity-based candidate sets at the target set size of 100 the field defaults to. Both strategies produced rankings inconsistent with the full ranking.

Step five is not a hyperparameter shrug, because the sweep has been published. SASRec's own scalability table varies maximum sequence length n on ML-1M across 10, 50, 100, 200, 300, 400, 500 and 600. NDCG@10 runs 0.480, 0.557, 0.571, 0.587, 0.593, 0.594, 0.596, 0.595. Training time runs 75, 101, 157, 341, 613, 965, 1,406 and 1,895 seconds. That is a 25x cost increase across the sweep. The authors' own reading: “Performance is better with larger n, up to around n = 500 at which point performance saturates (possibly because 99.8% of actions have been covered).” Past n = 200 there is 0.009 NDCG@10 left to gain, and buying it costs 4.1x more training time: 341 seconds at n = 200 against 1,406 seconds at n = 500. That is the shape of the curve a team should be producing for its own data.