Matrix Factorization for Explicit Feedback

Almost every cell of the Netflix rating matrix is an absence, and the shape of it is known to the digit. In October 2006 Netflix released a training set: “Netflix provided over 100 million ratings (and their dates) from over 480 thousand randomly-chosen, anonymous subscribers on nearly 18 thousand movie titles. The data were collected between October, 1998 and December, 2005”. The winning Grand Prize write-up gives the grid exactly — “This dataset gives ratings about m = 480,189 users and n = 17,770 movies” — then states the consequence in one sentence: “About 99% of the possible ratings are missing, because a user typically rates only a small portion of the movies.”

Regularization is not a stylistic preference in that regime. A model carrying a vector per user and a vector per item has far more parameters to place than it has ratings to place them with.

On a matrix of that kind, a rating model learned stable user and item vectors. Then a regional catalog expansion arrived. One factor analysts had been calling “quality” changed sign and began tracking availability and language mix. The predictions came through intact. The name did not.

A rating breaks into four parts: the global average, an item bias, a user bias, and the user-item interaction. The sentence it comes from is plain — “the observed rating is broken down into its four components: global average, item bias, user bias, and user-item interaction”. Three of the researchers who popularised these models wrote that in 2009.

The order of the stack is an empirical claim, not a convenience. The bias layer sits above the latent layer because the data demands it: “For example, typical collaborative filtering data exhibits large systematic tendencies for some users to give higher ratings than others, and for some items to receive higher ratings than others.” They draw the operational consequence themselves — “Since biases tend to capture much of the observed signal, their accurate modeling is vital.”

The latent vectors take what is left over. The L2-regularized squared-error objective stops them taking more than the evidence supports.

The model is a sum of four terms: a global mean, a user bias, an item bias, and the dot product of a user vector and an item vector. It is fitted by minimising squared error with an L2 penalty, and the canonical statement of it is the 2009 IEEE Computer article. The penalty does real work here. On the Netflix Prize matrix the parameters far outnumber the evidence they are learned from, because about 99% of the possible ratings are missing.

The factors are useful predictive coordinates, not guaranteed semantic axes. Their orientation can rotate, swap, or change across runs while the scores stay much the same. The size of that effect has been measured rather than merely feared. Change nothing but the random seed, and top-10 lists from the same method overlapped by less than 50% on three of five datasets.

Bias baselines come before latent factors, and the Netflix Prize turns that ordering into a number. Netflix published the score of its deployed system: “The company reported the RMSE performance of Cinematch trained on the Prize dataset against the quiz subset as 0.9514, a 9.6% improvement over simply predicting individual movie averages.” In BellKor's Grand Prize solution, a predictor built only from main effects “lowers RMSE to 0.9555”. That is almost as much of the data variability as the commercial system explained, with no user-item interaction term in it at all. Its biases were made time-aware: a linear user drift, a per-day user bias, time-binned item biases, and a per-day user rating-scale factor. Add a term for the number of ratings a user gave on that day and “RMSE drops from 0.9555 to 0.9278”. That is past the deployed commercial recommender, still with no latent factors anywhere in it. Without the per-day scaling factor the same baseline scored 0.9605. Step one is not a formality.

Step three, the rank and regularization sweep, decides more than it is usually credited with. Google re-ran the standard baselines on MovieLens 10M. Its publishers describe the set this way: “This data set contains 10000054 ratings and 95580 tags applied to 10681 movies by 71567 users of the online movie recommender service MovieLens.” The evaluation used a global random 90:10 split. Plain SGD matrix factorization “achieved an RMSE of 0.7720 for a 512-dimensional embedding and an RMSE of 0.7756 for 64 dimensions”. The literature had reported 0.8256 and 0.803 for that same method. Their Bayesian MF reached 0.7633. That beat MRMA at 0.7634, the best result any method had reported on that benchmark, and it beat the 0.8197 previously reported for the very same Bayesian model. Their summary, in 2019: “With a careful setup of a vanilla matrix factorization baseline, we are not only able to improve upon the reported results for this baseline but even outperform the reported results of any newly proposed method.” Validation curves, not factor-count folklore, are what stood between five years of published progress and its undoing.

Step five has a measurement of its own. In 2021 Gabbolini and colleagues compared top-10 lists from BPR-MF, Funk-MF and P-MF across five datasets, changing only the initial random seed. On three of the five the lists overlapped by less than 50% by Jaccard, and on BookCrossing by less than 30%. In their words: “For these datasets, more than 50% of the recommended items change by changing the initial random seed, i.e. by simply altering the initial values of the latent factors. For the BookCrossing dataset the instability is even more dramatic: lists overlap by less than 30%.” A seed comparison is not a nicety at that rate. It is the rate at which one fit disagrees with the same fit reseeded.

Evaluate rating error, ranking utility, coverage, calibration, and cold-start performance separately. A low RMSE does not guarantee that the top-N list is useful, because small rating differences can change ordering and exposure. When factors are used for retrieval or explanation, test neighborhood stability and avoid semantic labels unsupported by item content or user research.

The firmest statement of that limit came from the operator of the largest deployed rating-prediction system of its era. Netflix's own engineering leadership wrote in 2015: “Thus, while we do rely on offline experiments heavily, for lack of a better option, to decide when to A/B test a new algorithm and which new algorithms to test, we do not find them to be as highly predictive of A/B test outcomes as we would like.” Their diagnosis is that such experiments “assume that members would have behaved the same way ... if the new algorithm being evaluated had been used to generate the recommendations”. The ship decision rested on A/B tests measuring retention and engagement, not on an error metric. And this was the company whose own error metric had been the object of a public competition.

Random split optimism is a claim about the evaluation. Over- or under-regularization is a claim about the fit. A model can fail either one while passing the other, so test them apart. Otherwise a leaked future rating will be read as a well-tuned model. Two of the five below can be pinned to published evidence rather than to intuition. Seed instability is one: “by simply changing the initial values assigned to the latent factors, the same MF method generates very different embeddings of items and users, and we highlight that this effect is stronger for less popular items”. Factor storytelling is the other. Its canonical instance is a figure in the 2009 article, where the first two Netflix factors are annotated by hand with genre labels.