Item-Based Collaborative Filtering

Two items that keep turning up together may be an accessory used with the first, or a rival bought instead of it. The co-occurrence count says the same thing in both cases. A 2015 study by McAuley and colleagues made that distinction the object of measurement, and supplied the definition a similarity table cannot supply for itself: "substitutes are products that can be purchased instead of each other, while complements are products that can be purchased in addition to each other."

The two are treated as link types that have to be predicted separately, and it is done at catalog scale: an Amazon catalog of 9 million products, 237 million links and 144 million reviews. Their Sceptre model averaged 94.83% accuracy on substitute links and 90.23% on complement links. Across the binary prediction tasks, accuracy ran from 85.57% to 96.76%.

Typing the relation is not editorial advice about how to phrase a widget. It is a modelling task with its own error rate. A system that never performs it has silently answered both questions with one number.

Item-based retrieval settles what counts as a relation first — co-view, co-buy, sequence, wishlist, shared rating — and only aggregates for a user at the end. The steps between are where popularity, time, and task boundaries are kept from swamping the relation.

The temporal and contextual window is not a later refinement bolted onto a mature system. Amazon filed a patent on item-to-item similarity mappings on 18 September 1998. It was granted on 24 July 2001 as US Patent 6,266,649, two years before the 2003 paper. The abstract already describes recommendations served from "a previously-generated table which maps items to lists of 'similar' items". The same filing separately claims the same-task window: "In another embodiment, the recommendations are generated based on the current contents of a user's shopping cart, so that the recommendations tend to correspond to the current shopping task being performed by the user."

The precomputed table and the contextual window were written into one filing. They are two halves of one design, not a first version and a fix.

Similar interaction patterns make two items neighbors, and the recommender serves the neighbors of items already associated with the user. A 2001 paper by Sarwar and colleagues said why that axis is the practical one: "As we discussed earlier, item-based similarity is more static and allows us to precompute the item neighborhood."

The numbers behind that sentence come from MovieLens: 943 users by 1,682 movies, 100,000 ratings, sparsity level 0.9369. Adjusted cosine beat plain cosine and correlation, and was kept for the rest of the paper. Item-item reached MAE 0.726 against user-user's 0.732 at neighbourhood size 60. Accuracy was not the argument. What the precomputation costs was. A model retaining only 50 similar items per row reached 98.3% of the full scheme's accuracy while storing 3% of the items. At model size 25 it held 96% of accuracy on 1.9% of items. A table that small can be built ahead of time and read at request time. That is the whole proposition.

The relation still depends on how the event was defined. Co-view, co-buy, sequence adjacency, repeated substitution, and shared audience each build a different graph. They should not be collapsed into one undifferentiated similarity table.

Use item-item candidates alongside popularity, content, graph, and two-tower sources. That multi-source pool is not a whiteboard ideal. It is published architecture. The 2016 paper on YouTube's recommender lays out the funnel: candidate generation winnows a corpus of millions to hundreds, and ranking scores those few hundred with hundreds of features and returns dozens. Its Figure 2 shows "other candidate sources" entering the ranker alongside the learned generator. The authors state in print why the ranker has to reconcile them: "Ranking is also crucial for ensembling different candidate sources whose scores are not directly comparable." The same paper records that natural consumption produces very asymmetric co-watch probabilities between items. Even one source's own evidence about a pair depends on which item you started from.

If the scores arriving from different sources are not comparable, an unlabelled pool cannot be audited at all. Source labels make it possible to measure marginal recall, and to detect when one relation table dominates the candidate pool. For explanations, reveal a defensible relation: compatible accessory, similar topic, often considered together. Avoid claiming that collective behavior proves quality or suitability for the individual.

Raw count domination corrupts the whole table at once, since a popular item ends up related to everything. That failure was described for word pairs long before anyone applied it to catalogs. The log-likelihood-ratio co-occurrence score this lesson names among its similarity measures comes from a 1993 paper by Ted Dunning. On a 31,777-word financial corpus he analysed 2,693 bigrams. Of those, 2,682 fell outside the range where the normal chi-squared test is valid. The 11 that did qualify were exactly the ones you would not want: "It is notable that all of these bigrams contain the word the, which is the most common word in English." At the other end, the chi-squared ranking's top entries were dominated by pairs occurring exactly once. The naive association statistic survives for the most frequent token and rewards accidents. That is raw count domination, stated as a measurement.

Policy-induced co-occurrence has been simulated end to end. A 2018 study by Chaney and colleagues names the mechanism at the outset: "These systems are often evaluated or trained with data from users already exposed to algorithmic recommendations; this creates a pernicious feedback loop." They simulated communities of 100 users over 1,000 time intervals, with ten new items introduced per interval, each simulation repeated with ten random seeds and averaged. With repeated retraining on the system's own output, every non-random recommender homogenised user behaviour beyond the level ideal utility required. Popularity-based recommendation homogenised users the most globally. The users with lower relative utility were the more homogenised ones. The two traps meet there: the popularity that corrupts the table is also what the loop amplifies.

Seed contamination corrupts one user at a time. A single mistaken click can carry a chain of irrelevant recommendations for weeks.

An item relation without stated semantics is a number without a unit. The service fixes the semantics first, then normalizes frequency, validates examples, versions the table, and serves it behind safeguards.

Each step has a published precedent to borrow from. Normalization is where Dunning's log-likelihood ratio belongs, and the reason it exists is that the naive statistic left 2,682 of 2,693 pairs outside its valid range. Versioning is affordable because the table can be truncated: 98.3% of the full scheme's accuracy survived on 3% of the items. Serving inherits the YouTube problem. Scores from different candidate sources are not directly comparable, so the ranker, not the relation table, is where they are reconciled.