Re-Ranking for Diversity, Novelty, Serendipity, and Coverage

Take the failure in its simplest form. Ten tracks by one artist fill a playlist. Each of them scores highly on its own, and every one was relevant when it was evaluated alone. Listeners skip the later tracks anyway. Nothing in the scoring function is wrong about any single track. The decision procedure simply never asks what the ten look like together.

Diversity reduces redundancy. Novelty reflects unfamiliarity. Serendipity combines unexpectedness with usefulness. Coverage measures how broadly the system uses its catalog or intent space. These objectives are related but not interchangeable.

MMR-like and intent-aware re-rankers spend marginal relevance to buy incremental coverage. But what counts as redundant is decided upstream, by the similarity representation. That choice is a project of its own, not a detail. YouTube's diversification team evaluated dense and sparse audio embeddings, frame embeddings, thumbnail image embeddings and document text embeddings. They settled on Jaccard distances over sparse vectors of item tokens. Their worked example is the Saturday Night Live video 'Olive Garden - SNL'. It carries the tokens 'snl', 'olive garden', 'saturday night', 'night live' and 'sketch'. Two videos are near-duplicates for that re-ranker exactly to the extent those token sets overlap. Nothing else in the system has an opinion on the matter.

The abstract states the stake plainly: “Without a good match between the model’s definition of diversity and users’ perception of diversity, the model can easily degrade users’ perception of the recommendations.” The tokens are not a preprocessing decision. They are the definition of diversity the product will ship.

Relevance is the first row, not the whole ledger. The rest ask what the slate looks like once relevance has spoken. How much do the items repeat each other? How many distinct needs are covered? How far does the list reach from what the user already knows? And how widely is catalog and provider coverage spread across eligible supply?

The lower rows are not decoration, and one platform has put numbers on them. Listening breadth turns out to track whether a user stays and whether a user pays. Spotify measured it with a generalist-specialist score, the GS-score, over fine-grained interaction data of more than 100 million users, and published the result in 2020. In their own words: “Controlling for activity (the number of songs played), generalist users are up to 25 percentage points less likely to leave the platform and up to 35 percentage points more likely to become a paid subscriber.” At the highest activity level the gap is starker still. Extreme free-tier generalists convert 38 percentage points above average, against 3 percentage points for extreme specialists.

The same paper reports that user-driven listening is typically much more diverse than algorithm-driven listening. The narrow rows of the ledger, in other words, were the ones the system itself was filling.

Taxonomy theatre passes the metric and fails the reader: category diversity is high while the items remain functionally the same. A static tradeoff fails everyone slightly, applying one diversity weight to every user, surface and intent state. That failure has been measured. Spotify ran a one-week randomized A/B test on 540,000 free users, across seven 'All Out' decade playlists. The rankers were trained on a sample of 4 million users and 6.6 million sessions. One policy landed on two segments in two different places. Sorting by relevance over popularity gave generalists +10.03% song streams and +4.71% skips. It gave specialists +25.66% streams with only +2.89% skips. Their summary: “comparing the performance differences across specialists and generalists, we observe that sorting by relevance is disproportionately effective for specialists — the gain in using relevance over popularity is over twice as high for specialists than it is for generalists.” The learned ranker split the same way, cutting skips 9.76% for specialists while adding 0.90% skips for generalists.

A relevance collapse is the opposite risk, and the YouTube team hit a version of it from the metrics side. The deep Gramian DPP variant scored best on satisfied homepage watchers, at +1.72%, and was not deployed. It changed the ranking enough that secondary business metrics began to be significantly impacted.

Name the quality goal before reaching for a diversity formula. Redundancy reduction, intent coverage, discovery and supply coverage are four different objectives with four different tests. Define the goal first. Choose the similarity evidence second — attributes, embeddings, relations, or user-specific subprofiles. Preserve relevance third, with floors, constraints or tradeoff curves. Evaluate positions and users fourth. Finish online.

Step four is the one that needs a named instrument, because a page-level average will not supply it. The measures everyone was already using could not see a list at all. Clarke and colleagues opened their 2008 paper with the diagnosis: “The evaluation measures in widespread use — such as MAP, bpref [8] and nDCG [20] — assume that the relevance of each document can be judged in isolation, independently of other documents.” Their answer was alpha-nDCG, a rank-discounted measure that rewards novelty and diversity across a ranked list. A redundant item earns less the further down a list of things already seen it sits. They built its test collection from the TREC 2006 question answering track's 75 question series. That came to an average of 17.12 information nuggets per query — maximum 56, minimum 7 — and 3,243 query-document-nugget tuples. They set the novelty parameter alpha = 0.36 from the 36% precision of the TREC answer patterns.

That is what "measure top-weighted diversity" means in practice: nuggets defined per query, a discount that punishes repetition by position, and a novelty parameter someone had to justify with a number. Then the sequence ends online, in skips, saves, discovery, retention, and the controls users actually touch. Which is exactly where the Spotify A/B test found one policy splitting its two segments apart.