Recommender Evaluation Beyond Relevance

Logged interactions reveal responses to exposed items, not preferences over everything that could have been shown. A recommender that narrows exposure can reinforce its own evidence.

An eye-tracking study in 2005 measured how much of a click belongs to the item and how much to the slot it was put in. Participants searched Google through a proxy that silently rewrote the ranking before it reached the screen. There were 34 of them in the first phase, 22 in the second, and ten fixed questions. Swap the top two results and subjects still preferred link one, even when the second abstract had been judged more relevant. Reverse the whole list and the average rank of a clicked document moved from 2.66 to 4.03, while clicks per query fell from 0.80 to 0.64. Same documents, same queries, different positions, different clicks.

Joachims and colleagues ran it, and their own summary is the sentence to carry forward: “we conclude that clicks are informative but biased. While this makes the interpretation of clicks as absolute relevance judgments difficult, we show that relative preferences derived from clicks are reasonably accurate on average.”

It is not an artifact of one laboratory. Craswell and colleagues perturbed the ranking of a major search engine in 2008 and reported that “the probability of click is influenced by a document's position in the results page”. Different organisation, different method, same conclusion.

Evaluation therefore has to separate observed engagement from a question the log cannot answer: what would users have chosen under another ranking?

Each property has to be defined on its own, and these definitions have authors. Intra-list similarity — the measure behind every diversity number a recommender team reports — was introduced in 2005 by Ziegler and colleagues.

The evidence came in two parts. Offline: 361,349 BookCrossing ratings, condensed from a four-week crawl of 278,858 members and 1,157,112 ratings over 271,379 distinct ISBNs. Online: a three-week survey in which 2,125 users participated. Topic diversification lowered accuracy in every configuration. What it did to the list as a whole depended on the algorithm. It “has no measurable effects on user-based CF, but significantly improves item-based CF performance for diversification factors around 40%”.

The abstract states the trade-off without softening it: “Though being detrimental to average accuracy, we show that our method improves user satisfaction with recommendation lists, in particular for lists generated using the common item-based collaborative filtering algorithm.” The broader conclusion is the reason this section exists at all: “the user's overall liking of recommendation lists goes beyond accuracy and involves other factors, e.g., the users' perceived list diversity.”

So the sub-item below reading “may harm relevance” is not a caution. It is a measured cost that 2,125 people were nonetheless willing to pay — for one family of algorithms and not another.

And “depends on similarity measure” is literal, down to the sign of the quantity. A later survey by Castells and Jannach restates the average-pairwise-dissimilarity formula, then notes in a footnote that “Ziegler et al. [2005] and other authors define intra-list similarity instead of dissimilarity”. Two papers can report diversity and mean opposite directions.

Spotify ran a randomized field experiment on its own recommender in 2020. Some users were given personalized podcast recommendations. The rest were given the most popular podcasts among users in the same demographic group. Engagement and diversity were both measured off that one random assignment. That is why the two results cannot be argued away as different populations or different weeks.

The finding, in the authors' words: “We find that, on average, the treatment increased podcast streams by 28.90%. However, the treatment also decreased the average individual-level diversity of podcast streams by 11.51%, and increased the aggregate diversity of podcast streams by 5.96%, indicating that personalized recommendations have the potential to create patterns of consumption that are homogenous within and diverse across users, a pattern reflecting Balkanization.”

That is the abstract of the authors' working paper, by Holtz, Aral and colleagues. Spotify Research's page for it serves the identical abstract with the same three figures.

Personalization won on the metric a recommender is usually judged by. It lost on the one underneath it. Both numbers came out of the same experiment.

Quality spans users, items, and the ecosystem. Since 2022, part of this scorecard is no longer the team's to choose. The EU's Digital Services Act requires online platforms to set out “the main parameters used in their recommender systems” in plain and intelligible language, including the criteria most significant in determining what is suggested. That is Article 27. Where there is an option to switch, it has to be “directly and easily accessible”. Article 38 goes further for the largest services: “In addition to the requirements set out in Article 27, providers of very large online platforms and of very large online search engines that use recommender systems shall provide at least one option for each of their recommender systems which is not based on profiling as defined in Article 4, point (4), of Regulation (EU) 2016/679.”

Read against the rows below, that turns coverage into an external specification. At least one reachable ranking that is not built on profiling, plus a published account of what drives the one that is. Ireland's Digital Services Coordinator, Coimisiun na Mean, states the same obligation in user terms — “a right to choose a recommender system feed that is not based on the profiling of their personal data”, with the switch available at any time and easily accessible.

The long-term health row now has enforcement attached to it. On 5 May 2026 Coimisiun na Mean opened two separate formal DSA investigations into Meta, one over Facebook and one over Instagram. They assess contravention of Article 27 (in particular 27.3) and Article 25 (in particular 25.1). Can users select and modify a recommender feed not based on profiling, through a directly and easily accessible functionality? And do the interfaces deceive or manipulate users away from choosing it? A finding of non-compliance can carry an administrative financial sanction of up to 6% of turnover.

The regulator did not frame this as a technical matter. John Evans, Digital Services Commissioner at Coimisiun na Mean, put it this way: “Our message is clear: it is unacceptable for platforms to prevent people from using their rights under the law, or to try to manipulate people away from making empowered choices about whether or not recommender system feeds control what they see online.”

What is being measured there is not relevance at all. It is whether an alternative ranking is reachable.

Established users with rich histories often dominate offline metrics. New users, new items, sparse-interest users and long-tail creators can experience a different system.

Report warm- and cold-start results, exposure concentration, creator or supplier slices, and fallback quality. A global ranking metric can hide entire populations with weak support.

A 2019 study showed how much it can hide. Abdollahpouri and colleagues split users by their interest in popular items into three groups they called “Niche, Diverse and Blockbuster-focused”. Recommendations came back “extremely concentrated on popular items even if a user is interested in long-tail and non-popular items showing an extreme bias disparity”. The niche users were served the blockbusters anyway. A catalog-coverage number computed over the whole user base would have averaged that away and shown nothing.

Use staged evaluation, because no one dataset answers every question.

Step 1 is where the Netflix Prize stopped, and it is worth knowing how that ended. On 26 July 2009 BellKor's Pragmatic Chaos submitted an entry scoring RMSE 0.8567 on the test set, a 10.06% improvement on Netflix's Cinematch baseline. Twenty minutes later The Ensemble submitted an entry that also scored 0.8567. To two further decimal places the test-set scores were 0.856704 for the winners and 0.856714 for the runners-up. Feuerverger and colleagues, writing in Statistical Science in 2012, put it plainly: “Since the contest rules were based on test set RMSE, and also were limited to four decimal places, these two submissions were in fact a tie. It is therefore the order in which these submissions were received that determined the winner”. The National Institute of Statistical Sciences reported the same 0.8567 and the same 10.06% when the award was made on 21 September 2009.

A $1M prize was settled by the clock. The single offline accuracy number had run out of resolution while two very different systems were still indistinguishable on it. Steps 2 through 5 exist for that reason: portfolio diagnostics, counterfactual support checks, an online experiment with guardrails, and a delayed review of retention, content health, creator exposure and feedback loops. An accuracy figure can be the best one available and still unable to tell two candidates apart.