Ranking Metrics and Metric Portfolios

Almost every offline recommender evaluation takes the same shortcut. Instead of scoring each positive item against the whole catalogue, it scores it against a handful of sampled irrelevant items. Two researchers at Google worked out in 2020 what that costs. It is not extra noise. The shortcut is systematically misleading.

Their toy example has a catalogue of 10,000 items and 99 sampled irrelevant items per positive. Exact NDCG ranks recommender C best, 0.208 against A's 0.150 and B's 0.122. Run the same three recommenders under the sampled protocol and the order reverses. A comes first on 0.724, ahead of C's 0.460 and B's 0.444.

A bigger evaluation does not rescue this. The sampled rank follows a Binomial distribution, so repetition does not average the distortion away. In the small-sample limit every top-heavy metric collapses towards AUC. The same reversal then turned up on real recommenders and MovieLens data.

“In particular, for the recommender instances in the experiments, if the study would compare the recommenders only on the sampled metrics, it would draw the wrong conclusion for top-heavy metrics such as Recall, NDCG and AP, even with unlimited repetitions of the experiment. The worst recommender would be found to be the best one.” — Krichene and Rendle, 2021.

The number on the dashboard did not lie about the ranker. It answered a different question — how the ranker does against 99 items rather than 10,000. The answer to that question happened to order the systems the other way round.

Precision@K measures the relevant share of shown items, while Recall@K measures captured relevant items. Hit Rate asks whether one appears. MRR, MAP and NDCG emphasize different aspects of rank and gain. The metric also depends on relevance labels, candidate set, K, averaging population, duplicate policy, and whether unobserved items are valid negatives.

The dependence on the candidate set is a theorem, not an intuition. A 2013 analysis of NDCG-type ranking measures proved it:

“We first show that, whatever the ranking function is, the standard NDCG which adopts a logarithmic discount, converges to 1 as the number of items to rank goes to infinity.” — A Theoretical Analysis of NDCG Type Ranking Measures, 2013.

Whatever the ranking function is. The convergence is a property of the measure and the catalogue, not a reward for ranking well. So the absolute level of an NDCG figure is partly a statement about how many items were ranked. That is why the paper needed a second notion, “consistent distinguishability”, to establish that NDCG can tell good rankers from bad ones at all.

That property depends on how fast the discount decays, and 1/r is the critical point. A discount that decays substantially faster than 1/r loses the distinguishing power, and may not even converge. Hence the shape of the standard NDCG@k. A slow logarithmic decay, stopped by a hard cut-off.

These metrics disagree about which part of a ranking matters. MRR uses the rank of the first relevant result. It is strong for one-answer tasks and lookup-like surfaces, sensitive to the top position, and indifferent to every relevant item after the first. MAP averages precision at every relevant position, so it rewards a ranking that places several relevant items well. It usually assumes binary relevance, and it is sensitive to how completely the relevant set was labelled. NDCG uses graded gain with a logarithmic discount, supports multiple relevance levels, and is normalized by an ideal ranking. It also depends on the gain and discount you chose. That makes it the instrument of choice for top-heavy graded relevance, and the one with the most settings to disclose.

NDCG's discount was not picked for mathematical convenience. It was picked from a claim about readers, made in 2002 by the paper that introduced it:

“the greater the ranked position of a relevant document, the less valuable it is for the user, because the less likely it is that the user will ever examine the document.” — Järvelin and Kekäläinen, 2002.

A steeper discount — division by the rank alone — was rejected, because it would not “allow for user persistence in examining further documents”. Division by the log of the rank keeps the decay gentle. With base 2, a document at position 1024 still retains one tenth of its face value, since log2 1024 = 10.

The demonstration behind that claim was small and specific. Runs from five groups in the TREC-7 ad hoc manual track, 20 topics, rejudged on a novel four-point graded relevance scale (0–3), with the CG and DCG curves plotted over ranks 1 to 100. The log base, the graded scale, the depth of the plotted curve: each one is a modelling decision about readers. Each one arrives inside the number when a team reports NDCG today.

Relevance has to be defined as an event, with a weight, a time window and exposure conditions attached to it. Candidates and K have to match the actual surface and retrieval protocol. Averaging has to name the users, sessions, queries, empty cases and weighting it runs over. Metrics have to be paired, so that recall, ordering, slate quality, guardrails and outcomes are each covered by something. Then reporting is frozen: primary, secondary and diagnostic metrics declared before anyone has seen a result.

That specification exists in public, at benchmark scale. The TREC 2019 Deep Learning Track, run by NIST, set a document task over 3.2 million documents and a passage task over 8.8 million passages. Each had a reusable test set of 43 queries and four-level relevance judgments. Fifteen groups submitted 75 runs. The primary metric was declared in advance rather than chosen afterwards from whatever moved:

“Our main metric in both tasks is Normalized Discounted Cumulative Gain (NDCG)—specifically, NDCG@10, since it makes use of our 4-level judgments and focuses on the first results that users will see.” — Overview of the TREC 2019 Deep Learning Track, NIST.

The candidate protocol was fixed in advance too. Participation split into a “fullrank” subtask that ranked the whole corpus, and a “rerank” subtask in which every participant reranked the same provided set. That set was 100 candidates per query for documents, 1000 per query for passages. Two rerank runs are comparable because K, the candidate set and the primary metric were written down before the runs existed. Two production dashboards usually are not, for the opposite reason.

Every layer of the pipeline fails in its own way, so every layer gets its own metrics. Candidate generation answers with Recall@K, source recall, catalog coverage and empty-set rate. Ordered relevance answers with NDCG, MRR, MAP, Precision@K and Recall@K. Score quality answers with calibration, log loss and expected utility where probabilities matter. Slate quality answers with diversity, novelty, duplication, calibration and constraint satisfaction. The product and ecosystem row at the bottom — task completion, retention, provider value, complaints, long-term outcomes — answers for the thing the user actually got.

The top two rows coming apart is documented, not hypothetical. TREC 2019 reported Normalized Cumulative Gain at rank 100 for documents and rank 1000 for passages, alongside NDCG@10. The point was to test whether end-to-end runs recalled better candidates than the provided reranking set. The best document-task candidate generator was tuned BM25+RM3 over doc2query document expansion. It improved NCG@100 by 22.9% over the provided 100-candidate set, and the best passage run improved NCG@1000 by 20.7%. The best fullrank run then beat the best rerank run by 0.9% on NDCG@10 for documents, and 3.6% for passages.

“While it was possible for “fullrank” to achieve better NCG@k, it was also possible to make NCG@k worse, and achieving significantly higher NCG@k does not seem necessary to achieve good NDCG@10.” — Overview of the TREC 2019 Deep Learning Track, NIST.

A 22.9% gain in retrieved relevance arriving as a 0.9% gain in ordering is a fact about where a change landed. It is legible only because the two rows were measured separately.

The bottom row has an equally concrete reason to exist. The position discount is not only a weighting scheme. It is also the mechanism that hands out opportunity. Singh and Joachims showed this in 2018 with a worked job-search example. Three male applicants have relevance 0.80, 0.79 and 0.78; three female applicants have 0.77, 0.76 and 0.75. Rank them by the Probability Ranking Principle, weight the positions the way DCG does, and this is the result:

“If we consider a standard exposure drop-off (i.e., position bias) of 1/log(1 + j), where j is the position in the ranking, as commonly used in the Discounted Cumulative Gain (DCG) measure, the female applicants will get 30% less exposure – even though the average difference in relevance between male and female applicants is just 0.03” — Singh and Joachims, 2018.

No metric in the ordered-relevance row reports that gap. By that row's definition nothing went wrong: the ranker sorted by relevance, and the discount did what the discount does. The gap is visible only from a row that asks what the people on the other side of the ranking received.

When Recall@K improves and NDCG falls, the system may retrieve more relevant candidates but order them worse. When NDCG rises and coverage falls, head-user concentration may be increasing. Report confidence intervals across users or time windows, and inspect per-slice distributions. A small average gain can be operationally meaningless, or concentrated in one high-volume segment.

Baselines matter as much as metrics. The RecSys 2019 reproducibility study examined 18 neural recommendation methods and could reproduce only seven. Six of those seven were beaten by simple nearest-neighbour or graph-based baselines. The result did not soften at journal length. Ferrari Dacrema and colleagues extended the study in 2021, across neural collaborative-filtering papers from 2015 to 2018:

“The worrying outcome of the analysis of these recent works—all were published at prestigious scientific conferences between 2015 and 2018—is that 11 out of the 12 reproducible neural approaches can be outperformed by conceptually simple methods, e.g., based on the nearest-neighbor heuristic or linear models.” — Ferrari Dacrema and colleagues, 2021.

None of the computationally complex neural methods was consistently better than existing learning-based techniques such as matrix factorization. So beating a published neural number settles very little. In the reproducible fraction of that literature, 11 of 12 approaches were losing to a nearest-neighbour heuristic or a linear model that nobody had bothered to run.