Multi-Stage Ranking and Feature Serving

Each stage sees fewer candidates than the one before it and spends more on each. Retrieval is judged by what it fails to keep; final re-ranking is judged on the slate as a whole.

YouTube's engineers published this shape in 2016. They split the system along the classic two-stage retrieval dichotomy: a deep candidate generation model first, then a separate deep ranking model. Three years later Yi and colleagues published a sampling-bias correction for two-tower retrieval, applied to a corpus of tens of millions of videos.

The last stage asks a different question, and that question is older than either paper. Maximal marginal relevance was described in 1998, by Carbonell and Goldstein. It re-orders a list that has already been retrieved. Each document scores on a linear combination of query relevance and dissimilarity to the documents already selected. One end of the tuning parameter gives the plain relevance ranking, the other a maximal-diversity ranking. Their criterion: "a document has high marginal relevance if it is both relevant to the query and contains minimal similarity to previously selected documents."

The evidence underneath it was thin. The paper reports a pilot with five undergraduates, who were not told which ordering they were seeing. Of those five, 80% chose MMR when asked to pick a method for a search task. There was also a 70% accuracy score for the MMR-based summariser at the SUMMAC evaluation in May 1998. Five users, and the idea is now in every pipeline.

A per-item score is simply the wrong quantity at the last stage. That argument was made again in 2019, with deployed numbers behind it. Of the ordinary learning-to-rank stage, the Alibaba re-ranking paper says: "However, it may be sub-optimal because the scoring function applies to each item individually and does not explicitly consider the mutual influence between items, as well as the differences of users' preferences or intents."

Their re-ranker is a follow-up module after any ranking algorithm. It takes the existing model's list as its input, then uses self-attention to model relationships between every pair of items in that list. Alibaba tested it online across hundreds of thousands of users and millions of requests. Added on top of a DNN-based learning-to-rank ranker, it raised page views by 0.77%, item page views by 1.75%, CTR by 0.97% and GMV by 0.13% for DLCM. For PRM-BASE the same four numbers were 1.27%, 2.44%, 1.16% and 0.36%. The authors note that the 0.77% alone means about billions of extra items viewed. Each stage has its own labels, its own cost budget, and its own way of losing the right answer.

Taobao's recommender runs three stages. Candidate generation selects around 10^5 items, then coarse-grained ranking, then fine-grained ranking. In 2020 Alibaba published what goes wrong in the middle of that cascade.

The constraint is the ordinary one. The coarse-grained model "is strictly restricted, in order to grade tens of thousands of candidates in milliseconds". So the user-item interacted features that the later stage relies on are prohibited there. The paper's Figure 2 labels them the privileged features for coarse-grained ranking, and the abstract defines the category: "We define the features that are discriminative but only available during training as the privileged features."

The remedy is not to smuggle those features into the cheap stage. It is to distil them. A teacher that sees the privileged features trains a student that does not. Only the student is served.

Large systems progressively reduce candidates through retrieval, filtering, pre-ranking, ranking, and re-ranking. That shape was formalised in 2011, in a SIGIR paper on cascade ranking. The model "constructs a cascade of increasingly complex ranking functions that progressively prunes and refines the set of candidate documents to minimize retrieval latency and maximize result set quality."

What makes that paper the reference is that it prices the pruning instead of asserting it is cheap. Table 3 gives two columns per stage. One is the percentage of documents filtered from the previous stage. The other is "filter loss", defined in the caption as "% documents incorrectly pruned out of all documents passed from the previous stage". On Clue — the first English segment of ClueWeb09, 50,220,423 documents — stage 1 filtered 97.7% of documents at a filter loss of 0.09%, stage 2 filtered a further 68.3% at 0.18%, and stage 3 filtered 10.7% at 0.0%. On Gov2, 25,205,179 documents under TREC topics 701–850, stage 1 filtered 95.1% at 0.15% and stage 2 filtered 50.0% at 1.6%. Those are the two columns a pipeline owes its reviewers. The second one is never zero for free.

The reward bought with that loss is on the clock. With the effectiveness/efficiency tradeoff parameter γ set to 0.1, the learned cascade ran through stages 0–3. It was 32.7%, 48.7% and 34.7% faster than the monolithic AdaRank baseline on Wt10g, Gov2 and Clue: 0.175s against 0.260s, 2.00s against 3.90s, 4.28s against 6.55s. Effectiveness the authors describe as "similar (and many times slightly better)" than AdaRank. NDCG@20 was marginally higher on Wt10g (35.60 vs 35.49, +0.3%) and on Gov2 (47.44 vs 47.37, +0.1%), and 1.1% lower on Clue (30.60 vs 30.94).

Earlier stages must preserve enough optionality for later stages while meeting strict cost limits. So every stage needs a contract: input set, output budget, objective, available features, latency, recall target, failure fallback. Optimizing stages independently can create a pipeline that is locally strong and globally weak. And a cascade shipped without a filter-loss column is one whose designers have decided not to know.

Candidate budgets and latency are declared at every boundary first. Every candidate dropped along the way is logged last, with its reason, score, threshold, source, and stage.

Step 2, aligning objectives, is the step teams treat as paperwork. Facebook Search published what skipping it costs. In 2020 its engineers reported that changing the retrieval stage left the stages downstream of it mis-fitted. Because "the current ranking stages are designed for existing retrieval scenarios", new results from embedding-based retrieval could be "ranked sub-optimally by the existing rankers". The rule they draw is exactly what step 2 exists to enforce: "The model at each stage should be optimized for the distribution of results returned by the preceding layer."

Two repairs shipped, rather than one memo. The query–result embedding cosine similarity was propagated down the funnel as a ranking feature. And a closed feedback loop sent the results logged after embedding-based retrieval was enabled to human raters. Their labels re-trained the relevance model, so it could filter out the irrelevant results the new retriever had introduced.

Step 3 has a published form in the same work. The retrieval stage is scored on what it fails to keep, as recall@K: run KNN search over the whole index for 10,000 sampled search sessions, then average. Against that measure, switching from text embeddings to unified embeddings gave +18% recall for events search and +16% for groups search. Online hard-negative mining added +8.38% recall for people search, +7% for groups and +5.33% for events. A pipeline that cannot account for what it dropped has no recall story to tell.

A recall cliff is a loss, and a missing trace is the reason nobody finds it. The serving side of that problem has one systematic published account: Google's TFX team described the platform's data-validation component in 2019. Hundreds of product teams use it to monitor several petabytes of production data per day. The production analysis draws on a sample of more than 700 ML pipelines.

The paper names three kinds of training-serving skew. Their most common cause is "different code paths used for generation of training and serving data", which are required because offline and online generation have widely different latency and throughput characteristics. The skew, in other words, is manufactured by the same latency budget that made the staged pipeline necessary in the first place.