Candidate Generation as a Portfolio

Pinterest's Related Pins narrows billions of pins to roughly 1,000 candidates per query. For years one whole class of pin never reached those thousand. Local-language content was missing. When Pinterest published the system's history in 2017, it gave both halves of the reason: “First, there were not many local candidates in the first place, since the content did not appear on many boards. Second, there was much less historical engagement with local content, which caused our models to rank it lower, even if we did have local candidates.”

Note what that diagnosis rules out. Better ranking could not have fixed the first half. An item no source retrieves is not scored badly; it is not scored at all. The remedy came in three attempts: a local pin swap, a local pin boost the authors report as largely ineffective, and finally localizing the candidate sets themselves — filtering the board-pin graph by language and blending local candidates into the results. Local impressions went from 9% to 54%. The portfolio changed. The ranker was not what unlocked the content.

Source monoculture and ranking starvation sit at the two ends of the retrieval budget. One retriever supplies nearly everything that gets ranked. Or every retriever supplies so much that scoring truncates the list without telling anyone.

That budget is not a metaphor. Alibaba wrote it down in 2020, in the paper that introduced its COLD pre-ranking system: “Take the display advertising system in Alibaba as an example, the size M of the candidate set that is fed into the pre-ranking system often reaches ten thousand. Then the pre-ranking model selects top N candidates by certain metrics, e.g. eCPM (expected Cost Per Mille) for advertising system. The magnitude of N is usually several hundred.” Ten thousand candidates in, several hundred out, under a strict latency limit of 10-20 milliseconds.

The paper then prices what that limit buys. A vector-product DNN serves 60,000+ QPS at 2 ms. COLD serves 6,700 QPS at 9.3 ms. A full DIEN model manages 629 QPS at 16.9 ms. Candidate volume and model quality are drawn from the same account. A source that adds thousands of candidates is spending someone else's scoring capacity, and nothing in the slate announces which items fell off the end.

Sort the sources by what they know rather than by what they are called. Netflix's homepage is that sorting made visible: about 40 rows, up to 75 videos per row, each row typically driven by a different algorithm. PVR and Top-N know the member. Video-video similarity — “sims” — knows the catalog. Trending Now knows what is moving now. Continue Watching knows the session. A page-generation algorithm picks the rows from a pool of tens of thousands. The differences between those algorithms are designed, not incidental. Gomez-Uribe and Hunt said so in 2015: “In general, our different video ranking algorithms use different mathematical and statistical models, different signals and data as input, and require different model trainings designed for the specific purpose each ranker serves.” They report that the system influences about 80% of hours streamed, and estimate that personalization and recommendations save more than $1B per year.

Control and policy sources are there because a contract or an editor says so. The exploration source is the only one whose job is to create information the others cannot. That job has been measured. In 2010 a contextual bandit called LinUCB ran against the Yahoo! Front Page Today Module, and the paper reported: “Finally, using this offline evaluation method, we successfully applied our new algorithm to a Yahoo! Front Page Today Module dataset containing over 33 million events. Results showed a 12.5% click lift compared to a standard context-free bandit algorithm, and the advantage becomes even greater when data gets more scarce.” The offline evaluation method reuses previously recorded random traffic rather than requiring a live test. That is what makes an exploration source auditable before it ships.

No single retriever is guaranteed to cover every user state, catalog region, and product objective. YouTube built that assumption into the architecture itself. Google's 2016 paper on the YouTube recommender draws the funnel in one figure: millions of videos in the corpus, hundreds of candidates out of generation, dozens of results for the user. Then it draws a separate box labelled “other candidate sources”. That box's arrow skips candidate generation and feeds ranking directly. The reservation is deliberate: “Furthermore, this design enables blending candidates generated by other sources, such as those described in an earlier work [3].”

Somebody has to hold that portfolio together: quotas, deduplication, source metadata, latency budgets, and a way to measure what each source adds on its own. The capacity is fixed, and published. At ranking, “only a few hundred videos are being scored rather than the millions scored in candidate generation”. Scoring those millions at candidate-generation serving happens “under a strict serving latency of tens of milliseconds”. More candidates are not automatically better. They can overwhelm ranking, repeat evidence the ranker already holds, or eat the serving capacity.

Every source in the portfolio should be able to state its mission in one line. Pinterest published the audit trail for exactly that. Related Pins was assembled source by source — board co-occurrence through the Pixie random-walk service, session co-occurrence through Pin2Vec, search-based candidates, two visual-similarity sources, locale-segmented candidates. Each addition answered an identified hole rather than an ambition: “This led us to add new candidate sources as we identified gaps in coverage and recall”.

That sentence is the whole workflow in miniature. The gap is named and measured first. The source answers it. Then it has to fit inside a pipeline that narrows billions of pins to roughly 1,000 candidates per query, a budget every new source shares. Budgets, provenance-preserving deduplication and incremental recall follow. The last step switches each source off in turn, to see what the slate loses.

Where a candidate can be lost is what separates these three. Union with quotas loses almost nothing and pays for the overlap. A cascade is cheaper, and turns an early omission into a permanent one. Learned source routing adapts per request, and can quietly starve a source it never saw much evidence for.

Facebook Search had the straightforward union in front of it and turned it down. Simply combining the new embedding-KNN candidates with the existing Boolean-matching candidates carried a large performance cost and the maintenance cost of a dual index. Then a third objection: “the two candidate sets might have significant overlap which makes it inefficient overall”. The team built a hybrid retrieval framework instead. That is the price of union, stated by people who measured it.

The second half of their warning is about what happens after the merge: “However, since the current ranking stages are designed for existing retrieval scenarios, this could result in new results returned from embedding based retrieval to be ranked sub-optimally by the existing rankers.” A new source is not judged fairly by rankers trained on the old ones. So they added an embedding-similarity ranking feature and a human-rating feedback loop. The source was worth the work: the unified embedding produced +18% recall for events search and +16% recall for groups search over a text-only embedding.

Track raw candidates, eligible candidates, deduplicated candidates, ranked positions, impressions, and outcomes by source. Report unique contribution. Do not credit every source that returned the final item.

The ranker needs the same record, for a reason the YouTube team stated outright: “Ranking is also crucial for ensembling different candidate sources whose scores are not directly comparable.” Scores from a co-occurrence walk, a visual-similarity index and a trend counter are not on one scale. A merged slate without provenance hands the ranker numbers it cannot compare. It is also why Facebook Search had to add an embedding-similarity feature before a new source could be scored fairly.

Source attribution supports explanations and governance too. An item retrieved from an explicit compatibility rule should not be described as “because people like you chose it.”