Embeddings and Two-Tower Retrieval

A user or context tower produces a query embedding. An item tower produces item embeddings on its own. Similarity between the two is what makes large-scale retrieval possible, because item vectors can be precomputed and indexed.

The funnel has exact dimensions. Candidate generation reduces a corpus of millions of videos to hundreds. Ranking scores those hundreds down to dozens. Google published the design of YouTube's recommender in 2016. Retrieval itself is served as an approximate nearest-neighbour lookup in dot-product space, and it has to finish under a strict serving latency of tens of milliseconds.

Two orders of magnitude separate the stages, and that gap decides what each one is allowed to see. The paper puts it in one sentence: “During ranking, we have access to many more features describing the video and the user's relationship to the video because only a few hundred videos are being scored rather than the millions scored in candidate generation.”

Keeping the towers apart is what buys the scale. It is also what limits them. Cross-features between a specific user and a specific item cannot be computed inside either tower. Retrieval trades detailed pairwise reasoning for broad matching a machine can finish in milliseconds across millions of candidates. The scale is bought with a blind spot.

A learning app ran a powerful ranker behind a two-tower retriever. Offline, the ranker looked excellent on the candidates it was handed. The advanced course the user needed was not among them, because the item tower encoded only title and category. No amount of ordering recovers an item that candidate generation never passed on. The ranker scores the hundreds it receives, not the corpus.

Two towers, computed at different times. The item tower runs offline or nearline across the whole corpus — tens of millions of videos, in the Neural Deep Retrieval deployment. The query tower runs from user and context features while the request is waiting. It works inside the budget the YouTube paper records for retrieval: an approximate nearest-neighbour lookup in dot-product space, under a strict latency of tens of milliseconds.

They meet in the vector index. What comes back is not a recommendation. It is a candidate handoff — identifiers, scores, source labels, freshness. Everything downstream is an ordering of that handoff.

The retrieval target is a separate decision from the ranking target, and it gets written down first. Feature partition, similarity choice and index lifecycle follow. Source recall is the number that says whether retrieval did its job.

Step 3, choosing the similarity, has a published answer rather than a preference. In 2020 Rendle's group at Google re-ran the neural-collaborative-filtering experiments. The data was binarised MovieLens 1M and a Pinterest dataset. Each user's held-out item was ranked against 100 random items and scored by HR@10 and NDCG@10, at embedding dimensions of 16, 32, 64, 96, 128 and 192. The dot product beat the learned MLP similarity on every dataset, every metric, every dimension.

Their serving argument is separate from the accuracy one: “MLPs are too costly to use for item recommendation in production environments while dot products allow to apply very efficient retrieval algorithms”. Efficient sublinear-time retrieval exists for dot products and, to those authors' knowledge, not for MLP similarities. Validate norms in the same step, because dot product and cosine coincide only under the equal-norm condition.

Step 4, the index lifecycle, has a published operating curve too. ANN-Benchmarks fixes the datasets. GLOVE: 1,183,514 points and 10,000 queries in 100 dimensions, under angular/cosine distance. SIFT: 1,000,000 points and 10,000 queries in 128 dimensions, under Euclidean distance. Recall is defined against the true k nearest neighbours and plotted against queries per second. On SIFT with 10-NN in batched mode, the price of the last stretch of recall is explicit: “FAISS' inverted file index on the GPU is by far the fastest index, answering around 655 000 queries per second for .7 recall, and 61 000 queries per second for recall .99.”

Same index, same data, two very different machines. Index quality is a point you choose on that curve, not a property an index either has or lacks.

Tower feature omission is structural. The architecture cannot represent a user-item interaction that matters, and the loss is taken before any ranker is consulted. Facebook Search measured what the tower's feature set alone is worth: “For example, there are +18% recall improvement when switching from text to unified embeddings for events search, and +16% recall improvement for groups search.” The ranker was not touched. Only what the retrieval model was permitted to encode changed.

Ranker illusion is the procedural twin of that structural fault. The team measures the stage it can see while recall in the stage it cannot quietly falls away.

Measure recall@K against mature outcomes, source overlap, tail coverage, new-item recall, user-state slices and index freshness. Then compare exact search on samples with approximate search, to separate model loss from index loss.

Facebook Search names that second comparison and runs it. The metric is stated plainly: “We measured ANN accuracy using 1-recall@10, which is averaged recall of getting top result from exact KNN search in top 10 results of ANN search.” The paper reports it for eight quantization configurations of one 128-dimension embedding, all at comparable scan cost — 0.48% to 0.60% of the index. The worst, PCA,PQ16, scores 51.62%. The best, PQ64 and Flat, score 74.81%. Roughly 23 points of 1-recall@10, with the embedding model held fixed. That is how much recall the index alone can take from a retriever whose vectors never changed.

Geometric proximity is also not final utility, and that gap has been measured across a whole benchmark rather than argued. BEIR evaluated 10 retrieval systems across 18 public datasets: “Dense retrieval models (esp. ANCE and TAS-B), that map queries and documents independently to vector spaces, perform strongly on certain datasets, while on many other datasets perform significantly worse than BM25.” BM25 uses no embeddings at all and remained the robust baseline. DPR was the worst overall, −47.7% on average against it. The cross-attention re-ranker BM25+CE beat BM25 on 16 of the 18 datasets.

In-domain strength predicted none of it. On MS MARCO, in domain, BM25 trailed the neural approaches by 7 to 18 points.

Watch the handoff for the same reason when retriever and ranker are trained on different objectives. A retriever optimized for clicks can omit candidates a long-term-value ranker would have preferred.