Embeddings and Representation Spaces

An embedding maps an item to a point in a learned space. Nearby points often share properties useful for the objective that created the space — predicting context, say, or matching an image to its caption. The geometry is not a neutral map of the world. It reflects data, labels, sampling, model capacity and loss design. Similarity has to be read through the task that trained it.

The rest of this lesson takes that out of the abstract. A benchmark that ran 33 models over 58 datasets found no winner. A test that changes one word to its negation put most retrieval models at or below random ranking. A word-embedding association test predicted the share of women in 50 occupations at a correlation of 0.90. Each of those is a published number about what proximity in a learned space does and does not mean.

The word “embedding” covers several different objects. Lookup vectors attached to tokenizer IDs. Position-specific contextual states shaped by the tokens around them. Pooled or separately trained vectors for sentences, passages, images and audio. Cross-modal vectors trained so related items from different modalities can be compared. And the versioned product index vectors stored for retrieval, ranking, clustering and deduplication.

The gap between the second and the third of those has been measured. Finding the most similar pair in a collection of 10,000 sentences, by feeding every pair through the model, takes roughly 50 million inference computations. Sentence-BERT produces separately pooled sentence vectors instead, comparable with cosine similarity. Reimers and Gurevych stated the difference in 2019: “This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT.” — Nils Reimers and Iryna Gurevych, Sentence-BERT abstract, 2019.

65 hours against 5 seconds, on the same 10,000 sentences. That is why a stored sequence embedding is a different object from a contextual state, and not a convenience layer over it. A contextual state exists only in relation to the input it was computed with. A sequence embedding is computed once and compared afterwards.

The metric should match both the training objective and the retrieval contract. Cosine similarity compares vector direction after discounting magnitude. It is common for normalized retrieval embeddings, insensitive to uniform rescaling, and not a calibrated probability. Dot product combines direction and magnitude in one score. It is efficient for maximum inner-product search and often matches contrastive training objectives, but lets large norms dominate rankings. Euclidean distance measures straight-line separation. It is sensitive to scale and normalization, and can degrade in high dimensions.

“Not a calibrated probability” is the weak version of a stronger published result. Harald Steck and two colleagues at Netflix put the question in their title: “Is Cosine-Similarity of Embeddings Really About Similarity?” Working analytically on regularized linear models, they showed that the cosine similarity of learned embeddings can be non-unique. It is implicitly determined by the regularization used in training, so the same fitted model can be made to yield arbitrary similarity values. Their own conclusion, in the abstract: “Based on these insights, we caution against blindly using cosine-similarity and outline alternatives.” — Steck and colleagues, 2024.

The practical reading is narrow and useful. A cosine score is not a property of the two items. It is a property of the two items, the model, and the regularization that shaped that model's embeddings. Thresholds inherit all three.

A retrieval demo can look convincing while failing the decision it is meant to support. Three of the five failures below are not warnings. They are benchmark results.

Take two documents that differ by one word — the word that reverses the policy — and ask a retrieval model to rank them. That is the whole design of NevIR, published at EACL in 2024. The abstract states the outcome: “We find that most current information retrieval models do not consider negation, performing similarly or worse than randomly ranking.” — NevIR abstract, 2024. Cross-encoders came out best. Bi-encoder and sparse neural architectures came last, which is to say the architecture that products actually embed and index is the one that handled negation worst.

Crowding by popular neighbours has a name and a measurement too. Take 10,000 i.i.d. uniform points under Euclidean distance and watch the sample skewness of N₅ — how often a point turns up among another point's five nearest neighbours. It rises from 0.121 at d=3 to 1.541 at d=20 and 5.445 at d=100. Across 50 real datasets, the Spearman correlation between dimensionality and skewness was 0.62. The 2010 paper in the Journal of Machine Learning Research that reported this gave the effect its name: “Through theoretical and empirical analysis involving synthetic and real data sets we show that under commonly used assumptions this distribution becomes considerably skewed as dimensionality increases, causing the emergence of hubs, that is, points with very high k-occurrences which effectively represent "popular" nearest neighbors.” — “Hubs in Space”, JMLR, 2010. High-dimensional retrieval spaces are exactly where a few items become everyone's neighbour.

The social case is the best quantified of the three. Word vectors carry a gender association, and that association predicted the percentage of women in the 50 most relevant occupations in the 2015 U.S. Bureau of Labor Statistics data with Pearson's ρ = 0.90, p < 10⁻¹⁸. Aylin Caliskan and two colleagues reported it in Science in 2017, along with WEAT and WEFAT, the tests that produced it. In their words: “Our results indicate that language itself contains recoverable and accurate imprints of our historic biases, whether these are morally neutral as towards insects or flowers, problematic as towards race or gender, or even simply veridical, reflecting the status quo for the distribution of gender with respect to careers or first names.” — Caliskan and colleagues, 2017. A ρ of 0.90 is not a trace of bias surviving in the geometry. It is the labour market's gender composition legible in the vectors.

Use an evaluation set that reflects the errors your system cannot afford. Define what should count as a useful match for the product. Mine hard negatives: near-duplicates, negations, stale versions, same-topic wrong answers. Compare recall, ranking quality and threshold behaviour by slice. Read neighbourhoods rather than trusting aggregate scores. Rerun production-shaped regressions after every index rebuild or migration.

BEIR shows what the comparison step looks like when it is done at scale. It evaluated 10 state-of-the-art retrieval systems — lexical, sparse, dense, late-interaction and re-ranking — on 18 publicly available datasets, all zero-shot, in 2021. The abstract: “Our results show BM25 is a robust baseline and re-ranking and late-interaction based models on average achieve the best zero-shot performances, however, at high computational costs.” — BEIR abstract, 2021.

An embedding-free lexical baseline that dense and sparse retrievers often failed to beat out of distribution is the useful control your evaluation set probably lacks. Over 18 datasets the ordering changed with the dataset. A single in-distribution demo would have shown none of that.

Representations connect language models to retrieval, clustering, deduplication, routing, and multimodal matching. Their value comes from compressing useful structure into a scoreable form. 65 hours of pairwise inference reduced to about 5 seconds of vector comparison is that value stated as a number. That compression always discards information, and the discarded part is where negation, rare items, and group differences live. The next lesson compares model families that create and consume different representation types.