Embeddings and Learned Representation Spaces

An embedding layer stores one trainable vector per item, token, category, or entity, and an integer index selects the corresponding row during the forward pass.

Only the rows that a batch actually uses may receive gradients in that step. That is why large vocabularies create sparse access patterns and optimization concerns all of their own.

In production recommendation systems the table stops being an implementation detail and becomes the model. Facebook said as much when it published its Deep Learning Recommendation Model in 2019: “Embeddings contribute the majority of the parameters, with several tables each requiring in excess of multiple GBs of memory” — enough that the tables must be split across devices under model parallelism.

Desai and Shrivastava attached a public number to the same shape at NeurIPS 2022. The MLPerf recommendation benchmark of that period — DLRM trained on the Criteo terabyte click logs, 24 days of data with 13 continuous and 26 categorical features — “contains 100GB of embedding memory (25+Billion parameters)”. Their abstract states the general case: “Embedding tables dominate industrial-scale recommendation model sizes, using up to terabytes of memory.”

The neural network sitting on top of that table is the small part of the system. Everything about how a lookup is sharded, updated, and served follows from one fact: 100GB of it is being indexed one sparse row at a time.

The word “embedding” describes the vector role, not one generation method.

Embedding space does not begin with universal semantic axes. Training pressure is what determines which distinctions become useful.

The last step of that loop — validate downstream — is the one most often skipped, and it has a purpose-built instrument. The Massive Text Embedding Benchmark was built in 2023 precisely because embeddings were being validated on one task at a time. MTEB spans 8 tasks, 58 datasets and 112 languages. Its authors ran 33 models across all of it, and the abstract records the result: “We find that no particular text embedding method dominates across all tasks.” A community expansion by 85 authors, MMTEB, published in 2025, has grown the same benchmark to over 500 quality-controlled tasks across more than 250 languages. The finding survived the scale-up.

That verdict is the empirical form of this section's claim. If no method wins everywhere across 8 tasks and 58 datasets, quality is not a property the vectors carry with them. It is a property of the match between the objective a space was trained under and the task it is asked to serve. That is why the loop ends in downstream measurement rather than in a plot.

Dot product reflects both direction and vector magnitude. Cosine similarity normalizes magnitude and compares angle.

Euclidean distance introduces another geometry. The appropriate measure depends on training objective, normalization, indexing system, and downstream decision.

The space inside a trained language model is not directionally uniform, and in 2019 Ethayarajh measured what that costs an angle. He sampled words uniformly at random and compared their contextual vectors layer by layer. In GPT-2 the average cosine similarity between two random words is roughly 0.6 through layers 2 to 8. Then it climbs sharply, until “word representations in GPT-2’s last layer are so anisotropic that any two words have on average an almost perfect cosine similarity”. In a space like that, a similarity of 0.95 is not evidence of relatedness. It is near the floor. Every cosine reading has to be compared against a random-pair baseline from the same layer before it says anything.

The same crowding had already been diagnosed from the other end, as a defect of the training objective rather than a curiosity of the geometry. Six authors showed at ICLR 2019 that likelihood training with weight tying pushes most learnt word embeddings “into a narrow cone”. Their §3 is blunt about what that looks like when you measure it: “we find the cosine similarities between word embeddings are positive for almost all cases. That is, the words huddle together and are not well separated in the embedding space.” They then priced the repair. Regularising the cone wider bought 2.0 perplexity on WikiText-2, and 1.08 and 0.93 BLEU on WMT 2014 En–De and De–En. Ethayarajh measured the same non-isotropy from his side: in all layers of ELMo, BERT and GPT-2, less than 5% of the variance in a word's contextualised representations is explained by a static embedding. Two independent measurements, one pathology. The raw number a similarity function returns is not on a scale you can read without calibration.

Compact vectors support several reusable system components.

Nearest examples are persuasive, so they require disciplined checks.

Much of a persuasive neighborhood can be structure the model never learned. Linzen showed that in 2016. He re-ran the standard word-analogy evaluation against baselines that discard the analogy itself. ONLY-B “ignores both a and a∗ and simply returns the nearest neighbor of b”. IGNORE-A drops one term. The offset method did win overall, with ADD scoring .53, .60 and .58 across three word2vec spaces trained with different context windows. But the margin varied enormously by category. On plurals the accuracy of ONLY-B “reached .70”.

Those numbers need their denominator, and the benchmark supplies one because it is a fixed, countable object. Mikolov and three colleagues built it in 2013 from five semantic and nine syntactic categories: “Overall, there are 8869 semantic and 10675 syntactic questions.” The GloVe paper restates the total the following year — “The dataset contains 19,544 such questions, divided into a semantic subset and a syntactic subset.” — and reports 300-dimensional vectors trained on 42 billion tokens scoring 75.0% overall on it, 81.9% semantic and 69.3% syntactic.

Set the two readings side by side. A published system scores 75.0% on those 19,544 questions. A baseline that has thrown the analogy away reaches .70 on the plurals category. Before a nearest-neighbor list is evidence that a relationship was learned, it has to beat a baseline that knows nothing about the relationship.