Static Word Embeddings and Distributional Meaning

The distributional hypothesis proposes that words occurring in similar linguistic contexts often have related uses. A model can therefore learn coordinates from nearby words, documents, windows, or prediction tasks.

This principle captures association and functional similarity. It cannot guarantee reference, truth, or logical equivalence. Two antonyms often share contexts because they describe the same dimension.

The historical implementations differ, and both compress corpus statistics into dense vectors. Which of the two routes produces better vectors was settled by systematic experiment twice. The two experiments returned opposite verdicts.

The first ran at ACL 2014, from Baroni and colleagues at the University of Trento, under a title that was also an instruction: Don't count, predict! The scale is in two sentences of their paper. §2.1 reports “In total, 36 count models were evaluated.” §2.2 reports “In total, we evaluate 48 predict models”. Their abstract does not hedge the outcome: “The results, to our own surprise, show that the buzz is fully justified, as the context-predicting models obtain a thorough and resounding victory against their count-based counterparts.”

The comparison was re-run in 2015, by Levy and colleagues in Transactions of the Association for Computational Linguistics. This time the hyperparameters were held level across the two families, rather than each system arriving with its own tuning. Their abstract states what they think the first result had actually measured: “We reveal that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves”. They then size the uncontrolled variable: “the best combination of hyperparameters improves up to 15.7 points beyond the vanilla setting, and over 6 points on average”. The conclusion is blunt: “It appears that selecting the right hyperparameter settings often has more impact than choosing the most suitable algorithm.” With tuning held fixed, the verdict flips: “Overall, there does not seem to be a consistent significant advantage to one approach over the other, thus refuting the claim that prediction-based methods are superior to count-based approaches.”

So the three families below are a map of objectives, not a ranking. Up to 15.7 points of the difference between any two of them may belong to the settings rather than to the mathematics.

The geometry becomes meaningful only relative to the training objective and corpus.

One of the two families was designed around exactly this calculation. GloVe arrived at EMNLP 2014, from Pennington and colleagues: a global log-bilinear regression model trained only on the nonzero elements of a word-word co-occurrence matrix, rather than on the entire sparse matrix or on individual context windows. The Stanford NLP Group's project page states the objective in one line: “The training objective of GloVe is to learn word vectors such that their dot product equals the logarithm of the words' probability of co-occurrence.” A cosine neighborhood is a reading of that arithmetic and of nothing else.

The evidence the authors offered that the resulting space has structure was a single benchmark number. Their abstract reads: “The model produces a vector space with meaningful substructure, as evidenced by its performance of 75% on a recent word analogy task.”

Table 2 of the same paper says what that 75% is. The line reads “GloVe 300 42B 81.9 69.3 75.0” — one model, 300 dimensions, 42 billion Common Crawl tokens. The total decomposes into 81.9% on the semantic questions and 69.3% on the syntactic ones, two halves of one benchmark that do not agree. The benchmark itself is a fixed list. §4.1 records that “The dataset contains 19,544 such questions, divided into a semantic subset and a syntactic subset.” Mikolov and colleagues, who built that set, give the split — “Overall, there are 8869 semantic and 10675 syntactic questions” — across five semantic and nine syntactic question types.

Change the corpus and the headline moves. On 6B tokens the same architecture scored “GloVe 300 6B 77.4 67.0 71.7” against the authors' own skip-gram run at “SG† 300 6B 73.0 66.0 69.1”. The geometry is a consequence of a counting objective, scored on one question set, at one width, on one corpus. It is not a discovery about language.

A useful audit includes successful structure and predictable blind spots.

One item on that list has a known repair. Represent each word as a bag of character n-grams, attach a vector to each n-gram, and sum them: a word gets a vector even when the training data never contained it. Bojanowski, Grave, Joulin and Mikolov, at Facebook AI Research, published that method in 2017. Their abstract says how widely they checked it: “We evaluate our word representations on nine different languages, both on word similarity and analogy tasks.” §4.4 names them — Arabic, Czech, German, English, Spanish, French, Italian, Romanian and Russian. An open vocabulary answers rare and unseen forms. It does nothing about sense conflation, which is a different failure with a different cause.

The last bullet below is the one that has been measured hardest. Off-the-shelf GloVe vectors carry the gender make-up of the U.S. labour force. Caliskan and colleagues reported that in Science in 2017: “By applying a second method that we developed, Word Embedding Factual Association Test, (WEFAT), we find that GloVe word embeddings correlate strongly with the percentage of women in 50 occupations in the USA in 2015.” The strength of that correlation, against 2015 U.S. Bureau of Labor Statistics data, is Pearson ρ = 0.90 with p-value < 10⁻¹⁸. A companion figure gives ρ = 0.84, p < 10⁻¹³, for androgynous names against 1990 U.S. census data. The same vectors also reproduced a human result: a flowers-versus-insects pleasantness effect of d = 1.50 (p < 10⁻⁷) in the embedding, against d = 1.35 (p < 10⁻⁸) for the human IAT. A second group, at Stanford, reached the same class of result from different data: “We integrate word embeddings trained on 100 years of text data with the U.S. Census to show that changes in the embedding track closely with demographic and occupation shifts over time.”

The most-quoted single demonstration is narrower than its fame. It was run on the 300-dimensional word2vec GoogleNews vectors, and not on all of them. Bolukbasi and colleagues, at NIPS in 2016, record the filter in §3: “Starting with the 50,000 most frequent words, we selected only lower-case words and phrases consisting of fewer than 20 lower-case characters… After this filtering, 26,377 words remained”. They then generated 150 analogies and showed each to 10 crowd-workers: “Overall, 72 out of 150 analogies were rated as gender-appropriate by five or more out of 10 crowd-workers, and 29 analogies were rated as exhibiting gender stereotype by five or more crowd-workers”. Fewer than one in five was judged stereotypical, on a 26,377-word slice of one file. Projecting occupations onto the she–he axis did track the crowd's stereotypicality ratings, at “Spearman ρ = 0.51”. The bias is real and it is quantified. The headline analogy is a narrower object than the quantity.

A pretrained file can be fast to adopt and easy to misuse.

Step one has published answers for the files most often reused, and the answers are on the download page itself. The Stanford NLP Group's GloVe page lists each release with its corpus, token count, vocabulary size, casing and dimensionality: “Wikipedia 2014 + Gigaword 5 (6B tokens, 400K vocab, uncased, 50d, 100d, 200d, & 300d vectors, 822 MB download)”, “Twitter (2B tweets, 27B tokens, 1.2M vocab, uncased, 25d, 50d, 100d, & 200d vectors, 1.42 GB download)” and “Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors, 2.03 GB download)”. Those lines are the provenance record. They are also longer than the list this lesson used to give: the page now carries two 2024 rebuilds as well, “2024 Dolma (220B tokens, 1.2M vocab, uncased, 300d vectors, 1.6 GB download)” and “2024 Wikipedia + Gigaword 5 (11.9B tokens, 1.2M vocab, uncased, 300d vectors, 1.6 GB download)”. An audit that names three files when the shelf holds more has already gone wrong at step one.

What step one looks like when a research group actually does it is on the record. Caliskan and colleagues wrote their file's provenance into the Methods of their Science paper before reporting a single result: “We use the largest of the four corpora provided—the 'Common Crawl' corpus obtained from a large-scale crawl of the web, containing 840 billion tokens (roughly, words). Tokens in this corpus are case-sensitive, resulting in 2.2 million different ones. Each word corresponds to a 300-dimensional vector derived from counts of other words that co-occur with it in a 10-word window.” Corpus, token count, casing, vocabulary size, dimensionality, window — six facts, one sentence each, before any claim.

The reason to write them down is that each one is a constraint on what the vectors can mean. A cased file distinguishes a proper noun from the common noun spelled the same way; an uncased one cannot. A 400K vocabulary and a 2.2M vocabulary disagree about which of your domain terms exist at all. And a vector file trained on 2014 Wikipedia carries a 2014 vocabulary, a 2014 distribution of senses, and 2014's answer to the question of who writes encyclopedias.

Choose five target words from a specialized domain. Retrieve nearest neighbors. Then search the corpus for contexts that plausibly created each relationship.

Separate synonymy, association, antonymy, names, templates, and errors. Propose whether domain retraining, contextual representations, or no embedding expansion is the safer response. Two habits from the papers above carry over directly. Write down the file's corpus, token count, casing and vocabulary size before you interpret anything, as Caliskan and colleagues did. And check whether any score you quote was produced under a constraint, as the analogy papers show it usually is.