Text and Embedding Clustering

Every failure below has been measured on a named corpus, with a published number. None of them looks wrong on the plot. That is why text clusters have to be read through documents, terms and retrieval behavior instead of pictures.

Cluster raw term counts over a support inbox and the groups that come back can be short, medium and long tickets. Length is the loudest signal the representation carries, so length is what the algorithm returns. Replace counts with normalized TF–IDF and issue families appear. Hand the same text to a dense encoder and several operationally distinct policies merge into one region. Nothing about the clustering code changed in any of those three runs.

Text clustering is representation-sensitive because words, sequences and semantic embeddings preserve different relations. A modern encoder does not remove that choice. It hides it inside a trained objective. The rest of this lesson prices each layer of the choice with published measurements rather than assurances: the geometry of an encoder's raw output, the share of a canonical corpus that is boilerplate, the language axis inside a multilingual model, the standing of lexical matching against dense retrieval, and the gap between a model's automatic score and what a human reads in its topics.

The representation determines whether clusters emphasize vocabulary, latent co-occurrence, or learned semantic proximity. A cluster interpretation can never recover information the representation threw away.

Sparse lexical vectors — counts, n-grams, or TF–IDF — preserve explicit vocabulary evidence, so every membership can be traced back to terms a reviewer can read. Linear latent vectors, produced by truncated SVD, compress sparse co-occurrence into dense signed components. That reduces sparsity at the cost of the rare precise term. Neural embeddings map text into learned semantic or task-influenced spaces whose axes were fixed by a training objective you did not choose. Document aggregation sits underneath all three: chunking, pooling and metadata decide how a long text becomes one vector, and therefore decide what is being clustered at all.

Representation, chunking, clustering and naming should be evaluated separately, because each one can produce the whole result on its own.

1. Define the text unit. Choose message, paragraph, document, conversation, or fixed window according to the action the clusters will drive.

2. Remove structural noise. Handle boilerplate, signatures, markup, duplicates and language metadata explicitly, and treat this as a measurable step rather than hygiene. The scikit-learn User Guide sizes the effect on the canonical text benchmark. On its four-newsgroup example from 20 Newsgroups (alt.atheism, talk.religion.misc, comp.graphics, sci.space), the same multinomial Naive Bayes model scores a macro F1 of 0.88213 on the raw text, 0.77310 once headers, footers and quoted blocks are stripped from the test set, and 0.76995 when they are stripped from training too. The guide's own reading of the raw score is blunt: “With such an abundance of clues that distinguish newsgroups, the classifiers barely have to identify topics from text at all, and they all perform at the same high level.” The documented remedy is the loader argument remove=('headers','footers','quotes'). Eleven points of F1 were never topic structure.

3. Compare representations. Benchmark lexical, latent and neural spaces against domain contrast pairs you wrote down before looking at the clusters.

4. Fit several cluster models. Use cosine-compatible centroid, density, or hierarchical approaches where appropriate, and keep the disagreements rather than the prettiest run.

5. Name from evidence. Review exemplars, discriminative terms, boundary cases and contradictions before assigning labels. Do not delegate the judgement to an automatic coherence score: Doogan and Buntine re-tested those measures against human open-labelling in applied settings in 2021, and reported against them.

6. Test downstream use. Measure routing, retrieval, taxonomy coverage, or analyst discovery rather than plot beauty.

Pipelines compare both instead of treating one as universally superior, and the benchmark record supports that caution.

TF–IDF plus cosine groups documents through weighted term overlap: transparent vocabulary evidence, strong on domain-specific terminology, weak on paraphrase, sensitive to tokenization and rare terms. LSA plus clustering uses truncated SVD to compress term-document structure. It reduces sparsity and captures broad co-occurrence, but produces signed latent components and can blur the rare precise term. Dense sentence embeddings group learned semantic representations: many paraphrases are recognized, generic hubs appear, training-domain bias is inherited, and language and domain evaluation is required.

The trade-off has been measured rather than argued. The BEIR benchmark, published in 2021, evaluated 10 retrieval systems across 18 datasets. Its abstract reports that “Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs.” Four researchers pushed on the same seam later that year. Building EntityQuestions out of Wikidata facts, they measured DPR at 49.7% top-20 accuracy against BM25's 71.2% on average, with the gap on some question patterns reaching 60% absolute. Lexical evidence is not a legacy baseline you have outgrown.

The dense column carries a second, quieter problem. Cosine over an encoder's raw output is not a neutral distance. Kawin Ethayarajh measured the geometry of contextual embeddings in 2019 and found no layer isotropic: for GPT-2 the average cosine similarity between uniformly randomly sampled words is roughly 0.6 in layers 2 through 8, and rises to almost 1.0 in the last layer. His first finding says it outright: “The anisotropy in GPT-2’s last layer is so extreme that two random words will on average have almost perfect cosine similarity!” Timkey and van Schijndel traced the distortion in 2021, independently, to just 1–3 'rogue' dimensions that dominate cosine similarity until the dimensions are standardised. A high cosine similarity means nothing until you know what two unrelated documents score in the same space.

A topic name like “billing problem” should come from sampled documents, distinguishing terms and domain review. The algorithm supplies memberships, not the title and not the explanation. The automatic score you would rather trust instead has been tested against human readers, and it lost. Five researchers ran word-intrusion and topic-intrusion studies on Amazon Mechanical Turk in 2009, over 8,447 New York Times articles from 1987–2007 with an 8,269-type vocabulary, and over 10,000 Wikipedia articles. The model with the better held-out likelihood, CTM, produced the topics people read worst. Their abstract records it as a surprise: “Surprisingly, topic models which perform better on held-out likelihood may infer less semantically meaningful topics.” Twelve years later Doogan and Buntine reached the same verdict against automated coherence measures on specialised collections.

Version tokenizer, encoder, chunking, preprocessing and model together. A representation update can rewrite the topic map even when the clustering code is unchanged. A lineage record that names only the algorithm will not tell you which of the two happened.