Topic Models and Unsupervised Text Discovery

A model fitted to 300,000 NIH grant and journal-paper abstracts returned 500 topics. Two experts at the National Institute of Neurological Disorders and Stroke were then asked to grade the 148 topics that fell inside their expertise. Mimno and colleagues reported the verdict plainly in 2011: “Of the 148 topics selected for annotation, 90 were labeled as “good,” 21 as “intermediate,” and 37 as “bad.””

The bad ones were not bad in a single way. The annotators sorted the defects into named types and counted them: chained (23), intruded (21), random (3) and unbalanced (15). One of the categories the model returned was simply “grant mechanisms and publication types” — paperwork boilerplate, a perfectly coherent word list about nothing anyone wanted to study.

The same catalogue turned up from the other end two years later. Chuang and colleagues compared 10,000 topic model variants against 200 expert-provided concepts, and named the recurring failures junk, fused, missing and repeated topics.

The words came from the model. The grades, the labels and the failure taxonomy came from people.

The model is useful as an explanatory lens even when a different algorithm is deployed. Each document receives proportions over latent topics. That mixture influences which latent component generates each position. Each topic carries a distribution over vocabulary terms. Learning then estimates the topic–word and document–topic distributions from the observed counts. Only after that do people examine terms, documents, stability and usefulness.

Only the fourth stage is arithmetic. The first three are assumptions about how the text was produced. The fifth is the work the two annotators in the previous section were doing when they graded 148 word lists as good, intermediate or bad. The generative story explains what the numbers estimate. It does not certify that the estimate corresponds to anything a domain expert would recognise.

They may reveal related structure while optimizing different objects. Probabilistic topic models represent documents as mixtures of latent word distributions: interpretable word lists, an explicit mixture model, bag-of-words assumptions, and a topic count and priors that matter. Matrix factorization approximates a document–term matrix with lower-rank factors — a linear algebra view, comfortable with TF–IDF, sensitive to signs and scaling, and still requiring interpretation of each factor. Embedding clustering groups contextual or document vectors under a chosen distance and algorithm. It captures semantic similarity, depends on the geometry of the representation, imposes cluster shape assumptions, and leaves the labels human.

The fourth family — projecting high-dimensional representations into two dimensions — is the one most often mistaken for evidence. Wattenberg and colleagues took it apart in Distill in 2016, using controlled examples whose structure they already knew. On the reading everyone performs on such a plot, measuring how far apart the blobs sit: “The basic message is that distances between well-separated clusters in a t-SNE plot may mean nothing.” Size gives nothing either — “you cannot see relative sizes of clusters in a t-SNE plot”. And the clusters need not exist at all. 500 points drawn from a unit Gaussian in 100 dimensions, data with no group structure in it whatever, come out as apparently dramatic clusters at perplexity 2.

The distortion has since been measured rather than demonstrated. Chari and Pachter published the measurements in 2023. In 2-D t-SNE and UMAP embeddings, so few of each point's 30 nearest neighbours survived the projection that the Jaccard distance from the ambient space averaged consistently above 0.7, and max/min distance ratios were distorted 4- to 200-fold. They then built arbitrary “Picasso” embeddings shaped like an elephant or a flower, and those scored comparably to t-SNE and UMAP on the same distance correlations. Reduction to two dimensions, their abstract states, “inevitably induces significant distortion of high-dimensional datasets”. A picture is a useful way to look for something to check. It is not the check.

A topic whose top words frequently co-occur may look coherent while representing a website template, citation style, or demographic proxy. Conversely, a useful operational theme can contain diverse vocabulary and score poorly.

Combine automated coherence with document inspection, intruder tasks, stability across runs, coverage, distinctiveness, and a defined user task. Report disagreements among reviewers.

That assumption was tested directly in 2009. Chang and colleagues built two human tasks. In word intrusion, “subjects are presented with a set of words and asked to select the word which does not belong with the others”. In topic intrusion, the subject must find “a topic that was not associated with the document by the model”. Both were set against held-out likelihood. The result inverted the usual reading: “Surprisingly, topic models which perform better on held-out likelihood may infer less semantically meaningful topics.” The paper is called Reading Tea Leaves.

In 2021, Hoyle and colleagues asked whether the field had absorbed that. Surveying forty neural topic-modelling papers, they found 0 of 40 — 0% — reporting a human evaluation of coherence, only 28% naming the reference corpus used to compute NPMI, and 30% preprocessing inconsistently across their own datasets. Then they ran the comparison themselves, on New York Times and Wikipedia text, with at least fifteen crowdworkers per topic. NPMI declared D-VAE the winner. The human ratings favoured G-LDA. Their abstract closes on the consequence: “Automated evaluations declare a winning model when corresponding human evaluations do not, calling into question the validity of fully automatic evaluations independent of human judgments.”

A separate benchmark reached the same verdict from a different direction, testing algorithms against a known Wikipedia clustering in PLOS ONE in 2022. Its finding is that “coherence (at least the way it is currently measured in the literature) does not correlate with accuracy”, and that “clustering results that yield coherent topics do not necessarily guarantee correctness”. Two independent groups, two designs, one conclusion. The score and the reader disagree often enough that the score cannot stand in for the reader.

The output should support a question rather than merely produce colorful labels. State the exploration goal first — archive navigation, issue discovery, sampling, or trend hypotheses. Build lexical and embedding baselines, comparing simple term clusters, metadata groups and nearest documents. Sweep defensible settings: topic count, preprocessing, seeds, time periods. Inspect words and documents together, naming topics only after reading representative and boundary cases. Then validate actionability and risk, testing whether the organization helps users without reifying sensitive or misleading categories.

The sweep is the step teams skip, and it is not a formality. Schofield and Mimno moved a single preprocessing switch with everything else held fixed: rule-based, truncation, S-removal and dictionary-based stemmers, plus a no-stemmer control, across four corpora — arXiv articles from early 2015, New York Times articles from 2007, IMDb biographies and Yelp reviews. For every corpus, treatment and K in {10, 50, 200} they trained at least nine MALLET LDA models. Their conclusion, in 2016: “Despite their frequent use in topic modeling, we find that stemmers produce no meaningful improvement in likelihood and coherence and in fact can degrade topic stability.”

The degradation was measured, not impressionistic. For IMDb, NYT and Yelp at 200 topics, and for NYT at 50, the variation of information between the untreated corpus and the Krovetz-stemmed corpus was significantly greater than the untreated corpus's variation with itself. A step most pipelines apply by default, and that no one records in a methods section, moves the solution further than refitting the same corpus does.

Fit or simulate solutions with a smaller and a larger number of topics. For each, identify stable themes, splits, merges, boilerplate components, and documents with mixed membership.

Ask a domain expert to complete a document-intrusion task and explain which resolution supports a real decision about navigating or analysing the corpus.

Doogan and Buntine ran that protocol in 2021 and published what happened. They produced 390 topics — 130 per model–dataset combination — with LDA and MetaLDA at K = 10–60 over the Auspol-18 Twitter corpus, itself built from a 123,629-tweet subset of the 1,830,423 tweets carrying the hashtag #Auspol. Four subject-matter experts were asked to label them, yielding 3,120 labels. No coherence measure showed a significant correlation with the number of raters able to label a topic. One LDA topic ranked 9th of 60 on NPMI coherence — “red, wear, flag, blue, gold, black, tape, tie, green, iron” — and could not be labelled at all. The authors put it drily: “Interestingly, the SMEs reported several topics they could not label despite their high coherence scores.” A near-top-ranked topic that four experts cannot name is the whole argument for asking them.

The corpus side of Reading Tea Leaves is worth copying too. One of its corpora was 8447 articles from the New York Times, published between 1987 and 2007, with a vocabulary of 8269 unique types and around one million tokens. The other was a sample of 10000 Wikipedia articles, with 15273 types and three million tokens. Proper nouns were stripped with a part-of-speech tagger for the benefit of the human subjects; without that, “success in early experiments required too much encyclopedic knowledge”. 20 per cent of each corpus was held out for the predictive measures, and the intrusion judgements were collected on Amazon Mechanical Turk. The disagreement between those two families of evidence is the finding. Reproducing it on your own archive is the exercise.