TopiCLEAR: Adaptive embedding clustering for interpretable topic discovery from short texts

AI summary

Overview

Research area: Natural Language Processing — topic discovery (topic modeling and embedding-based clustering) for short and informal texts.

Technical level: Intermediate. The paper assumes familiarity with document embeddings and clustering algorithms, but the underlying framework is conceptually simple and is described without heavy derivation.

Scope: The paper proposes and evaluates TopiCLEAR, a framework that discovers human-interpretable topics by clustering document embeddings in a low-dimensional space obtained through adaptive dimensionality reduction.

What This Paper Is About

Topic discovery — automatically identifying the themes in a large text collection — traditionally relies on topic models such as LDA, which represent topics as lists of words and struggle with short or informal texts where word co-occurrence is sparse. A newer line of work instead clusters document embeddings from pre-trained language models, but the paper argues that the design principles connecting the geometry of embedding spaces to human-interpretable topic organization remain unclear. TopiCLEAR addresses this by testing the hypothesis that human-interpretable topics correspond to low-dimensional geometric structures in embedding spaces, and by iteratively projecting documents into such a low-dimensional topic space before clustering them.

Key Contributions

1. The authors introduce TopiCLEAR (Topic discovery by CLustering Embeddings with Adaptive dimensionality Reduction), a simple framework that identifies topics by clustering documents in a low-dimensional topic space obtained via Adaptive Dimensionality Reduction (ADR). The framework requires neither heuristic preprocessing nor gradient-based training and does not require GPUs.

2. They provide an evaluation approach for topic discovery that combines quantitative measurement of alignment with human-labeled data (using AMI and ARI) and qualitative assessment in the absence of ground-truth labels.

3. They benchmark TopiCLEAR against five topic-model baselines (LDA, BTM, ProdLDA, ETM, CTM) and BERTopic as a reference method across four datasets spanning formal and informal texts, including an ablation study covering embedding choice, clustering method, ADR, and preprocessing.

4. They present a case study on Twitter data showing that TopiCLEAR recovers both human-annotated topic structure and coherent sub-topic structure more interpretably than LDA.

Main Findings

• Strongest quantitative agreement: TopiCLEAR(GMM) achieved the highest AMI of all methods on all four datasets: 0.588 ± 0.006 (20News), 0.489 ± 0.000 (AgNewsTitle), 0.462 ± 0.000 (Reddit), and 0.382 ± 0.004 (TweetTopic), with ARI results reported in the appendix.

• The advantage grows on short and informal texts: LDA performed relatively well on 20News (AMI 0.378 ± 0.014, ranking 4th) but deteriorated to 0.042 ± 0.011 on AgNewsTitle and 0.037 ± 0.018 on TweetTopic. BTM, designed for short texts, reached 0.372 ± 0.036 on AgNewsTitle but remained below AMI 0.3 on Reddit (0.284 ± 0.012) and TweetTopic (0.181 ± 0.019).

• Consistent performance across document lengths: When 20News was split into word-count groups [1–150], [151–250], [251–400], [401+] and Reddit into [1–100], [101–200], [201–500], [501+], TopiCLEAR(GMM) achieved the highest AMI and ARI in every group.

• ADR improves results across the board: For k-means, AMI rose from initial to converged values of 0.553→0.600 (20News), 0.451→0.478 (AgNewsTitle), 0.407→0.452 (Reddit), and 0.329→0.377 (TweetTopic). For GMM, the corresponding gains were 0.562→0.588, 0.452→0.489, 0.417→0.462, and 0.357→0.382. ADR also improved ARI on the two formal-text datasets but its ARI effect was not significant for Reddit and TweetTopic.

• PCA preprocessing matters: Using the original embeddings gave AMI of 0.564 (20News), 0.448 (AgNewsTitle), 0.411 (Reddit), and 0.361 (TweetTopic), compared with 0.588, 0.489, 0.462, and 0.382 at D = 64. D = 128 was slightly better on 20News (0.592) but the gains were limited and dataset-dependent, so D = 64 was chosen as a practical trade-off.

• Embedding choice matters most for informal text: all-MiniLM-L6-v2 performed best on Reddit (0.462) and TweetTopic (0.382), while intfloat/e5-small-v2 was close on the news datasets (0.586 on 20News, 0.457 on AgNewsTitle). paraphrase-MiniLM-L6-v2 was weakest on all four (0.533, 0.419, 0.347, 0.306). TopiCLEAR outperformed all baselines regardless of embedding.

• Matching K helps, and C_v is an approximate guide: Except for TweetTopic, AMI was highest when K matched the true value — for example, 0.588 at K = 20 on 20News, 0.489 at K = 4 on AgNewsTitle, and 0.462 at K = 5 on Reddit. On TweetTopic, AMI was highest at K = 3 (0.504) rather than the true K = 6 (0.382). The value maximizing C_v provides an approximate estimate of the true K but tends to overestimate it (e.g., C_v peaked at 0.584 at K = 40 for 20News and 0.452 at K = 13 for Reddit).

• Competitive runtime on CPU only: All experiments ran on a 2023 14-inch MacBook Pro (Apple M3 Pro, 11-core CPU, 18 GB unified memory) using the CPU only. At 100,000 documents, TopiCLEAR(k-means) took 9.40 ± 0.75 seconds and TopiCLEAR(GMM) took 49.4 ± 2.2 seconds, both faster than BERTopic (74.9 ± 4.5 seconds) and comparable to LDA (11.3 ± 0.6) and BTM (12.1 ± 1.4). ProdLDA, ETM, and CTM all exceeded the 7,200-second cutoff at that scale.

• BERTopic comparison is not controlled: BERTopic automatically determines the number of topics via HDBSCAN, whereas the other methods require K in advance, so the authors treat BERTopic as a reference method and note substantial discrepancies between the topic granularity BERTopic infers and that defined by human annotations.

• Qualitative advantage over LDA on Twitter: On TweetTopic with K = 6 and the same number of topics for both methods, greedy matching by cosine similarity produced low similarity scores (≤ 0.14), indicating the two methods found different topics. Most of the words highlighted for the human-annotated "pop culture" topic appeared in a single TopiCLEAR topic (Topic 5), whereas in LDA those words were scattered across three topics (Topics 4, 5, and 6). According to annotator classification, 98.3% and 99.5% of tweets in TopiCLEAR Topics 1 and 3 were "sports & gaming," and 98.0% of tweets in Topic 5 were "pop culture."

• Sub-topic structure within sports: TopiCLEAR Topic 1 focused on internationally popular sports such as football (Premier League) and UFC, while Topic 3 centered on sports popular in the United States, such as the NBA and NFL. Topic 6 consisted mainly of personal messages about holidays such as Easter; annotators classified it as a mixture of multiple topics, but its content remained interpretable.

Methodology in Plain English

TopiCLEAR takes a document collection and a specified number of topics K and proceeds in three steps.

First, every document is converted into a fixed-size vector using Sentence-BERT — specifically the all-MiniLM-L6-v2 model, producing 384-dimensional embeddings.

Second, an initial clustering is produced. The embeddings are compressed with principal component analysis (PCA) to D = 64 dimensions, normalized, and then reduced again to K − 1 dimensions, which is the maximum number of dimensions that Fisher Discriminant Analysis can extract from K topic labels. A Gaussian Mixture Model then assigns each document to the topic with the highest posterior probability.

Third, the clustering is iteratively refined using Adaptive Dimensionality Reduction. Each round treats the current topic assignments as pseudo-labels: Fisher Discriminant Analysis learns a projection that maximizes the ratio of between-topic to within-topic scatter, the documents are re-projected into that low-dimensional space, and the Gaussian Mixture Model reclusters them. This alternation repeats until the assignments stop changing, with a maximum of 10 iterations in all experiments (the ADR variant of Ding and Li, which uses k-means instead of GMM, was reported to converge in about 10 iterations).

The framework is generic: the authors also report a k-means variant, TopiCLEAR(k-means). Evaluation uses Adjusted Mutual Information and Adjusted Rand Index to measure agreement with human-provided topic labels. The authors deliberately exclude coherence metrics from their main evaluation, since those metrics may not be robust to label noise, and they exclude prompt-driven LLM approaches from the baselines to keep the comparison reproducible. For qualitative analysis, they compare top words and representative documents, using delta TF-IDF to select words characteristic of the human-annotated "pop culture" topic.

Why This Matters

Impact on research: The paper offers a testable design principle — that interpretable topics live in low-dimensional geometric structures of embedding space — and shows that this principle, operationalized through iterative projection, yields clusters that agree more closely with human annotation than both classical topic models and an established embedding-clustering method. It also argues for evaluating topic discovery through alignment with human labels rather than relying on top-word coherence metrics, which the cited literature has found can fail to correlate with human interpretability on tweet corpora.

Real-world applications:

• Social media analytics, where posts are short, full of slang and misspellings, and where the paper's Twitter case study demonstrates recovery of human-annotated topic structure and coherent sub-topics such as distinct sports communities.
• Analysis of online news at scale, including headline-only data, since AgNewsTitle consists of 119,794 article titles averaging 5.2 words, each.
• Opinion and public-perception monitoring, the application areas the introduction cites for topic discovery generally.
• Analysis of user discussion forums such as Reddit, where the dataset is 37,933 parent comments and replies across five subreddit-based categories.

Industry relevance: The method needs no GPU, no gradient-based training, and no heuristic preprocessing such as stopword removal, and it processes 100,000 documents in under ten seconds with the k-means variant on a laptop CPU. That makes it practical for practitioners who want fast, interpretable topic structure over large volumes of short user-generated text without a specialized training pipeline. Its runtime advantage over ProdLDA, ETM, and CTM, all of which exceeded the 7,200-second cutoff at 100,000 documents in this setup, is directly relevant to production-scale triage.

Future Directions

• Estimating K without ground truth: The ablation shows AMI is highest when K matches the true value on three of four datasets, but the best C_v-based estimate tends to overestimate K. Better principled ways to select the number of topics remain open.

• Understanding why ARI gains from ADR are weaker on informal text: ADR improved ARI on 20News and AgNewsTitle but the effect was not significant on Reddit and TweetTopic, a gap the paper reports without fully resolving.

• Comparing with prompt-driven LLM approaches: The authors explicitly set these aside as related work because they use text as both input and output, leaving open how representation-based methods like TopiCLEAR compare against them under a fair protocol.

• Extending and stress-testing the topic representation hypothesis: TopiCLEAR is presented as a generic framework that can be extended to other clustering methods including k-means; how far the low-dimensional-structure hypothesis holds under other embedding models, languages, and domain shifts is not resolved here.

Target Audience

Researchers and practitioners in NLP, text mining, and computational social science who work with short or informal text such as social media posts and news headlines, and who need topic discovery that is interpretable, fast on CPU, and reproducible. It is also relevant to methodologists interested in the link between embedding geometry and human topic judgments, and to those designing evaluation protocols for unsupervised topic discovery where ground-truth labels may be absent.

Authors’ abstract

Topic discovery is a fundamental technique for text mining that identifies abstract topics within large document collections. A recent approach to topic discovery is to cluster document or sentence embeddings, typically obtained from pre-trained language models, and represent each cluster as a topic. Despite their strong empirical performance, the design principles linking the geometry of embedding spaces to human-interpretable topic organization remain unclear. Clarifying the relationship between these geometric structures and topic interpretability is therefore a key challenge in topic discovery. In this study, we propose TopiCLEAR (Topic discovery by CLustering Embeddings with Adaptive dimensionality Reduction), a simple framework that integrates document embeddings with iterative clustering based on adaptive dimensionality reduction. TopiCLEAR is guided by the hypothesis that human-interpretable topics correspond to low-dimensional geometric structures in embedding spaces and leverages adaptive dimensionality reduction to identify them. We evaluate topic quality using a document-level approach that combines quantitative evaluation based on human-labeled data with qualitative assessment in the absence of ground-truth labels. Experiments on four benchmark datasets, covering both formal and informal texts, show that TopiCLEAR consistently achieves strong agreement with human annotations, particularly for short and informal texts. Furthermore, a case study on Twitter data demonstrates that TopiCLEAR produces more interpretable topics than Latent Dirichlet Allocation (LDA), recovering both human-annotated topic structure and coherent sub-topic structure. These results highlight the effectiveness of clustering documents in a low-dimensional topic space for topic discovery.