TopicMerge: A Divide and Conquer Approach to Topic Modeling

AI summary

Overview

Research area: Natural Language Processing — topic modeling, text clustering, and the integration of Large Language Models (LLMs) into unsupervised text analysis pipelines.

Technical level: Intermediate. The paper is readable by anyone familiar with document embeddings and clustering, but the three-stage pipeline, the merging algorithm, and the metric set (P, P1, ARI, NMI) assume some background in NLP evaluation.

Scope: The paper proposes TopicMerge, a three-stage pipeline that over-segments a corpus with efficient clustering and then uses an LLM to hierarchically merge and relabel the resulting micro-clusters, evaluated on the Bills and Wiki benchmarks.

What This Paper Is About

Topic modeling research has split into two camps: embedding-based methods that scale well but score lower on standard clustering metrics, and LLM-based methods that reach state-of-the-art clustering quality but cost far too much because they query the model per document. TopicMerge's goal is to get the clustering quality of the LLM-based camp at a fraction of its cost, by having an LLM reason about a small number of cluster-level decisions instead of a large number of document-level ones.

Key Contributions

1. The TopicMerge pipeline. A three-stage design: (1) an initial clustering stage that deliberately over-segments the corpus into fine-grained micro-clusters, (2) an initial labeling stage that gives each micro-cluster a label and description, and (3) an LLM-guided cluster merging stage that hierarchically consolidates them.

2. A context-aware LLM merging algorithm. The LLM acts as a semantic arbiter over candidate cluster pairs, receiving local neighboring clusters, more distant representative clusters, and example topic labels from the domain as context, plus a self-consistency requirement of two-fold agreement and a decision cache to avoid redundant queries.

3. State-of-the-art clustering results. On the Wiki dataset TopicMerge reports an ARI of 0.71, NMI of up to 0.74, and P1 of 0.73; on Bills it reports the best NMI (0.53) and ARI (0.42) among all compared approaches, with P1 of 0.57.

4. Over 92% fewer LLM prompts. TopicMerge required 645 prompts on Bills and 562 on Wiki, versus the 8,024+ and 15,242+ lower bounds reported for PromptTopic, TopicGPT, and LiSA.

Main Findings

• TopicMerge is the top performer on Wiki. It reports ARI 0.71, NMI up to 0.74, and P1 0.73, ahead of all baselines including the strong LLM-based LiSA (ARI 0.66, NMI 0.71, P1 0.74) and TopicGPT (ARI 0.60, NMI 0.70, P1 0.74).

• TopicMerge leads on the harder Bills dataset. It achieves the best NMI (0.53) and ARI (0.42) across all approaches with P1 of 0.57. Note that EDTM posted the highest Bills P1 (0.58), but its ARI and NMI are not reported in Table 1.

• Both LLM backbones work; the newer one is better. With Gemini 2.5 Flash Lite, TopicMerge scored Bills P1 0.57 / ARI 0.42 / NMI 0.53 and Wiki P1 0.72 / ARI 0.71 / NMI 0.73. With Gemini 3.1 Flash Lite, Bills was P1 0.57 / ARI 0.42 / NMI 0.52 and Wiki was P1 0.73 / ARI 0.71 / NMI 0.74. The authors report that Gemini 3.1 Flash Lite was noticeably more consistent at arbitrating merges and synthesizing multi-level abstractions.

• Prompt cost is dramatically lower. TopicMerge used 645 prompts on Bills (8,024 documents) and 562 on Wiki (15,242 documents), versus the 8,024+ and 15,242+ lower bounds the authors report for PromptTopic, TopicGPT, and LiSA — a reduction of over 92%.

• Context clusters matter. Removing context caused Wiki ARI to fall from 0.71 to 0.63, Wiki P1 from 0.72 to 0.70, and Wiki NMI from 0.73 to 0.70. On Bills the drops were smaller (P1 0.57 to 0.56, ARI 0.42 to 0.41, NMI unchanged at 0.53).

• Starting fine-grained matters. On Wiki, starting from 80 clusters (initial purity 0.88) yielded a final ARI of 0.71, while starting from 30 clusters (initial purity 0.79) dropped ARI to 0.50. Bills performance was stable at high initial cluster counts (ARI 0.42–0.43) before falling at lower granularities. On Wiki, the drop between 60 and 40 initial clusters was also notable (ARI 0.71 to 0.57).

• Traditional and simple embedding baselines lag. On Bills, LDA scored 0.52/0.31/0.42 and BERTopic 0.41/0.18/0.37 (P1/ARI/NMI); NMF was weakest at 0.24/0.04/0.15.

Methodology in Plain English

The pipeline works in three stages.

Stage 1 — Clustering. Documents are embedded (256-dimensional Gemini clustering embeddings), normalized, then reduced with PaCMAP to 64 dimensions for the clustering step. Agglomerative clustering with cosine distance and complete linkage produces an initial set of 80 micro-clusters. The key trick is deliberate over-segmentation: a Silhouette score is used only to establish a lower bound on plausible topic count, and the actual cluster count is set well above it. The goal is many small, semantically pure clusters rather than a few broad, mixed ones.

Stage 2 — Labeling. For each micro-cluster, Farthest Point Sampling selects a diverse subset of documents (32 per cluster) rather than documents near the centroid, so the sample covers the cluster's full semantic range. An LLM turns that sample into a short label and a description, with a few example topic labels from the dataset supplied to calibrate granularity.

Stage 3 — Merging. This iterates until no new merges are approved. Each iteration has four phases:

1. Candidate generation: build an augmented Reciprocal Nearest Neighbor graph over cluster centroids (5 neighbors), adding minimum-cost bridge edges so the graph stays connected. This avoids comparing every pair of clusters.
2. LLM arbitration: for each candidate pair, the LLM sees the two clusters plus 3 local and 5 global context clusters and 2–3 example topic labels — labels and descriptions only, not raw text. The example given in the paper: "Soccer" and "Basketball" may not look mergeable alone, but a neighboring "Baseball" cluster supplies the context to infer "Sports."
3. Purity ranking and set selection: approved pairs get a local silhouette-based purity score; the list is sorted and a greedy maximal independent set is chosen so no cluster merges twice in one pass, preventing cascading instability.
4. Merge and relabel: merges execute via a Union-Find structure, and the LLM relabels each newly formed super-cluster using the source labels, descriptions, and FPS-sampled documents from the combined set.

This makes the whole thing "divide and conquer": clustering handles global structure, the LLM handles local semantic judgment, and the two meet at the cluster level rather than the document level.

Why This Matters

TopicMerge reframes the cost problem in LLM-based topic modeling. Instead of asking "how do we make per-document LLM calls cheaper," it asks "how do we make the LLM's unit of reasoning a cluster instead of a document." That shift is what produces the over-92% prompt reduction while still reporting top clustering scores. It also gives the LLM something existing LLM-based topic models struggle with: a global view of the corpus, supplied by the clustering stage, which the LLM then refines locally.

Potential real-world applications:

• Legislative and policy analysis: the Bills dataset (8,024 summaries from the 110th–114th U.S. Congresses) is the kind of corpus where analysts need thematic groupings across thousands of documents.
• Encyclopedic and knowledge-base organization: the Wiki dataset (15,242 Wikipedia articles, 15 high-level topic labels) mirrors the content-organization problem faced by knowledge bases and documentation sites.
• Large-scale document triage: any archive — news, legal filings, internal reports — too large for per-document LLM processing but where quality labels matter.
• Literature and research synthesis: organizing large paper collections thematically without prohibitive inference cost.

Industry relevance: The efficiency argument is the practical one. A corpus of 15,242 documents costing 562 LLM prompts instead of 15,242+ changes the economics of deploying topic modeling in production, especially for organizations that want LLM-quality topic labels but cannot absorb per-document inference bills. The code is publicly available at https://github.com/pnnl/TopicMerge.

Future Directions

• Open-source LLM backbones. The authors identify reliance on a closed-source LLM as a limitation, citing data contamination on benchmark datasets, output stochasticity, and reproducibility challenges, and suggest adapting the pipeline for open-source models.
• Short-text corpora. The paper notes that performance is tied to initial embedding quality, which is a problem for very short texts where embeddings are less discriminative. Integrating specialized short-text embedding techniques is proposed.
• Where the granularity floor actually lies. The ablations show TopicMerge is robust above a minimum initial cluster count but drop sharply below it (Wiki ARI 0.71 at 80 clusters versus 0.50 at 30). How to set that threshold automatically per corpus is an open question the paper does not resolve.
• Bias auditing. The ethical considerations section notes that inherited LLM bias can surface in generated topic labels and merge decisions, and that the opacity of these models makes the source of such bias difficult to trace.

Target Audience

This paper is most useful to NLP researchers and graduate students working on topic modeling, text clustering, or hybrid LLM-plus-classical pipelines; to practitioners who need scalable topic discovery over corpora in the thousands to tens of thousands of documents; and to engineers evaluating whether LLM-based topic labeling is affordable at their data scale. Readers wanting the full algorithmic detail, the exact prompts, and the complete final topic labels should consult Algorithm 1 and Appendices A through D of the paper itself.

Authors’ abstract

Recent advancements in topic modeling have split into two main paths: efficient embedding-based methods that offer scalability but are often outperformed on standard clustering metrics, and Large Language Model (LLM) approaches that achieve state-of-the-art clustering performance but at the cost of prohibitive computational expense and poor scalability. This paper introduces TopicMerge, a novel multi-stage pipeline that bridges these approaches to achieve both state-of-the-art clustering performance and computational efficiency. TopicMerge first employs an efficient clustering pipeline to deliberately over-segment a document corpus into a large set of fine-grained, semantically pure micro-clusters, which are then given initial semantic labels. It then introduces a novel, iterative merging algorithm where an LLM, guided by local neighborhood context, arbitrates on consolidating candidate pairs. These approved merges are then selectively executed and semantically relabeled by the LLM, hierarchically building broader, more coherent topics. Extensive experiments on the Bills and Wiki datasets show that TopicMerge achieves state-of-the-art clustering performance while reducing required LLM calls by over 92% compared to document-level methods, offering a practical and scalable solution.