GloCTM: Cross-Lingual Topic Modeling via a Global Context Space

AI summary

Overview

• Research area: Natural Language Processing, specifically Cross-Lingual Topic Modeling (CLTM) and multilingual representation learning.
• Technical level: Advanced — the paper assumes familiarity with variational autoencoders, topic models (LDA, neural topic models), Kullback-Leibler divergence, and Centered Kernel Alignment.
• Scope: The paper introduces GloCTM, a dual-pathway VAE-based topic model that builds a shared "Global Context Space" spanning input representations, latent topic proportions, and output topic-word distributions, and evaluates it on topic coherence, diversity, and cross-lingual classification across English-Chinese and English-Japanese datasets.

What This Paper Is About

Most cross-lingual topic models learn topic proportions (θ) and topic-word distributions (β) separately for each language and then try to bridge the two spaces with an auxiliary loss or a bilingual dictionary. That split lets the same topic index drift to different meanings in different languages — the paper's example is a topic that means "video games" in English but "footwear" in Japanese, illustrated with misaligned InfoCTM topics in English and Japanese. GloCTM's goal is to make cross-lingual alignment a structural property of the model rather than an after-the-fact correction, by constructing a unified semantic space across the whole pipeline and grounding it in multilingual pretrained embeddings.

Key Contributions

1. A dual-pathway framework for CLTM. GloCTM combines a global VAE pathway with language-specific local VAE pathways and leverages multilingual pretrained embeddings to guide topic learning across languages.
2. Polyglot Augmentation. A dynamic global bag-of-words construction that enriches each document with semantically related intra-lingual and cross-lingual neighbors, producing alignment-aware inputs in a joint vocabulary space.
3. A dual alignment strategy. Structural alignment through a unified topic-word matrix (the global decoder concatenates β⁽¹⁾ and β⁽²⁾ horizontally), plus representational alignment via a KL divergence loss between local and global posteriors and a Centered Kernel Alignment (CKA) loss tying the latent topic space to multilingual contextual embeddings.
4. Empirical validation. GloCTM is reported to consistently outperform strong baselines on topic diversity and cross-lingual alignment across multiple datasets and language pairs, with code released at https://github.com/tienphat140205/GloCTM.

Main Findings

• Highest Topic Quality (TQ) on all three datasets. TQ is defined as max(CNPMI, 0) × TU. GloCTM reaches 0.070 on EC News (versus InfoCTM at 0.041), 0.056 on Amazon Review (versus XTRA at 0.050), and 0.037 on Rakuten Amazon (versus InfoCTM at 0.028).
• Highest topic diversity (TU) across datasets. GloCTM records TU of 0.985 on EC News, 0.958 on Amazon Review, and 0.925 on Rakuten Amazon. The paper notes TU drops slightly relative to some settings and attributes this to minor overlaps among key meaningful words, an expected effect of tighter semantic vocabularies.
• Best coherence on the Japanese-English dataset. On Rakuten Amazon, GloCTM's CNPMI is 0.040, above the best baseline InfoCTM at 0.033. On Amazon Review its CNPMI is 0.058 (best) and on EC News 0.071, where SVD-LR (0.083) and u-SVD (0.082) score higher.
• Clear advantage in cross-lingual classification transfer. Using topic vectors as SVM features, GloCTM leads in the cross-lingual (-C) settings, with EN-C 0.763 and ZH-C 0.642 on Amazon Review, versus InfoCTM at 0.672 and 0.601.
• Competitive intra-lingual classification. GloCTM reports EN-I 0.818 and ZH-I 0.728 on Amazon Review, described as a slight but consistent advantage over remaining baselines in intra-lingual settings.
• Ablation confirms both losses matter. Removing the CKA loss degrades cross-lingual classification (EN-C 0.718); removing the KL loss causes a larger drop (EN-C 0.708); replacing the KL loss with a cosine similarity loss is less effective (EN-C 0.758), versus full GloCTM at 0.763.
• LLM-based evaluation favors GloCTM on the harder language pair. Using a 1-3 scale averaged over four runs and rounded, the models perform comparably on the two English-Chinese datasets, but on English-Japanese GloCTM produces substantially more strong, well-aligned topics and avoids the sharp fluctuations seen in competing models.
• Qualitative topic inspection. In a Media and Content topic, InfoCTM includes unrelated terms such as "sky" and "dream," SVD-LR shows noise and misalignment, and u-SVD mixes in sentiment, while GloCTM produces focused themes (e.g., "cinema," "film") aligned with their Chinese counterparts. In a Product Quality topic, GloCTM yields "timely" and "arrival," while SVD-LR introduces off-topic words such as "guns" and shows topic repetition.

Methodology in Plain English

GloCTM runs two kinds of encoders side by side. The local encoders see each language's own bag-of-words and capture language-specific nuance. The global encoder sees an enriched input built by Polyglot Augmentation: for every active word in a document, the method pulls in its top-k nearest neighbors within the same language (found by cosine similarity over word embeddings) and its top-k nearest neighbors across languages, then concatenates the resulting intra-lingual and cross-lingual parts into one vector over the combined vocabulary. Because a document about football in either language ends up containing overlapping features such as "soccer," "goal," "stadium," and "player," the global encoder does not have to infer alignment — the alignment is already in the input.

Alignment is then enforced in three places. At the output level, the global decoder uses a single topic-word matrix formed by concatenating the language-specific matrices, so each topic row spans both vocabularies; reconstructing multilingual text forces both halves of a row to mean the same thing. At the latent level, a KL divergence term pulls each local posterior toward the global posterior for the same document. Finally, a CKA loss aligns the inferred topic proportion matrix with multilingual pretrained language model embeddings, which is useful because CKA compares the geometry of two representation spaces even when their dimensions (K topics versus M embedding dimensions) differ. The whole model is trained end-to-end by minimizing the global and local ELBO losses plus weighted KL and CKA regularizers.

Datasets used: EC News (English-Chinese news articles across six categories), Amazon Review (English-Chinese reviews, cast as binary classification where five-star = 1 and others = 0), and Rakuten Amazon (Japanese Rakuten and English Amazon reviews, also binary). Evaluation uses CNPMI for cross-lingual coherence, TU for diversity, and TQ, all computed over the top 15 words per topic, plus SVM classification in intra-lingual (-I) and cross-lingual (-C) settings and LLM ratings of coherence and alignment. The paper does not report dataset sizes, the numeric value of k, or the values of the λ1 and λ2 weights.

Why This Matters

• Research impact. The paper argues that alignment in CLTM should be structural rather than an auxiliary patch, and it is the first work (per its own framing) to apply CKA in topic modeling to ground topic proportions in multilingual contextual embeddings. This reframes where alignment happens in the pipeline — input, latent space, and decoder — and offers a baseline that future CLTM work must address.
• Cross-cultural analysis at scale. Aligned topics let researchers compare how the same theme is discussed in different languages without parallel corpora.
• Multilingual content organization. Consistent topic indices across languages support unified tagging, search, and recommendation over mixed-language document collections.
• Low-resource and cross-lingual transfer. Strong cross-lingual classification results (EN-C 0.763 with GloCTM) suggest topic distributions learned in a high-resource language can serve as features for another language.
• Business intelligence and market research. The Rakuten Amazon setup, aligning Japanese and English reviews on identical rating tasks, mirrors how companies compare product perception across regional markets.

Industry relevance. The method depends on multilingual pretrained embeddings and offline neighbor lookup rather than scarce parallel corpora or static bilingual dictionaries, which is a more practical footing for production systems already running multilingual language models. The released code lowers the barrier to adoption for teams doing multilingual topic discovery.

Future Directions

• Extending beyond two languages. The formulation is written for two languages (L₁ and L₂) with a concatenated decoder matrix; generalizing the Global Context Space to many languages at once is an open question.
• Hyperparameter and coverage sensitivity. The augmented input depends on top-k neighbors and the weights λ1 and λ2, none of which are given numeric values in the reported content; understanding how sensitive results are to these choices, and to embedding quality, is a natural next step.
• Explaining the TU trade-off. GloCTM achieves the highest TU overall yet the paper notes a slight drop in some settings due to overlapping meaningful words; whether that overlap can be reduced without losing alignment is unresolved.
• Beyond topic quality to deeper evaluation. The paper's LLM evaluation uses a coarse 1-3 scale rounded over four runs; finer-grained human or LLM evaluation, and broader language pairs beyond English-Chinese and English-Japanese, would test how widely the alignment claim holds.

Target Audience

Researchers and graduate students working on topic modeling, multilingual NLP, or representation alignment; practitioners building multilingual search, recommendation, or content-analytics systems who need topics that stay consistent across languages; and readers interested in applying kernel-based representation similarity methods such as CKA to generative models.

Authors’ abstract

Cross-lingual topic modeling seeks to uncover coherent and semantically aligned topics across languages - a task central to multilingual understanding. Yet most existing models learn topics in disjoint, language-specific spaces and rely on alignment mechanisms (e.g., bilingual dictionaries) that often fail to capture deep cross-lingual semantics, resulting in loosely connected topic spaces. Moreover, these approaches often overlook the rich semantic signals embedded in multilingual pretrained representations, further limiting their ability to capture fine-grained alignment. We introduce GloCTM (Global Context Space for Cross-Lingual Topic Model), a novel framework that enforces cross-lingual topic alignment through a unified semantic space spanning the entire model pipeline. GloCTM constructs enriched input representations by expanding bag-of-words with cross-lingual lexical neighborhoods, and infers topic proportions using both local and global encoders, with their latent representations aligned through internal regularization. At the output level, the global topic-word distribution, defined over the combined vocabulary, structurally synchronizes topic meanings across languages. To further ground topics in deep semantic space, GloCTM incorporates a Centered Kernel Alignment (CKA) loss that aligns the latent topic space with multilingual contextual embeddings. Experiments across multiple benchmarks demonstrate that GloCTM significantly improves topic coherence and cross-lingual alignment, outperforming strong baselines.