Evaluating Cross-Lingual Classification Approaches Enabling Topic Discovery for Multilingual Social Media Data

AI summary

Overview

• Research area: Natural Language Processing, specifically cross-lingual text classification and topic modelling for multilingual social media analysis.
• Technical level: Intermediate. The paper assumes familiarity with transformer encoders (BERT, mBERT), fine-tuning, and topic modelling, but the four approaches are described in an accessible, comparative way.
• Scope: This paper compares four cross-lingual approaches for filtering relevant tweets from a decade-long multilingual hydrogen-energy corpus (English, Japanese, Hindi, Korean), then applies topic modelling to the best-performing approach's output.

What This Paper Is About

Keyword-based collection of social media data pulls in large amounts of irrelevant content, and when the data spans several languages this filtering problem becomes much harder because labelled training data usually exists only in one language (here, English). The authors take a decade of hydrogen-energy tweets in four languages and ask which cross-lingual classification strategy best separates relevant from irrelevant posts, given only 5,000 annotated English tweets to start from. They then use the winning strategy to uncover what topics different language communities actually discuss.

Key Contributions

1. Cross-lingual classification comparison. A comparative evaluation of four approaches for relevance classification in large-scale multilingual Twitter data: language-specific models, an English-only translation pipeline, English fine-tuned mBERT in zero-shot mode, and a hybrid translation-plus-multilingual model.
2. Key themes and phrases. An integrated pipeline that applies topic modelling to the filtered output to uncover thematic trends over a decade.
3. Temporal evolution. Insights into the evolution of hydrogen energy discourse from 2013 to 2022, including periods of heightened user engagement.
4. First decade-long multilingual analysis. Described by the authors as a pioneering decade-long, multilingual analysis of hydrogen energy discourse across English, Japanese, Hindi, and Korean, with a scalable framework intended for future clean energy studies.

Main Findings

• Approach 2 (English-only translation pipeline) gave the best overall results. A single bert-base-uncased model trained on 5,000 English annotated tweets reached 97.72% ± 0.23% accuracy and 97.70% ± 0.23% F1 on English, 79.85% ± 0.66% accuracy / 79.30% ± 0.69% F1 on Japanese, 86.03% ± 0.22% / 85.78% ± 0.23% on Korean, and 90.59% ± 0.27% / 90.53% ± 0.28% on Hindi.
• Approach 1 (language-specific BERT models) was strongest for Korean. Using monologg/kobert, it achieved 91.40% ± 0.31% accuracy and 91.35% ± 0.31% F1 on Korean, but lower results for Japanese (cl-tohoku/bert-base-japanese, 75.79% ± 0.49% accuracy, 74.88% ± 0.56% F1) and Hindi (ai4bharat/indic-bert, 78.08% ± 1.10% accuracy, 78.04% ± 1.10% F1). English reached 97.72% ± 0.23% accuracy and 97.70% ± 0.23% F1.
• Approach 3 (zero-shot mBERT) failed on non-English languages. English was high (96.68% ± 0.28% accuracy, 96.66% ± 0.28% F1), but Japanese fell to 50.21% ± 0.14% accuracy and 33.80% ± 0.30% F1, Korean to 50.56% ± 0.18% and 34.56% ± 0.39%, and Hindi to 53.59% ± 0.28% and 40.97% ± 0.55%. The authors attribute the gap between accuracy and F1 to a severe precision-recall imbalance.
• Approach 4 (hybrid) was the most balanced multilingual option. English reached 94.81% ± 0.58% accuracy / 94.79% ± 0.58% F1, Japanese 78.49% ± 0.44% / 77.78% ± 0.49%, Korean 83.57% ± 0.52% / 83.27% ± 0.54%, and Hindi 81.76% ± 0.48% / 81.18% ± 0.52%.
• Translation quality appears to matter more than native-language modelling for this task. Approach 2 outperformed Approach 1 on Japanese and Hindi, which the authors interpret as evidence that high-quality translation into English produces a more consistent feature representation than language-specific models trained on translated annotations.
• Relevance rates varied sharply by language and approach. Across the full corpora, English relevance ranged from 47.80% to 50.61% across approaches, Korean from 0.64% to 21.08%, Japanese from 0.44% to 21.50%, and Hindi from 5.01% to 35.80%. Approach 3 classified only 0.44% of Japanese and 0.64% of Korean tweets as relevant, consistent with its weak classification scores.
• Volume varied enormously by language. The full corpus for classification comprised 4,769,850 English tweets, 4,364,391 Japanese tweets, 148,040 Korean tweets, and 27,258 Hindi tweets (under Approach 1's totals).
• Post-2019 acceleration appears in every language. Year-wise relevant-tweet percentages peak in 2021–2022, reaching 25.91%–26.36% for English in 2022, 18.82%–28.30% for Japanese, 22.89%–24.01% for Korean, and 49.70%–69.15% for Hindi. The authors align this with the EU's Hydrogen Strategy (2020), Japan's National Hydrogen Strategy, and India's 2020 National Green Hydrogen Mission.
• Top topics differed by language. English was led by "hydrogen gas" (topic T7, 629,378 tweets, 26.0% of discussions), "hydrogen energy renewable" (T4, 363,249 tweets, 15.1%) and "hydrogen production" (T10, 298,415 tweets, 12.4%). Japanese was led by "hydrogen energy renewable" (T3, 224,413 tweets, 27.3%) and "first hydrogen power" (T10, 141,273 tweets, 17.2%). Hindi was led by "green hydrogen India" (T3, 2,753 tweets, 28.2%), "hydrogen fuel cell" (T2, 1,860 tweets, 19.1%) and "car nitin gadkari" (T1, 1,473 tweets, 15.09%). Korean was led by "green hydrogen energy" (T4, 6,091 tweets, 23%) and "going hydrogen" (T10, 3,730 tweets, 14.1%).
• Note on a reported discrepancy. The Results section reports "hydrogen gas" at 26.0% and "hydrogen energy renewable" at 27.3%, while the Discussion and Conclusion cites these as 26.2% and 28.4% respectively. The paper does not reconcile these figures.

Methodology in Plain English

The authors collected 30.7 million hydrogen-related tweets through the Academic Twitter API v2 covering January 2013 to December 2022, using keywords curated from literature review and expert consultation across English, Japanese, Hindi, and Korean. After deduplication and preprocessing, roughly 9.9 million unique tweets remained, and filtering strictly to the 2013–2022 window (removing retweets and content originally posted outside the window) left approximately 9.3 million tweets.

Because only English had labels, three domain experts annotated 5,000 English tweets as relevant or irrelevant, sampled uniformly across years. Agreement was strong (Fleiss' kappa of 0.85), and about 45% of tweets were labelled irrelevant, confirming that automated filtering was necessary.

The 5,000 English tweets were then translated into Japanese, Korean, and Hindi using the facebook/nllb-200-distilled-600M neural machine translation model. To test fairly, the authors additionally hand-annotated 750 original (untranslated) tweets in each of Japanese, Hindi, and Korean, uniformly spread across years.

Four approaches were then compared:

1. Language-specific models — translate the English annotations and train separate encoders per language (bert-base-uncased, cl-tohoku/bert-base-japanese, monologg/kobert, ai4bharat/indic-bert).
2. English-only model — train one bert-base-uncased on English and translate everything into English at prediction time.
3. English fine-tuned mBERT — fine-tune bert-base-multilingual-uncased on English annotations and apply it zero-shot to the other languages.
4. Hybrid — fine-tune mBERT on English annotations plus their translations, forming one multilingual training set.

All models shared a similar head: the encoder, a dropout layer with p = 0.3, a fully connected projection, a tanh activation, and a softmax output. Experiments used a ten-run protocol with ten different seeds, a 70%/15%/15% train/validation/test split, and early stopping with patience of 5 epochs. Results are reported as mean and standard deviation using accuracy and F1-score. (The Table 2 caption states standard deviations are across five runs, while the text describes ten runs; the paper does not reconcile this.)

After classification, the authors chose Approach 2's filtered output for topic modelling because it was the most accurate and most interpretable, and because the translation step presents all discourse in one language. Non-negative Matrix Factorisation (NMF) was selected over Latent Dirichlet Allocation (LDA) for interpretability and robustness. The TF-IDF representation combined unigrams, bigrams, and trigrams, filtered infrequent terms, and capped the feature space at 10,000 terms, extracting up to ten topics per language.

Why This Matters

Impact on research. The paper provides a controlled comparison of translation-based versus multilingual-model strategies for a real-world, low-resource-annotation setting. Its finding that high-quality translation can beat native-language models for languages like Japanese and Hindi challenges the intuition that native modelling always preserves nuance better, and its zero-shot mBERT results (F1 of 33.80%–40.97% on non-English languages) quantify how badly multilingual transfer can fail on morphologically distant languages. The NMF-based pipeline and the four-approach evaluation design are reusable templates for other clean-energy or policy discourse studies.

Real-world applications:

• Energy policy monitoring. Tracking how hydrogen policy announcements in different countries translate into public discussion, with the paper's own timeline connecting peaks to the EU Hydrogen Strategy, Japan's National Hydrogen Strategy, and India's National Green Hydrogen Mission.
• Multilingual social listening for industry. Filtering irrelevant keyword noise out of millions of posts so that firms can see genuine market signals, as with the "car nitin gadkari" topic linked to a hydrogen-powered car arriving at Indian Parliament in March 2022.
• Cross-lingual content moderation and relevance filtering. The relevance-classification comparisons give practical guidance on which architecture to deploy when annotated data exists in only one language.
• Regional market intelligence. The differentiated topic patterns — global scalability in English, technological leadership in Japanese, policy implementation in Hindi, infrastructure rollout in Korean — support market-entry and investment analysis.

Industry relevance. Organisations that monitor public discourse across languages typically cannot afford to annotate data for every language. This paper shows that a single English model plus translation can outperform both native-language models and zero-shot multilingual models for some language pairs, and that a hybrid mBERT approach offers competitive consistency where translation quality varies — practical guidance for building cost-effective multilingual pipelines.

Future Directions

1. Sentiment trends over time. The authors plan to extend the pipeline from topic discovery to tracking sentiment in multilingual hydrogen discourse across the decade.
2. Aspect-based sentiment analysis. Moving beyond document-level polarity to identify sentiment toward specific aspects, which the authors frame as a route to market insights.
3. Cross-lingual methods that handle native-language data better. The authors explicitly call for improved methods that rely less on translation and handle original-language content more effectively.
4. Open question — translation quality as a hidden variable. The paper attributes Approach 2's strength partly to high-quality translation, but does not report a systematic evaluation of translation quality or error analysis; whether the same ranking holds with a different translation model or for other language pairs is not addressed.
5. Open question — generalisation beyond hydrogen energy. The authors position the framework as reusable for future clean energy studies, but no results on other domains, platforms, or time periods are reported. The paper also does not report computational cost, runtime, or a comparison against models such as XLM-R, which are discussed in the literature review but not evaluated.

Target Audience

NLP researchers working on cross-lingual transfer, low-resource classification, and multilingual text mining will find the four-approach comparison and its metric tables most directly useful. Computational social scientists and digital humanities researchers studying public discourse on energy or climate policy will benefit from the decade-long, four-language case study and the topic-modelling pipeline. Data and ML engineers building multilingual social listening or relevance-filtering systems get concrete guidance on the trade-offs between translation and multilingual models. Energy analysts and policy researchers interested in regional differences in hydrogen discourse will find the language-specific topic and temporal patterns valuable, though the paper's classification techniques are the primary focus.

Authors’ abstract

Analysing multilingual social media discourse remains a major challenge in natural language processing, particularly when large-scale public debates span across diverse languages. This study investigates how different approaches for cross-lingual text classification can support reliable analysis of global conversations. Using hydrogen energy as a case study, we analyse a decade-long dataset of over nine million tweets in English, Japanese, Hindi, and Korean (2013--2022) for topic discovery. The online keyword-driven data collection results in a significant amount of irrelevant content. We explore four approaches to filter relevant content: (1) translating English annotated data into target languages for building language-specific models for each target language, (2) translating unlabelled data appearing from all languages into English for creating a single model based on English annotations, (3) applying English fine-tuned multilingual transformers directly to each target language data, and (4) a hybrid strategy that combines translated annotations with multilingual training. Each approach is evaluated for its ability to filter hydrogen-related tweets from noisy keyword-based collections. Subsequently, topic modeling is performed to extract dominant themes within the relevant subsets. The results highlight key trade-offs between translation and multilingual approaches, offering actionable insights into optimising cross-lingual pipelines for large-scale social media analysis.