Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language

AI summary

Overview

Research area: Natural Language Processing for low-resource languages — specifically sentence-level topic classification of Nepali text using transformer-based (BERT-family) pre-trained language models.

Technical level: Intermediate. Readers benefit from familiarity with transformer fine-tuning, classification metrics, and the distinction between multilingual, Indic, and monolingual pre-trained models.

Scope: The paper benchmarks ten BERT-based models — one English, three multilingual, four Indic, one Hindi, and one Nepali — on a balanced 25,006-sentence Nepali dataset spanning five topic domains, evaluated with accuracy, weighted precision, weighted recall, weighted F1-score, and AUROC.

What This Paper Is About

Nepali, an Indo-Aryan language written in Devanagari, has received far less NLP attention than high-resource languages, largely because of scarce annotated corpora and limited computational resources. While many BERT variants exist — some multilingual, some built for Indian languages, and one trained specifically on Nepali — no systematic comparison of these model families had been conducted on Nepali topic classification. This paper fills that gap by fine-tuning and evaluating ten different BERT-based models on the same balanced Nepali sentence-level dataset across five domains, establishing a baseline for future Nepali classification work.

Key Contributions

1. A broad cross-family benchmark: Ten BERT-based models are evaluated on a single Nepali task, spanning an English model (RoBERTa), three multilingual models (mBERT, XLM-R, mDeBERTa), four Indic models (MuRIL-base, MuRIL-large, DevBERT, IndicBERT), one Hindi model (HindiBERT), and one Nepali model (NepBERTa).

2. A sentence-level, balanced, annotated dataset: 25,006 Nepali sentences from the ILPRL corpus, filtered and deduplicated, distributed nearly evenly across five categories — Agriculture (5,012 sentences, 20.04%), Health (4,999, 19.99%), Education & Technology (4,999, 19.99%), Culture & Tourism (4,998, 19.99%), and General Communication (4,998, 19.99%).

3. Comprehensive multi-metric evaluation: Results are reported not only overall using accuracy, weighted precision, weighted recall, weighted F1-score, and AUROC, but also broken down per category, together with parameter counts and training time for each model — adding an efficiency dimension to the comparison.

4. A baseline for document-level extension: The authors frame the work as a foundation for future document-level classification, where sentence-level signals could be extended to full articles.

Main Findings

• Indic models led overall: MuRIL-large achieved the best results across every metric reported, with accuracy 90.56%, weighted precision 90.68%, weighted recall 90.56%, F1-score 90.60%, and AUROC 0.984. It used 304M parameters and 65.75 minutes of training time.

• The weakest model was also an Indic model: IndicBERT performed lowest of all ten models, with accuracy 80.57%, weighted precision 80.93%, weighted recall 80.57%, F1-score 80.66%, and AUROC 0.948 — but it was also the fastest to train at 29.77 minutes with 110M parameters.

• NepBERTa was competitive and efficient: The Nepali-only model reached accuracy 88.32%, weighted precision 88.25%, weighted recall 88.32%, F1-score 88.26%, and AUROC 0.967, using 110M parameters and 34.78 minutes of training — substantially cheaper than MuRIL-large's 304M parameters and 65.75 minutes, or XLM-R's 278M parameters and 45.13 minutes. The authors note their initial hypothesis that NepBERTa would beat the Indic and multilingual models was not confirmed, and suggest the reason is that MuRIL and XLM-R were also trained on sizeable amounts of Nepali text.

• Multilingual models were close behind: XLM-R scored accuracy 89.72%, weighted precision 89.67%, weighted recall 89.72%, F1-score 89.67%, and AUROC 0.979 (278M parameters, 45.13 minutes). mBERT reached F1-score 87.78% (177M parameters, 44.68 minutes) and mDeBERTa reached 86.92% (125M parameters, 39.95 minutes).

• The other Indic and Hindi models clustered near the top: MuRIL-base scored F1-score 89.62% (110M parameters, 49.60 minutes), DevBERT 90.06% (110M parameters, 51.98 minutes), and HindiBERT 89.63% (110M parameters, 49.45 minutes).

• The English model trailed: RoBERTa recorded the lowest overall scores with accuracy 83.81%, weighted precision 83.81%, weighted recall 84.09%, F1-score 83.83%, and AUROC 0.966 (125M parameters, 44.28 minutes).

• Category-level performance varied: MuRIL-large scored F1-scores of 93.18% (Agriculture), 88.87% (Health), 91.03% (Education & Technology), 94.86% (Culture & Tourism), and 84.37% (General Communication). Across all models, General Communication was consistently the hardest category, while Agriculture, Health, Education & Technology, and Culture & Tourism were classified more accurately.

• Confusion matrix detail: For MuRIL-large, 492 of 533 Agriculture-related sentences were correctly classified with 41 misclassified, whereas only 394 of 459 General Communication-related sentences were correctly classified with 65 misclassified. The authors attribute the gap to General Communication being derived from multiple sub-categories.

• Overall pattern: Models pre-trained on linguistically and scriptually related Indic data (MuRIL, DevBERT, HindiBERT) outperformed broad multilingual models on Nepali text.

Methodology in Plain English

The researchers took an existing Nepali corpus from the Information and Language Processing Research Lab (ILPRL) and cleaned it — filtering and deduplicating — to produce a balanced set of 25,006 sentences across five topic categories. They then picked ten pre-trained BERT-family models chosen to represent different pre-training philosophies: broad multilingual coverage, Indian-language coverage, Hindi-only coverage, Nepali-only coverage, and an English-only comparison point.

Each model was fine-tuned on the same data using the Hugging Face library, with an identical training recipe: 20,005 training sentences, 2,500 validation sentences, and 2,501 test sentences; 10 epochs; a learning rate of 2e-5; batch size 8; gradient accumulation of 2; and a maximum sequence length of 256. All experiments ran on a personal desktop with an Intel Xeon Platinum 8255C CPU at 2.50 GHz and a GeForce RTX 4060 Ti 16 GB GPU.

Performance was measured with accuracy, weighted precision, weighted recall, weighted F1-score, and AUROC. The authors also recorded each model's parameter count and training time in minutes, and visualized the confusion matrices of the top model from each group (MuRIL-large, XLM-R, NepBERTa, and RoBERTa) to see where errors concentrated. A per-category F1 breakdown supported the analysis.

Why This Matters

Impact on research: This is the first systematic head-to-head comparison of multilingual, Indic, Hindi, and Nepali BERT variants on Nepali text classification. It gives future researchers a reproducible baseline and a clear signal that regional and monolingual pre-training can outperform broad multilingual models in low-resource settings — reinforcing prior findings about Indic models capturing morphology and syntax more effectively.

Real-world applications:

• Automatic topic tagging and routing of Nepali news articles into Agriculture, Health, Education & Technology, Culture & Tourism, and General Communication sections.
• Content recommendation and personalization for Nepali-language news portals and digital platforms.
• Media monitoring and content analytics for Nepali-language publications, helping editors track coverage across domains.
• Organizing and searching large Nepali text archives, where domain labels improve retrieval.

Industry relevance: The efficiency findings matter commercially. NepBERTa achieved an F1-score of 88.26% with 110M parameters and 34.78 minutes of training, compared with MuRIL-large's 90.60% at 304M parameters and 65.75 minutes. Organizations with limited GPU resources can weigh that tradeoff directly. The paper also supplies a consistent baseline for anyone deploying Nepali text classification in production.

Future Directions

• Extending the framework to document-level or article-level classification, using sentence-level insights to capture broader contextual representations within full articles.
• Expanding the dataset quantitatively with additional categories, and building on suggestions to continue pre-training large Indic models like MuRIL on diverse Nepali corpora or mixed Indic–Nepali datasets.
• Conducting detailed error analysis to understand the strengths and weaknesses of Indic versus Nepali models, especially on the harder General Communication category that mixes sub-categories.
• Exploring ensemble learning techniques to increase classification accuracy.

Target Audience

Researchers and graduate students in low-resource and Indic NLP; engineers building Nepali-language text processing or news-classification systems; and practitioners deciding which pre-trained model family — multilingual, Indic, Hindi, or Nepali-specific — offers the best accuracy-versus-cost tradeoff for a Devanagari-script task. It is also useful to teams evaluating whether a smaller monolingual model such as NepBERTa can substitute for a much larger Indic model in production.

Authors’ abstract

Transformer-based models such as BERT have significantly advanced Natural Language Processing (NLP) across many languages. However, Nepali, a low-resource language written in Devanagari script, remains relatively underexplored. This study benchmarks multilingual, Indic, Hindi, and Nepali BERT variants to evaluate their effectiveness in Nepali topic classification. Ten pre-trained models, including mBERT, XLM-R, MuRIL, DevBERT, HindiBERT, IndicBERT, and NepBERTa, were fine-tuned and tested on the balanced Nepali dataset containing 25,006 sentences across five conceptual domains and the performance was evaluated using accuracy, weighted precision, recall, F1-score, and AUROC metrics. The results reveal that Indic models, particularly MuRIL-large, achieved the highest F1-score of 90.60%, outperforming multilingual and monolingual models. NepBERTa also performed competitively with an F1-score of 88.26%. Overall, these findings establish a robust baseline for future document-level classification and broader Nepali NLP applications.