PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation

AI summary

Overview

Research area: Natural Language Processing — pre-trained sequence-to-sequence (PS2S) models for Khmer natural language generation.

Technical level: Intermediate. The paper assumes familiarity with encoder-decoder pre-training (BART-style denoising), subword tokenization, and standard NLG evaluation metrics (BLEU, ChrF, COMET, Rouge-L), but it explains each linguistic module concretely.

Scope: The paper introduces PrahokBART, the first compact PS2S model pre-trained specifically for Khmer, and evaluates it against mBART50 on machine translation, text summarization, and headline generation.

What This Paper Is About

Khmer text has no word boundaries, has encoding ambiguities in its complex script, and uses spaces for functional purposes (as commas or for readability) rather than as word delimiters. Existing multilingual models such as mBART50 ignore these properties because they rely on language-agnostic subword tokenization. This paper asks whether linguistic modules — normalization and word segmentation — still add value for Khmer PS2S models, and whether a model can learn to generate Khmer "functional spaces" correctly. The authors build a compact Khmer-specific model from scratch to answer both questions.

Key Contributions

1. PrahokBART itself: described as the first compact PS2S model specifically designed for Khmer, incorporating normalization and word segmentation as integral linguistic modules. Two sizes are released: base and big.
2. Three-task evaluation: the model is evaluated on machine translation, text summarization, and headline generation, and is reported to outperform the multilingual mBART50 model in efficiency and generation quality as measured by BLEU, ChrF, and Rouge-L.
3. Module-level analysis: the paper isolates the impact of each linguistic module and quantifies how well the model generates functional spaces, measured as a score difference ΔS = S_all − S_content.
4. Data curation: a cleaned Khmer–English pre-training corpus is assembled from Common Crawl (mC4) and Wikimedia (Wikipedia, Wikibooks) plus WMT2020 Khmer monolingual data, totaling approximately 4.2 billion tokens (0.7 billion Khmer and 3.5 billion English).

Main Findings

• Pre-training beats random initialization. All pre-trained models substantially outperform the Random baseline, which shares PrahokBART base's architecture (6 encoder and 6 decoder layers, 8 attention heads, hidden and intermediate dimensions of 512 and 2048).
• Compact model is competitive. PrahokBART base (62 million parameters, 1x FLOPs) is reported as comparable to mBART50 (611 million parameters, 10x FLOPs) despite being ten times smaller. PrahokBART big (211 million parameters, 3x FLOPs) improves on all tasks while remaining three times smaller than mBART50.
• Best scores on the three tasks. PrahokBART big reaches 53.54 ChrF (English→Khmer), 53.26 ChrF (Khmer→English), 26.23 Rouge-L (TextSum), and 22.92 Rouge-L (HeadGen). It also achieves the best COMET scores: 77.69 (English→Khmer) and 82.00 (Khmer→English), versus 74.71 and 78.47 for mBART50 and 70.51 and 72.41 for Random.
• Summarization beats extractive baselines. On TextSum, PrahokBART scores 30.60 Rouge-1, 18.00 Rouge-2, and 26.23 Rouge-L, compared with Lead-3 (8.03, 4.52, 7.85), LexRank (7.59, 4.57, 7.38), and an Oracle upper bound of 65.36, 57.56, and 63.51. The best model remains far behind the oracle.
• Normalization makes corpora more predictable. After removing invisible characters then applying encoding normalization, perplexity on the pre-training corpora drops: Common Crawl from 4.31 to 4.08 and Wiki from 5.51 to 5.12. ALT drops from 4.30 to 4.16 and Lr-sum from 3.46 to 3.45. Applying encoding normalization reduced unique vocabulary by roughly 20k items for CC and 2k for Wiki.
• PrahokBART's tokenizer is more efficient than mBART50's but less "compact" than plain Unigram. Fertility is 9.27 for mBART50, 7.09 for Unigram, and 7.81 for PrahokBART; average length ratio is 0.313, 0.238, and 0.289 respectively.
• Word segmentation improves downstream quality. With word segmentation, PrahokBART big scores 53.54 (English→Khmer), 53.26 (Khmer→English), 26.23 (TextSum), and 22.92 (HeadGen); without it (the "Unigram" setting) scores fall to 53.01, 52.93, 18.07, and 20.49.
• Better functional-space generation. In ΔS terms, PrahokBART leads on English→Khmer (1.99 vs 1.86 for Unigram and 0.78 for mBART50), TextSum (2.63 vs 1.51 and 2.05), and HeadGen (1.00 vs 0.86 and 0.91).
• More data helps. Training on Low (39M), Medium (401M), and High (4.

Authors’ abstract

This work introduces {\it PrahokBART}, a compact pre-trained sequence-to-sequence model trained from scratch for Khmer using carefully curated Khmer and English corpora. We focus on improving the pre-training corpus quality and addressing the linguistic issues of Khmer, which are ignored in existing multilingual models, by incorporating linguistic components such as word segmentation and normalization. We evaluate PrahokBART on three generative tasks: machine translation, text summarization, and headline generation, where our results demonstrate that it outperforms mBART50, a strong multilingual pre-trained model. Additionally, our analysis provides insights into the impact of each linguistic module and evaluates how effectively our model handles space during text generation, which is crucial for the naturalness of texts in Khmer.