Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhala

AI summary

Overview

Research area: Natural Language Processing — intrinsic evaluation of language models on low-resource, script-diverse languages (Sinhala).

Technical level: Intermediate. The paper is readable without deep mathematical background, though familiarity with perplexity, tokenization and Spearman rank correlation helps.

Scope: A benchmark of 24 open-source decoder-only language models (350M to 14B parameters) scored by perplexity on 1,500 Sinhala sentences across three script conditions — Unicode, Romanized and mixed-script.

What This Paper Is About

Sinhala is written in Unicode in formal settings, but in everyday digital communication — social media, messaging, forums — it is frequently written in Latin letters ("Romanized" Sinhala), and users often mix both scripts in the same sentence. Existing benchmarks evaluate Sinhala almost only in Unicode, so nobody knew how modern language models actually cope with the Romanized and mixed-script text that dominates real usage.

This paper builds a new evaluation corpus covering all three script conditions and measures how 24 open-source models perform on each, asking whether strong Unicode performance predicts Romanized or mixed-script ability, and whether bigger models handle script variation better.

Key Contributions

1. First systematic intrinsic benchmark of Sinhala across three script conditions. The authors evaluate 24 open-source generative models spanning 350M to 14B parameters on Unicode, Romanized and mixed-script Sinhala using perplexity, addressing what they describe as a gap where no comparative analysis across both scripts existed.

2. A new evaluation corpus capturing authentic script variation. Two datasets totalling 1,500 sentences: a 500-sentence-pair Unicode–Romanized parallel set and a 500-sentence authentic mixed-script set, assembled from YouTube comments, Facebook posts, blogs and online forums, then selected via K-means clustering on sentence embeddings rather than random sampling.

3. A mixed-script evaluation setting introduced explicitly to test robustness under realistic script switching, which the authors argue single-script evaluation cannot capture.

4. Statistical evidence that model scale does not predict script-handling competence, alongside quantified degradation ratios and rank correlations that show Unicode performance predicts mixed-script robustness but not Romanized capability.

Main Findings

• Severe script sensitivity. Median performance degradation from Unicode to Romanized text exceeds 300 times; the reported median perplexity ratio is 312.3×, with a range of 149.1× to 769.7×. Some models degrade by nearly three orders of magnitude.

• Mixed-script degradation is far milder. The median perplexity ratio for mixed-script text is 2.0× (range 1.6× to 3.1×), suggesting Unicode tokens in a sentence partially anchor the model's predictions even when Romanized segments are poorly handled.

• Scale does not help. Parameter count correlates negligibly with performance in every condition: Unicode ρ = 0.085 (p = 0.693), Romanized ρ = −0.085 (p = 0.693), mixed-script ρ = −0.018 (p = 0.934). Smaller models often outperform architectures 28 times larger. Pythia-410M achieves the lowest Unicode perplexity of the suite (2.58) and a mixed-script perplexity of 4.92, ahead of Mistral-Nemo-Base-2407 (12B: Unicode 2.82, mixed 6.72) and Phi-4 (14B: Unicode 4.01, mixed 12.07).

• Unicode performance predicts mixed-script but not Romanized ability. Spearman correlations: Unicode vs mixed-script ρ = 0.957 (p < 0.001), Unicode vs Romanized ρ = 0.519 (p = 0.009), Romanized vs mixed-script ρ = 0.514 (p = 0.010). Of the top five Unicode models, only 2 of 5 remain in the top five for Romanized text, whereas 4 of 5 stay in the top five for mixed-script.

• Romanized perplexities routinely exceed 1,000. Examples from the results table include Mistral-7B-v0.3 (Unicode 5.26, Romanized 921.40 — the lowest Romanized value of the 24 models), Llama-3.1-8B (3.22 / 970.64), Mistral-Nemo-Base-2407 (2.82 / 1978.10), Hormoz-8B (5.98 / 4175.56), SmolLM3-3B (6.98 / 5372.60) and Bloom-560M (12.64 / 4915.05 — the highest).

• BLOOM variants perform poorly. Unicode perplexities of 6.95 to 12.64 against Romanized perplexities of roughly 3,200 to 4,900, and mixed-script values the paper reports in the 13 to 28 range.

• Mixed-script is where models diverge most. The coefficient of variation was highest for mixed-script text (92.6%) versus Unicode (66.4%) and Romanized (62.3%).

• Tokenization failure is not the whole story. For the Romanized sentence "amma bath uyanawa" (Mother is cooking rice), perplexity ratios were 836× for Mistral-7B, 2723× for Pythia-410M and 6565× for Bloom-560M. Even though "bath" tokenizes as a single unit because it overlaps English vocabulary, genuinely Sinhala Romanized words fragment — "uyanawa" → [uy, an, awa] and "gewamu" → [ge, w, am, u]. The authors attribute the gap mainly to the near-complete absence of Romanized Sinhala in pre-training corpora rather than to tokenization granularity alone.

• Within-family scaling helps, cross-family scaling does not. The Llama family improves consistently with scale across all script conditions, but several 7B–8B models lag behind more compact models on specific script conditions.

• Llama-3.2-1B is highlighted for a favourable quality-efficiency trade-off among 1B-parameter models, offering the best Unicode and mixed-script perplexities and competitive Romanized performance in that size class.

Methodology in Plain English

The authors collected public Sinhala text from YouTube comments, Facebook posts, technology and culture blogs, and online forums. They cleaned it by removing URLs, emails, hashtags and non-textual elements, filtered out non-Sinhala content, deduplicated repeated or bot-generated posts, segmented into sentences, and kept sentences between 2 and 15 tokens.

From this cleaned corpus they built two evaluation sets. The first contains 500 sentence pairs with identical meaning in Unicode Sinhala and in Romanized transliteration, so that script is the only thing that differs between the two versions. The second contains 500 sentences of authentic mixed-script Sinhala, where the two scripts co-occur within individual sentences.

Instead of picking sentences at random, they encoded each sentence with the multilingual sentence-transformer model sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, ran K-means clustering with k = 500, and picked the sentence nearest each cluster centroid. This spreads the evaluation across topics while avoiding redundancy.

The final corpus is 1,500 sentences. The Unicode and Romanized subsets have identical mean length (6.07 ± 1.83 tokens) with vocabulary sizes of 1,184 and 1,171 respectively. The mixed-script set has a mean length of 9.49 ± 4.03 tokens and a vocabulary size of 2,262.

They then ran 24 open-source decoder-only models through the text, computing perplexity — a measure of how surprised a model is by the text, where lower is better. Each model used its own native tokenizer, unmodified, so tokenization behaviour was preserved as part of what was being measured. Perplexity was aggregated over all 500 sentences per script condition. Experiments ran on NVIDIA A100 GPUs with 40GB memory, using PyTorch 2.0 and Hugging Face Transformers 4.36, with models loaded in float16 precision. Differences across scripts were then quantified using perplexity ratios and Spearman rank correlations.

Why This Matters

Impact on research. The paper argues that single-script evaluation substantially underestimates real-world deployment challenges, and that existing multilingual benchmarks including Sinhala only in Unicode form miss the deficiencies that appear in Romanized and mixed-script text. Because the evaluation datasets are released publicly, the protocol is intended to be adopted directly for other script-diverse low-resource languages without collecting new data. The finding that parameter count carries essentially no signal for script-handling competence challenges the default assumption that scale solves low-resource problems.

Real-world applications:

• Social media and messaging platforms that receive large volumes of Romanized Sinhala input, where models tuned or evaluated on Unicode alone risk severe degradation.
• Machine translation and transliteration systems between Sinhala and other languages, where prior Sinhala NLP work has concentrated.
• Content moderation and sentiment analysis on Sri Lankan user-generated content, which the paper notes is dominated by Romanized and mixed-script forms.
• Resource-constrained deployments choosing a model, where the paper identifies Llama-3.2-1B as a favourable quality-efficiency option and warns that some larger models become unstable on at least one script condition.

Industry relevance. The practical takeaway is that script-aware evaluation pipelines are needed, and that preprocessing strategies such as robust transliteration, script normalization or targeted fine-tuning on Romanized and mixed-script data may be necessary before deployment in multi-script environments. The paper notes that Sinhala is spoken by about 22 million people, and that its findings generalize to other languages with script duality such as Hindi, Tamil and Arabic dialects.

Future Directions

• Investigate cross-lingual vocabulary overlap in tokenization. Romanized Sinhala words that coincidentally match English or other South Asian words (for example "bath" and "api") tokenize more cleanly than genuinely unique Sinhala Romanized words, which the authors warn may underestimate the true severity of tokenizer deficiency.

• Test transliteration to Unicode before inference as an immediately deployable fix for the perplexity gap on Romanized input, without requiring retraining.

• Complement perplexity with downstream task evaluation. The authors state explicitly that perplexity is intrinsic and does not directly measure performance on machine translation, sentiment analysis or named entity recognition, and that the observed script sensitivity may manifest differently across those tasks.

• Broaden the corpus. The 1,500-sentence corpus may not fully capture dialectal variation or domain-specific usage, and its social-media origin may over-represent informal registers while under-representing academic or professional writing.

Target Audience

Researchers and practitioners working on low-resource and morphologically rich language technology, particularly those handling Sinhala or other script-diverse languages such as Hindi, Tamil and Arabic dialects. It is also useful for NLP engineers selecting models for deployment in multi-script user environments, for benchmark designers who want a replicable protocol for script-condition evaluation, and for anyone studying tokenizer behaviour and pre-training data composition as drivers of model performance.

Authors’ abstract

The performance of Language Models (LMs) on low-resource, morphologically rich languages like Sinhala remains largely unexplored, particularly regarding script variation in digital communication. Sinhala exhibits script duality, with Unicode used in formal contexts and Romanized text dominating social media, while mixed-script usage is common in practice. This paper benchmarks 24 open-source LMs on Unicode, Romanized and mixed-script Sinhala using perplexity evaluation across diverse text sources. Results reveal substantial script sensitivity, with median performance degradation exceeding 300 times from Unicode to Romanized text. Critically, model size shows no correlation with script-handling competence, as smaller models often outperform architectures 28 times larger. Unicode performance strongly predicts mixed-script robustness but not Romanized capability, demonstrating that single-script evaluation substantially underestimates real-world deployment challenges. These findings establish baseline LM capabilities for Sinhala and provide practical guidance for model selection in multi-script low-resource environments.