One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography

AI summary

Overview

Research area: Natural Language Processing — multilingual language model pretraining, cross-lingual transfer, and script/input representation design.

Technical level: Intermediate. Readers should be comfortable with autoregressive language model pretraining, subword tokenization, and standard evaluation tasks (NLI, intent classification, summarization), but the paper's central comparison is conceptually simple: change only the writing system of the input and measure what happens.

Scope: A controlled, from-scratch pretraining study comparing three input representations — native orthography, IPA transcription, and Uroman romanization — for autoregressive multilingual models at three scales across eight languages, evaluated under prompting and fine-tuning on both seen and unseen languages. Code is released at https://github.com/skai-research/one-form-transfer.

What This Paper Is About

Multilingual language models transfer knowledge across languages mainly through shared subword vocabulary, and that mechanism breaks down when related languages use different writing systems. Prior work tried to fix this by "script equalization" (romanization or IPA transcription), but comparisons were rare, focused on encoder-only models, and mostly adapted already-pretrained models rather than pretraining from scratch. This paper asks a direct question: if you build the multilingual model yourself and choose the input form deliberately, which representation transfers best — and does it matter when you apply it, at pretraining or at fine-tuning?

Key Contributions

1. A controlled comparison of three input representations in autoregressive multilingual pretraining. Orthographic text, IPA transcription (Phonemizer), and Uroman romanization are compared with architecture, data, vocabulary size (100K Byte-Level BPE), and training procedure held constant, across three scales — 467M, 709M, and 1.03B parameters (83M, 197M, and 387M non-embedding).
2. An eight-language, four-pair experimental design. English–Spanish, Russian–Polish, Hindi–Urdu, and Tamil–Malayalam were chosen to vary the relationship between orthography and phonology, with a single model trained jointly on all eight languages.
3. The first direct comparison of phonemic and romanized pretraining for autoregressive LMs, showing the two are not interchangeable: they produce their highest subword overlap on different language pairs.
4. A qualification of the standard romanized fine-tuning recipe. The paper shows that fine-tuning a text-pretrained model on romanized data regresses on languages the base model already covers, and helps only when script coverage is absent — reproducing prior gains in a controlled English-only GPT-2 setup.

Main Findings

• Romanized pretraining wins everywhere. Pretraining on romanized text yields the strongest cross-lingual transfer in every evaluation regime and at every model size — prompting on XStoryCloze and XCOPA, fine-tuning on XNLI, MASSIVE, and XL-Sum, and transfer to unseen languages via MASSIVE and XNLI — and the advantage over text widens with scale.
• IPA beats text in most settings but trails romanization. IPA improves over orthographic text in most evaluations and matches romanization only on Hindi–Urdu, the pair that is most phonologically aligned and most orthographically disjoint. Outside that pair, and especially on unseen languages, romanization leads.
• Subword overlap does not map cleanly onto performance. Under text, meaningful overlap appears only for the Latin-script English–Spanish pair; the three different-script pairs sit at or near zero. Romanization lifts Russian–Polish and Tamil–Malayalam into a range comparable to English–Spanish but does not close the gap for Hindi–Urdu, while IPA produces its highest overlap on Hindi–Urdu and lags romanization on every other pair.
• Sequence length inequality is large under text. On the FLORES parallel corpus, Hindi, Tamil, and Malayalam tokens inflate under text by factors of roughly three times their romanized counterparts, implying roughly three times the inference latency and API cost relative to Latin- and Cyrillic-script peers. IPA and romanization compress them into the same range.
• Romanized fine-tuning of a text-pretrained multilingual model hurts. The Text→Rom condition produces large regressions on all pretraining languages across all benchmarks (for example, XNLI and MASSIVE macro scores drop substantially relative to text-pretrained baselines at every scale). On unseen Greek and Bengali, where the pretrained model lacks script coverage, it helps, but by much smaller margins than romanized pretraining. On unseen MASSIVE the benefit holds only at the smallest scale and disappears at Medium and Large.
• The prior recipe reproduces only for script-uncovered languages. Fine-tuning English-pretrained GPT-2 checkpoints (GPT-2, GPT-2 Medium, GPT-2 Large) on romanized data recovers reported gains: Hindi, Urdu, Tamil, and Malayalam gain 20–30 F1 on MASSIVE at larger scales and move from near-zero to 10–15 ROUGE-L on XL-Sum, while Latin-script languages show small, sometimes negative changes. Those gains still fall well short of pretraining-time romanization.
• Gains are conservative under a compute disadvantage. Word-matched corpora yield 50B Text tokens versus 33B IPA/Romanized tokens, putting the IPA and Romanized models at a compute disadvantage.
• Diacritic stripping matters for overlap. Retaining stress, length, tone, and nasalization diacritics in IPA yielded roughly 50% the cross-lingual overlap (weighted Jaccard) of the stripped version.

Methodology in Plain English

The team built their own multilingual models rather than adapting someone else's. They drew monolingual documents from FineWeb-2 for each of four language pairs, sized each pair's corpus to match OpenWebText while preserving the natural resource imbalance within the pair, and sampled about 3.2M words per pair for validation (roughly 21.7B words total). The same underlying content was then rendered three ways: kept in native script, converted to IPA with Phonemizer (with diacritics stripped and the library modified to preserve digits and out-of-script characters), or romanized with Uroman. For each representation they trained one 100K Byte-Level BPE tokenizer jointly over all eight languages, so the only thing that changes between runs is the surface form of the text.

Pretraining used a modified NanoGPT at three scales with value embeddings, a dual optimizer (AdamW for embeddings, head, and scalars; Muon for hidden matrices), and 524,288 tokens per step on 8 NVIDIA H100 GPUs, for 3 epochs with the lowest-validation-loss checkpoint selected. Evaluation happened two ways: directly prompting the checkpoints zero-shot and few-shot on XStoryCloze and XCOPA, and fine-tuning on XNLI, IndicNLI, MASSIVE, and XL-Sum for classification (macro-F1) and summarization (ROUGE-L). Inputs and references were converted into the matching representation. To test generalization, they fine-tuned on unseen Arabic, French, Bengali, and Greek. They also measured two language-agnostic properties of each representation — frequency-weighted subword overlap and average tokens per document — to explain the downstream patterns. Significance was assessed with an approximate randomization test at p<0.05.

Why This Matters

Impact on research. The paper reframes input representation as a first-class pretraining design decision alongside data mixture and tokenizer choice, rather than a downstream patch. It also provides the first direct phonemic-versus-romanized comparison for autoregressive models, showing the two mechanisms are not interchangeable and that improvements do not simply track vocabulary overlap — echoing earlier findings that overlap is not a reliable predictor of transfer. The negative result on romanized fine-tuning qualifies a widely used recipe and explains when it will and will not work.

Real-world applications (note that the paper's own limitation section flags that deploying these models requires phoneme-to-grapheme or romanization-to-text conversion, which it did not evaluate):

• Multilingual intent classification and virtual assistants, where the MASSIVE results show the largest romanization gains and where the paper notes sensitivity to surface-form noise in short trigger phrases.
• Cross-lingual abstractive summarization for non-Latin-script and low-resource languages, per the XL-Sum results.
• Serving and cost planning for multilingual inference, since text inflates Hindi, Tamil, and Malayalam token counts by roughly three times, with corresponding penalties in latency, per-token API cost, and effective context window use.
• Transfer to languages with no pretraining script coverage, such as Bengali and Greek in this study, where romanized pipelines offer the largest relative benefit.

Industry relevance. Most modern open-source and open-weight models are multilingual and already cover many scripts, which is exactly the condition under which the paper finds romanized fine-tuning to degrade performance. Teams deciding how to add a new script or language to an existing model should weigh pretraining-time romanization rather than assuming a transliterated fine-tuning pass will help. The sequence-length results are also directly actionable for inference economics.

Future Directions

• Scale. The models here top out at roughly 1.03B parameters (387M non-embedding), far below the 7B-to-100B-plus range of contemporary multilingual models; the paper notes the romanization advantage widens with scale and calls for larger-scale replication.
• Broader writing systems and language relationships. The selected languages cover alphabetic and abugida systems only; logographic systems such as Chinese are absent, and the authors call for study of how language family distance, phonological similarity, and shared loanwords modulate the benefit of each representation.
• Better phonemization and IPA encoding. Improving grapheme-to-phoneme quality (Phonemizer's eSpeak backend has incomplete support for several languages) and finding alternatives to Unicode for phoneme tokenization — including compound symbols and diacritic composition — could change the reported IPA trends.
• Reversibility. Phoneme-to-grapheme and romanization-to-text conversion is needed for generative use by ordinary users; the paper notes both operations are lossy and not fully reversible, and leaves evaluation of effectiveness, reversibility, and computational overhead to future work.
• Mechanistic characterization. Understanding how representation choice plays out in the model's internal representations is described as a natural next step.

Target Audience

Researchers and engineers working on multilingual pretraining, tokenizer and data-mixture design, and cross-lingual transfer — particularly those deciding how to bring new scripts or low-resource languages into an existing model. It is also useful for practitioners reasoning about multilingual inference cost and context-window fairness, and for anyone evaluating claims about transliteration or phonemization as a downstream fix. Readers seeking large-scale (7B+) evidence or results on logographic scripts should note the paper's stated limitations rather than treat it as settled guidance at those settings.

Authors’ abstract

Multilingual language models transfer knowledge across languages through shared subword vocabulary, a mechanism that breaks down when related languages use different writing systems. Prior work addresses this via script equalization (romanization or IPA transcription), but direct comparisons are rare; the focus has been on encoder-only models, with most work adapting existing pretrained models. We systematically compare different input representations in autoregressive multilingual pretraining, comparing orthographic text, IPA, and romanization in a controlled setup across three scales (467M, 709M, and 1.03B) on eight languages in four typologically motivated pairs. Across a wide range of downstream tasks on seen and unseen languages, romanized pretraining yields the strongest cross-lingual transfer, and the advantage over text widens with scale. IPA improves over text in most settings but trails romanization. Surprisingly, finetuning a text-pretrained model on romanized data hurts performance on languages already covered by the base model, only marginally helping when the model lacks script coverage. Our results indicate that for multilingual models spanning typologically diverse scripts, to obtain maximum benefits, romanization should be treated as a core design choice applied at pretraining rather than a post hoc fix.