ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality

AI summary

Overview

Research area: Natural language processing, specifically scaling laws for large language model pretraining and finetuning in multilingual settings.

Technical level: Advanced. The paper assumes familiarity with Chinchilla-style scaling laws (irreducible loss, power-law terms in model size N and data D), compute-optimal training budgets, transfer learning, and cross-lingual evaluation methodology.

Scope in one sentence: The paper presents ATLAS, a scaling law for multilingual pretraining and finetuning, fitted on 774 multilingual training experiments (10M–8B parameters, 400+ training languages, 48 evaluation languages), and uses it to derive a 38 × 38 cross-lingual transfer matrix, a "curse of multilinguality" scaling law, and a pretrain-versus-finetune compute crossover formula.

What This Paper Is About

Scaling laws have been developed mostly for English, even though the most prominent AI models explicitly serve billions of international users. Public multilingual scaling-law research is limited: only Llama-3 briefly discusses multilingual scaling laws (training on only 8% non-English tokens), and prior rigorous work covers smaller multilingual models (under 45M and under 1.2B parameters respectively). The paper's goal is to build a scaling law that accounts for data repetition and cross-lingual transfer in many languages, and to use it to answer practical questions about language mixtures, language coverage expansion, and whether to pretrain or finetune.

Key Contributions

1. The Adaptive Transfer Scaling Law (ATLAS), which generalizes better to larger/unseen N, D, C, and training mixes M than prior work — often improving out-of-sample R² by more than 0.3. The paper also estimates a compute efficiency tax between multilingual and monolingual training.
2. A 38 × 38 Cross-Lingual Transfer Matrix (1444 language pairs), described as the largest resource for empirically measured transfer benefits and interference between languages.
3. A scaling law for the curse of multilinguality, telling practitioners how much to scale N and D to accommodate expansions in a model's language coverage.
4. A general pretrain-versus-finetune formula, telling practitioners whether it is more efficient to pretrain from scratch or start from a multilingual Unimax checkpoint.

Main Findings

• Monolingual scaling is consistent in form across languages. Per-language optimal scaling trajectories are nearly parallel, indicating similar exponents and comparable returns to additional data and parameters (full law parameters are in Table C.1).

• There is a compute efficiency tax for multilingual training. Both multilingual vocabulary and Unimax training shift the loss frontier upward relative to monolingual vocabulary plus monolingual training. This tax is most pronounced for English, indicating English benefits less from language transfer than other languages do from English. For Hindi and Swahili the right tail bends upward, consistent with diminishing returns from severe data repetition after many epochs.

• ATLAS improves held-out generalization. In the monolingual setting, all scaling laws perform comparably across most dimensions, but laws that model data repetition generalize better to the largest model sizes: ATLAS reaches R²(N) = 0.88 versus 0.68 (Chinchilla Scaling Law) and 0.78 (Data-Constrained Scaling Law).

• Monolingual laws fail at the largest multilingual models. In the multilingual setting, monolingual scaling laws cannot generalize to the largest models on R²(N) at all; Chinchilla reports R²(N) = -0.99 and the Multilingual Scaling Law (MSL) reports -0.65 in Table 1. MSL does better on unseen language mixtures (Table 1 reports R²(M) = 0.70; the text states 0.69 > 0.64) by modeling the whole language family.

• Separating data sources is what makes ATLAS work. Adding the D_other term raises multilingual fit to 0.98 on the overall R², 0.89 on R²(N), 0.97 on R²(D), and 0.97 on R²(C). Only after adding the Transfer Language term (Σ over K_t of D_i) does the law outperform MSL on unseen mixtures, reaching R²(M) = 0.82.

• English is the most frequent top transfer source. English appears as a top-5 helpful source language in 19 of 30 targets, followed by French (16 of 30), Spanish (13 of 30), and Hebrew (11 of 30). Related languages such as Spanish/Catalan/Portuguese, Swedish/Norwegian, and Indonesian/Malay show notable positive transfer. Low-resource languages such as Urdu or Pashto show significant negative transfer with all other languages considered.

• Script and family predict transfer. Sharing a script or family produces statistically significant shifts in mean transfer score (always p < .001). Pairs sharing a writing system have a mean score of -0.23 versus -0.39 for pairs with disparate scripts. The larger effect size for script suggests surface-level representation and subword vocabulary sharing is a primary transfer mechanism.

• Transfer is not generally reciprocal. Across all language pairs the Pearson correlation of A-to-B versus B-to-A transfer is only r = -0.11, so observing that language A helps language B does not imply the reverse. Pairs sharing both family and script cluster near the identity line and in the top-right quadrant; (French, Spanish) and (Russian, Ukrainian) are highly symmetric, while (Chinese, Farsi) and (Russian, Vietnamese) are highly asymmetric.

• The curse of multilinguality is measurable but mild. Fitting per-target-language loss as L(K, N, D_t) = L_∞ + A·K^φ/N^α + B·K^ψ/D_t^β with D_tot = K·D_t gives a robust R² ≥ 0.87, with φ = 0.11 and ψ = -0.04. A positive φ means loss degrades from limited model capacity as languages are added; a negative ψ means less data per language is needed (though total data grows). The law was fit for each of 8 languages with R² > 0.8 for each.

• Model size helps more than data for language coverage. The partial derivatives satisfy |∂S/∂log N| > |∂S/∂log D|, indicating N delivers larger marginal gains than D_tot.

• Language-expansion iso-loss recipe. Expanding from K to r·K languages requires scaling N by r^(φ/α), per-language data by r^(ψ/β), and total data by r^(1+ψ/β). Expanding to 4·K languages requires D_tot to grow by 2.74 and N by 1.4, which corresponds to sampling 1 − (2.74/4) = 32% less data per language. The paper states practitioners should expand their compute budget by C · r^0.97 to expand language coverage by r.

• Pretrain versus finetune crossover. For 2B parameter models across eight languages, a finetuned warm-started model performs better at first, but pretraining from scratch surpasses it after 144B–283B tokens depending on the language. The guidance given is that finetuning a multilingual checkpoint is usually more effective below 144B tokens, and pretraining from scratch above 283B tokens. English converges fastest; it is allocated a 5% sampling rate within Unimax, while each of the other languages is allocated 1.4%.

• Crossover power law. Figure 7 reports C = 10283128 × N^1.65; the body text instead writes log(C) = 1113708 × N^1.65 for the same relationship. This model does not account for data repetition.

Methodology in Plain English

The authors train a very large set of controlled models and then fit mathematical curves to the resulting loss values.

Recipes and scale. All experiments use the MADLAD-400 dataset, a CommonCrawl-based corpus covering over 400 languages. Models range from 10M to 8B parameters, use a 64k SentencePiece vocabulary, and are trained with hyperparameters similar to those in Kudugunta et al. (2024). The paper trains 280 monolingual models, 240 bilingual models, 120 multilingual mixtures, and finetunes 130 monolingual models from Unimax checkpoints — over 750 independent training runs. The abstract and Section 2 describe the parameter range as 10M–8B, while Section 1 describes the 774 experiments as spanning 10M–2B. Core languages for scale, token, and mixture sweeps are English, French, Chinese, Hindi, Swahili, Russian, Spanish, Portuguese, and Japanese, with some experiments evaluating across 50 languages. The abstract reports 48 evaluation languages; Section 2 states that a random test set is partitioned for 50 languages to evaluate vocabulary-insensitive loss for fairer cross-lingual comparison.

The ATLAS formula. ATLAS keeps the standard form L(N, D_eff) = E + A/N^α + B/D_eff^β, but replaces raw data with an "effective data exposure" term split into three parts: the target language's own data D_t; an optional set of transfer languages with learned weights τ_i; and a remainder term D_other covering all other training tokens. A saturation function handles repetition — the first pass over a language's unique tokens counts fully, and additional epochs decay smoothly according to a shared repetition parameter λ. Compared to the Data-Constrained Scaling Law, ATLAS uses one fitting stage instead of two, adds fewer parameters in the monolingual variant, and can be extended to an open-ended number of languages. The transfer weights are initialized from the language transfer scores derived elsewhere in the paper. The transfer set K_t is set to the 3 most highly co-sampled languages alongside the target across mixtures.

Evaluating generalization. Following Muennighoff et al. (2024) and He et al. (2024), the authors report R² on held-out test splits designed to isolate specific dimensions: R²(N) for the largest model sizes, R²(D) for the largest token ranges, R²(C) for the largest compute runs (with C = 6ND), and R²(M) for training language mixtures not seen during fitting. This separation responds to prior work emphasizing the importance of rigorous test-set splits. R² is averaged over languages [EN, FR, RU, ZH, HI, SW] in both settings, plus [ES, DE] for the multilingual setting.

Measuring language transfer. The Bilingual Transfer Score measures the relative training efficiency of a bilingual (s,t) model with a 50%/50% mixture compared to the monolingual model t in reaching the same loss. A pre-defined target step of 42B tokens defines d_mono; the score is zero-centered, so 0 means no transfer, positive means positive transfer, and negative means interference requiring more than double the monolingual steps. Scores were measured directly for 80 language pairs and estimated for the rest using other training signals with R² = 0.85 fidelity. Transfer matrices are computed on 2B parameter models; Figure 2 shows 30 × 30 and Figure C.2 shows the full 38 × 38.

Curse of multilinguality. To isolate the effect of language count K, the authors run experiments varying K while holding N or D fixed, sampling tokens from each language uniformly and not explicitly modeling repetition. Figure 4 fixes D = 25B tokens while varying model size, and fixes N = 2B parameters while varying tokens. A cubic spline provides a surface estimate over the raw empirical points.

Pretrain versus finetune. For eight languages, the authors compare 2B parameter models pretrained monolingually from scratch against models finetuned monolingually from the Unimax base model (a widely used mixture trained for 1B tokens), and interpolate where the loss curves cross.

Why This Matters

Impact on research. This is described as the largest multilingual scaling laws study to date, and it directly addresses two gaps in prior formulations: existing multilingual scaling laws account for only one of target-language data, total data, or target-plus-transfer-language data, and they do not model multi-epoch data repetition or cross-lingual transfer beyond the target's language family. The 38 × 38 transfer matrix is described as among the most expansive and rigorous fully symmetric transfer matrices, complementing prior work on syntactic/phonological distances and smaller-model transfer measurements between high- and low-resource languages.

Real-world applications.

• Choosing data mixtures for multilingual model training based on measured transfer scores rather than intuition, using the top source languages per target.
• Deciding how much to grow model size, per-language data, and total data when adding languages to an existing deployment, using the iso-loss frontier.
• Choosing between pretraining from scratch and finetuning a public multilingual checkpoint under a given compute budget.
• Estimating the compute efficiency cost of multilingual vocabulary or multilingual training mixtures relative to a monolingual setup.

Industry relevance. The paper notes that most major closed and open models now explicitly target massively multilingual uses (citing OpenAI, Anthropic, Google DeepMind, DeepSeek-AI, and OLMo), yet proprietary lab reports cover multilingual scaling only briefly. The pretrain-versus-finetune heuristic and the transfer matrix are framed as directly actionable for practitioners who may start from public multilingual checkpoints or train one checkpoint to serve many downstream models. The authors caution that not all multilingual base models are trained with the same mixture or for sufficiently long, and that those factors would affect the crossover points.

Future Directions

• Understanding why convergence differences exist between pretrained-from-scratch and finetuned models. The paper explicitly leaves the reason for these convergence differences to future work, stating it has no clear hypothesis.
• Extending the transfer matrix and transfer weighting to more languages. The transfer weights τ_i are currently initialized from the transfer scores and selected as the 3 most highly co-sampled languages alongside the target.
• Testing the pretrain-versus-finetune crossover with other multilingual checkpoints, mixtures, and training lengths, since the current estimate uses one Unimax mixture trained for 1B tokens.
• Refining the curse-of-multilinguality law, which currently assumes uniform sampling across languages and does not explicitly model token repetition.

Target Audience

Practitioners and researchers who design multilingual pretraining mixtures, set compute budgets for multilingual foundation models, or study scaling laws. It is most useful to engineers deciding language coverage and data allocation, and to researchers extending scaling-law formulations with cross-lingual transfer and repetition terms. Readers without a background in scaling-law fitting or compute-optimal training will find the equations dense, though the transfer matrix findings and the practical pretrain-versus-finetune guidance are accessible on their own.

Authors’ abstract

Scaling laws research has focused overwhelmingly on English -- yet the most prominent AI models explicitly serve billions of international users. In this work, we undertake the largest multilingual scaling laws study to date, totaling 774 multilingual training experiments, spanning 10M-8B model parameters, 400+ training languages and 48 evaluation languages. We introduce the Adaptive Transfer Scaling Law (ATLAS) for both monolingual and multilingual pretraining, which outperforms existing scaling laws' out-of-sample generalization often by more than 0.3 R^2. Our analyses of the experiments shed light on multilingual learning dynamics, transfer properties between languages, and the curse of multilinguality. First, we derive a cross-lingual transfer matrix, empirically measuring mutual benefit scores between 38 x 38=1444 language pairs. Second, we derive a language-agnostic scaling law that reveals how to optimally scale model size and data when adding languages without sacrificing performance. Third, we identify the computational crossover points for when to pretrain from scratch versus finetune from multilingual checkpoints. We hope these findings provide the scientific foundation for democratizing scaling laws across languages, and enable practitioners to efficiently scale models -- beyond English-first AI.