The Pulse
Apple Study Finds English Data Outperforms More Arabic Hyperparameter Tuning
Apple’s October 2026 research post describes experiments in which adding English data to Arabic language-model training beat hyperparameter tuning alone. Across models from 150 million to 1.43 billion parameters, the study found that mixing
AI.info Team · Written with AI ·
Two ways to stretch scarce Arabic data
When a language-model team runs short of Arabic text, it can try to squeeze more from the existing corpus by tuning training settings—or bring in English data. A new study highlighted by Apple Machine Learning Research finds the second approach generally delivered larger gains in the experiments, even when the tuning comparison included broad searches over learning rate and weight decay.
The paper, “Mix, Don’t Tune,” compares those strategies in roughly 1,000 pre-training runs. The team trained models ranging from 150 million to 1.43 billion parameters, using Arabic as the target language and English as a higher-resource auxiliary language. Across the larger model sizes, adding English improved Arabic validation loss more than hyperparameter tuning alone, and the gap grew with scale.
English mixing pulled ahead as models grew
The experiments use 200 million unique Arabic tokens, with models trained on repeated Arabic data or on mixtures of Arabic and English. The researchers varied the Arabic share alongside the training settings. At 1.43 billion parameters, mixing improved Arabic validation loss by 12.9% against the monolingual baseline; hyperparameter tuning alone improved it by 3.1%. Tuning on top of mixing added less than 1% on that metric at each scale.
“Mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size,” write Paul Jeha, Anastasiia Sedova, Louis Béthune, Skyler Seto, Jes Frellsen, Pierre Ablin and Natalie Schluter, authors of the paper.
The preferred mix shifted toward more English at larger sizes. In the tuned experiments, the optimal Arabic fraction fell from 0.33 at 380 million parameters to 0.10 at 1.43 billion. The result suggests that repeatedly showing a larger model the same limited Arabic corpus can make it more prone to overfitting, while fresh English text diversifies its training signal.
Benchmark gains exceeded what validation loss showed
The researchers also evaluated models on ARC Easy, a knowledge-focused multiple-choice benchmark tested in English and Arabic. For the Arabic evaluation, they translated the English questions using NLLB-200-3.3B. At 1.43 billion parameters, bilingual training with transferred hyperparameters matched the benchmark accuracy of a monolingual model trained on the equivalent of 1.6 billion unique Arabic tokens, despite using 200 million unique Arabic tokens in the bilingual setup.
When the bilingual model also received a full hyperparameter sweep, the paper estimated its benchmark performance matched a monolingual model trained on 13.4 times as much unique Arabic data—equivalent to 2.7 billion tokens. That estimate comes from the study’s fitted comparison, not from a direct training run at 2.7 billion Arabic tokens. The difference between that benchmark result and the smaller gain on Arabic validation loss is central to the authors’ warning: validation loss alone can miss some downstream benefits of language mixing.
One language pair leaves a clear limit
The result is narrower than a general rule for multilingual model training. The study tests one language pair, Arabic and English, and its downstream claims rest on ARC Easy; the Arabic version is machine-translated rather than natively written. The authors also note that translation effects could contribute to the apparent advantage for models trained on English.
For the setting they tested, the authors recommend prioritizing the language-mixing ratio over an extensive hyperparameter search and transferring training settings from a smaller model using maximal update parametrization, or μP. Their evidence is strongest for Arabic-English pre-training under data scarcity: at 1.43 billion parameters, the benchmark comparison pairs 200 million unique Arabic tokens in bilingual training with 2.7 billion equivalent tokens for a monolingual model after additional tuning.