AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages

AI summary

Overview

Research area: Natural Language Processing — multilingual large language model adaptation, continued pre-training (CPT), and low-resource language modeling with a focus on African languages.

Technical level: Intermediate. The paper assumes familiarity with concepts such as continued pre-training, data mixtures, catastrophic forgetting, and standard multilingual benchmarks, but its central arguments are about experimental design rather than new model mathematics.

Scope: The paper presents AfriqueLLM, a suite of open LLMs adapted to 20 African languages through continued pre-training on 26B tokens, and uses controlled ablations to show how the composition of the CPT data mixture and the choice of base-model architecture determine downstream performance.

What This Paper Is About

Open LLMs still lag behind proprietary systems on African languages, and while continued pre-training is a practical way to adapt a model to new languages, gains on demanding capabilities such as mathematical reasoning remain limited. The authors argue this is partly because low-resource corpora have uneven domain coverage and are missing task-relevant knowledge. Their goal is to determine empirically which matters more for CPT success — the data mixture, the base model's existing language coverage, or the base model's underlying capability — by adapting several open base models with systematically varied 26B-token mixtures.

Key Contributions

1. AfriqueLLM model suite. A collection of open LLMs adapted to 20 African languages via continued pre-training on 26B tokens, covering Llama 3.1 8B, Gemma 3 (4B and 12B), and Qwen 3 (4B, 8B, and 14B). The abstract describes this as "five base models spanning sizes and architectures"; the experimental section enumerates the six checkpoints above across three families.

2. A curated 26B-token CPT corpus and mixture variants. African monolingual text drawn from FineWeb2, WURA, and MADLAD-400, plus four high-resource languages (English, French, Portuguese, Arabic) capped at 1B tokens each, combined with Python code, educational math, machine-translated synthetic data, and filtered NLLB parallel data. This yields named mixtures such as CM, CMS, and CMSP (24.9B, 25.2B, and 25.6B tokens respectively).

3. A systematic data-mixture ablation. An experiment on Gemma 3 4B and 12B isolating the effect of monolingual data, code, math, synthetic data, and parallel data on Flores, AfriMGSM, AfriMMLU, and AfriXNLI.

4. A cross-architecture and cross-scale study. The CMS recipe is applied across the Llama 3.1, Gemma 3, and Qwen 3 families, plus additional ablations on the Qwen 3.5 4B base, an extended code/math budget (ExtendedCM), and expansion from 20 to 50 African languages.

Main Findings

• Data mixture is the primary driver of CPT gains. Adding code, math, and synthetic translated data consistently improved performance, including on reasoning-oriented evaluations. The authors adopt CMS (monolingual + code/math + synthetic) as their primary recipe.

• Monolingual-only adaptation helps non-reasoning tasks but can hurt reasoning. Adding only monolingual data (22B tokens) improved Flores and AfriXNLI by over 10% relative to the base model, but for the 12B model the authors observed declines on AfriMGSM (24.1 to 23.8) and AfriMMLU (48.2 to 46.7). Adding roughly 2B tokens of code and math reversed this trend.

• Larger models are more sensitive to noisy parallel data. Parallel NLLB data (filtered at a 0.7 SSA-COMET threshold, yielding 4M samples, about 456M tokens) gave marginal gains for the 4B model but was detrimental for the 12B model compared with CMS. CMS achieved the highest 12B scores on AfriMGSM (30.9) and Flores (66.2), whereas CMSP reduced performance.

• A "Zero-to-Hero" effect in Qwen 3. Qwen 3 posted the largest relative improvements: 74.4% (4B), 76.5% (8B), and 57.8% (14B) over their respective base models, versus 18.8% (4B) and 7.3% (12B) for Gemma 3, despite Qwen 3 having the weakest African-language baseline (Qwen 3 8B average 33.47).

• Architecture can outweigh scale. AfriqueQwen-14B scored 63.58 overall, ahead of Gemma 3 27B at 60.71 by +2.87 points, with advantages of +9.64 on AfriMGSM and +15.36 on Flores, at less than half the size. Within a fixed architecture, larger models generally performed better.

• Base-model multilingual coverage does not predict post-CPT results. Strong foundation ability and task-aligned data were more predictive than prior language familiarity. Switching from Qwen 3 4B (base 31.49) to the more multilingual Qwen 3.5 4B (base 46.01) raised the absolute post-CPT score (57.12 vs. 54.94) but shrank the relative gain (+24.2% vs. +74.4%).

• Gains concentrate on languages seen during CPT. All models improved substantially on CPT-covered African languages, with Qwen 3 8B showing the highest gain (+24.9 points). Languages absent from both base pre-training and CPT (e.g., Ewe, Lingala) remained largely unchanged; AfriqueQwen models showed modest positive transfer (+4.4, +2.7), which the authors attribute to cross-lingual transfer within the same language family.

• Forgetting of high-resource languages is contained but uneven. Llama 3.1 8B showed the steepest average decline on English and French (-10.2% relative), followed by Gemma 3 (-5.6% to -6.2%). The Qwen 3 series showed the smallest degradation (-1.6% for 8B, -3.8% for 14B).

• Low-resource languages benefited most within the covered set. Across 11 CPT-covered African languages, AfriqueQwen-8B exceeded +25 absolute points in 10 of 11 languages, peaking at +37.1 in Hausa. Oromo and Yoruba showed the largest deltas among Gemma 4B adaptations (+16.1 and +17.2), while Swahili gains were more modest.

• Strong long-context document translation without in-domain training. On AfriDoc-MT (health domain, k=10 pseudo-documents, 3-shot, d-chrF), AfriqueGemma-12B reached the best eng-to-African average (60.2), beating the task-specific Llama 3.1 8B SFT10 baseline (47.8) by +12.4. For African-to-English, AfriqueQwen-14B led at 73.5 and AfriqueGemma-12B followed at 73.3, both above the SFT10 baseline of 68.0.

• Adding unseen languages to the corpus unlocks large gains. Extending the CPT corpus from 20 to 50 African languages matched the ExtendedCM score exactly on the original 20 languages (58.30 vs. 58.30) and raised the overall score on newly added benchmark-covered languages from 31.97 to 48.58 (+16.61 points, +52.0%), with Flores rising from 24.06 to 56.33 and Injongo from 32.41 to 61.37.

• Comparison with prior African CPT work. Against Lugha-Llama-8B-wura (45.94 overall) on the same Llama 3.1 8B base, AfriqueLlama-8B (49.41) outperformed it on 4 of 7 tasks, notably reasoning (AfriMGSM 17.51 vs. 9.46) and translation (Flores 63.60 vs. 49.90).

Methodology in Plain English

The team started by assembling a training corpus rather than designing a new model. They pooled African-language web text from three existing sources and added documentation-level data and extra coverage for lower-resource languages. Because African corpora are highly imbalanced, they used UniMax sampling: high-resource languages were capped at roughly 1B tokens, while lower-resource languages were upsampled for up to five epochs. To limit forgetting of high-resource ability, they included English, French, Portuguese, and Arabic, also capped at 1B tokens each.

They then layered in data intended to preserve reasoning: about 1B tokens of Python code from CornStack, about 1B tokens of educational math from FineMath-4+, and 324M tokens of GPT-4.1 machine translations covering ten Web Organizer domains plus OpenMathReasoning chain-of-thought material. They also tested 456M tokens of filtered NLLB parallel data.

Using the LLaMA-Factory framework on up to 16 nodes and 64 NVIDIA H100 GPUs, they ran hyperparameter searches on the Gemma 3 models: a learning rate sweep from 1e-6 to 2e-4 settled on 5e-5; context windows of 4k, 16k, and 32k were compared, with 16k performing best on AfriMGSM; and a cosine schedule with minimum learning rate ratio 0.01 and warmup ratio 0.001 was used, with a global batch size of 4M tokens.

Evaluation used the AfroBench-Lite subset: AfriMGSM, AfriMMLU, AfriXNLI, Belebele, Flores, Injongo, and SIB-200, with 5-shot prompting except 8-shot for AfriMGSM, and SSA-COMET rather than ChrF++ for translation. Each recipe variant was first validated on Gemma 3 4B and 12B, then the winning recipe was scaled to Llama 3.1 8B and the Qwen 3 models. Additional ablations examined a more multilingual Qwen 3.5 4B base, a five-times-larger code/math budget, and corpus expansion from 20 to 50 languages.

Why This Matters

Impact on research. The paper reframes CPT for low-resource languages as a data-composition problem rather than purely a scale problem. Its finding that noisy parallel corpora help small models but hurt large ones, and that a base model's general capability predicts CPT outcomes better than its prior language coverage, gives concrete guidance for future adaptation experiments and challenges the assumption that more multilingual pre-training automatically translates into better low-resource results.

Real-world applications:

• Machine translation for African languages, including document-level translation, where several Afrique models outperformed a task-specific supervised fine-tuned baseline on the AfriDoc-MT health domain.
• Educational and reasoning tools in African languages, since the code and math additions were what recovered performance on AfriMGSM and AfriMMLU.
• Knowledge and classification services such as intent classification and topic classification, where Injongo and SIB-200 scores improved substantially after adaptation.
• Deployment in resource-constrained settings, because the strongest results were achieved by models under 15B parameters rather than by scaling to much larger ones.

Industry relevance. The paper provides a reproducible data-mixture recipe and shows that architecture choice can substitute for parameter count — AfriqueQwen-14B beating Gemma 3 27B overall. For organizations serving African-language users, this is a lower-cost path to usable multilingual models, and the release of models on HuggingFace and code on GitHub makes the recipe directly reusable.

Future Directions

• Extending beyond 20 languages. The authors note that hundreds of African languages remain unsupported, and that languages with minimal digital presence remain challenging. Their own 50-language experiment shows that explicit in-language data is required for gains, since multilingual priors alone produced essentially no transfer to the newly added benchmark-covered languages.

• Scaling past 14B parameters. Resource constraints limited experiments to 14B; the authors state that larger models (30B+) may show different adaptation dynamics, naming Qwen3-30B-A3B-Base and Gemma 3 27B PT as candidates.

• Instruction tuning for African languages. This work covers base-model CPT only, and the authors identify the scarcity of high-quality instruction data as a bottleneck for downstream deployment.

• Training stability and efficiency. Intermittent gradient norm spikes were observed, prompting the authors to suggest matrix optimizers such as Muon. They also report model FLOPs utilization of 31–34% and point to specialized frameworks such as Megatron-LM as a path to improvement.

Target Audience

Researchers and engineers working on multilingual or low-resource language modeling, particularly those planning continued pre-training runs and deciding how to allocate a token budget between monolingual text and reasoning-oriented data. It is also relevant to practitioners building African-language products who need to choose a base model and adaptation recipe, and to NLP researchers studying catastrophic forgetting, cross-lingual transfer, and the relationship between base-model capability and adaptation returns.

Authors’ abstract

Large language models (LLMs) are increasingly multilingual, yet open models continue to underperform relative to proprietary systems, with the gap most pronounced for African languages. Continued pre-training (CPT) offers a practical route to language adaptation, but improvements on demanding capabilities such as mathematical reasoning often remain limited. This limitation is driven in part by the uneven domain coverage and missing task-relevant knowledge that characterize many low-resource language corpora. We present \texttt{AfriqueLLM}, a suite of open LLMs adapted to 20 African languages through CPT on 26B tokens. We perform a comprehensive empirical study across five base models spanning sizes and architectures, including Llama 3.1, Gemma 3, and Qwen 3, and systematically analyze how CPT data composition shapes downstream performance. In particular, we vary mixtures that include math, code, and synthetic translated data, and evaluate the resulting models on a range of multilingual benchmarks. Our results identify data composition as the primary driver of CPT gains. Adding math, code, and synthetic translated data yields consistent improvements, including on reasoning-oriented evaluations. Within a fixed architecture, larger models typically improve performance, but architectural choices dominate scale when comparing across model families. Moreover, strong multilingual performance in the base model does not reliably predict post-CPT outcomes; robust architectures coupled with task-aligned data provide a more dependable recipe. Finally, our best models improve long-context performance, including document-level translation. Models and code have been released on [Huggingface](https://huggingface.co/collections/McGill-NLP/afriquellm) and [Github](https://github.com/McGill-NLP/AfriqueLLM).