Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates

AI summary

Overview

• Research area: Natural Language Processing — cross-lingual adaptation of instruction-tuned large language models, with a focus on continual pre-training and catastrophic forgetting.
• Technical level: Intermediate. The method itself is conceptually simple (score parameters, freeze some, train), but understanding the results requires familiarity with continual pre-training, parameter-efficient adaptation, and standard LLM evaluation benchmarks.
• Scope: The paper proposes a selective parameter-update strategy called Source-Shielded Updates (SSU) that protects source-language abilities while adapting instruct LLMs using only unlabeled target-language text, and validates it on five languages at two model scales.

What This Paper Is About

Instruct LLMs perform poorly on languages that are underrepresented in their training data, but adapting them usually requires expensive, specialized target-language instruction data — and training on raw target-language text tends to erase the model's original abilities (catastrophic forgetting), especially the chat and instruction-following behavior that makes it useful in the first place. This paper introduces SSU, which identifies the model parameters that matter most for the source language using a small sample of source data, freezes them in a structured way before adaptation begins, and then continually pre-trains on unlabeled target-language text. The goal is to gain target-language proficiency comparable to full fine-tuning while retaining nearly all source-language capability.

Key Contributions

1. A source-focused method for target-language adaptation without target instruction data. SSU adapts instruct models using only raw, unlabeled target-language text, addressing a key bottleneck in expanding linguistic accessibility.
2. Proactive, parameter-level shielding during adaptation, not after. Rather than post-hoc model merging or task-vector approaches, SSU scores parameter importance on source data before training and freezes the most critical structural units.
3. A column-wise freezing strategy. Element-wise importance scores are aggregated into column-level scores, and entire weight-matrix columns (complete input-feature pathways) are frozen, preserving feature transformations intact.
4. Extensive validation across five languages and two model scales. Experiments with 7B and 13B OLMo 2 Instruct models, plus ablations on freezing structure, importance metric, calibration data, and freezing ratio, and a qualitative analysis of code-mixing.

Main Findings

• Large reduction in source-language forgetting. SSU reduces performance degradation on monolingual source tasks to 3.4% (7B) and 2.8% (13B) on average, compared with 20.3% and 22.3% for full fine-tuning.
• Best performance on chat and instruction-following. SSU-Wanda shows the smallest average relative drops from the Source model — 5.9% (7B) and 4.7% (13B). AdaLoRA is second at 9.0% (7B) and 6.1% (13B); HFT drops 18.0% (7B) and 15.1% (13B); the target-data-driven GMT drops 27.7% (7B) and 26.3% (13B); and full fine-tuning (FFT) drops 34.1% (7B) and 32.3% (13B).
• Safety alignment is preserved. SSU-Wanda loses only 0.1% (7B) and 2.0% (13B) on the Tülu 3 safety suite. FFT and GMT drop by up to 10.2%.
• Source-language monolingual tasks are nearly untouched. Relative drops on SUM, MRC and MMLU never exceed 2.0% (7B) and 1.0% (13B). Target-to-English machine translation improves by up to 52.3% relative to the Source model.
• Target-language gains rival or beat full fine-tuning. The abstract reports SSU outperforming FFT on all benchmarks for 7B models and the majority for 13B models; Section 5 describes this as all benchmarks for 7B and half for 13B.
• Column-wise freezing beats row-wise and element-wise freezing. Column-wise achieves the best chat, safety and source-language performance while staying competitive on target tasks, with only a 1.2 point drop on MT relative to element-wise freezing.
• Source-driven scoring is essential; the specific metric is not. Data-free variants lag far behind, with SSU-Rand at 18.2% (7B) and 16.0% (13B) below Source on chat/instruction-following, and SSU-Mag at 23.0% (7B) and 21.7% (13B). By contrast, alternative source-driven scorers perform similarly to Wanda: SSU-SparseGPT shows a 4.3% average drop on monolingual source tasks and SSU-FIM 3.5%, versus 4.0% for SSU-Wanda.
• A trade-off exists in the freezing ratio. Sweeping from 0% to 87.5% in 12.5% increments on Igbo shows source performance (chat, safety) generally improving with more freezing, while target performance degrades — with a sharp MMLU drop after 37.5%. The authors recommend 50% for balanced use, at least 60% for source priority, and at most 40% for target priority.
• Logic and reasoning abilities are shielded, not just English. On HumanEval, the 7B SSU-Wanda model shows a 10.4% average relative drop versus 49.7% for FFT, and the 13B model only 4.2%.
• Strong performance under extreme data scarcity. With only 20M tokens (10% of the default adaptation set), SSU-Wanda is best or second best on 10 of 13 tasks.
• Less code-mixing. On AlpacaEval 2.0, SSU restricts code-mixing to 1.0% of responses on average for 7B models, compared with 6.4% for HFT and 16.9% for GMT.
• Robust to calibration data choice. Using Alpaca instead of the original OLMo 2 instruction-tuning data produces a maximum difference of 1.0, and reducing calibration samples from 500 to 128 yields negligible differences.

Methodology in Plain English

The researchers start from an instruct model and a small sample of source-language data (500 samples at a sequence length of 2,048, drawn from OLMo 2's original instruction-tuning mixture). They then run three stages:

1. Score parameter importance. Using the Wanda pruning criterion, each weight θ_ij gets a score equal to its absolute magnitude multiplied by the L2 norm of its corresponding input activations (|θ_ij| · ||X_j||_2). This highlights weights that are both large and consistently active on source data. Embeddings and the language modeling head are excluded from scoring and are always updated, following prior work.
2. Build a column-wise mask. Scores are summed per column (per input feature) to produce a total importance S_j for each column. Columns are ranked by S_j, and the top k% — 50% by default — are frozen, with the mask set to 0 there and 1 elsewhere. For 1D parameters such as biases, each element is treated as its own column.
3. Continually pre-train on target data. The model is trained with a standard causal language modeling objective on unlabeled target-language text, with the static mask multiplied into the gradients so frozen columns never update. The chat template is removed during this training, since raw text lacks role annotations.

Evaluation covers five typologically diverse, low-resource target languages (Nepali, Kyrgyz, Amharic, Hausa, Igbo) that appear at least 840x less frequently than English in Common Crawl. Adaptation uses 200M tokens per language from a clean subset of MADLAD-400. Baselines are the unadapted Source model, full fine-tuning (FFT), AdaLoRA, HFT (a static random-freezing method), GMT (a dynamic gradient-based method), and two data-free ablations of SSU itself. Evaluation spans AlpacaEval 2.0, IFEval, MT-Bench, GSM8K, the Tülu 3 safety suite, FLORES-200 MT, XL-SUM summarization, Belebele reading comprehension, MMLU, and Global MMLU.

Why This Matters

Impact on research. The paper reframes catastrophic forgetting mitigation as something to do during adaptation rather than after it, and shows that source-data-driven, structurally coherent freezing outperforms both random freezing and target-data-driven selection. It also offers evidence that the specific importance metric matters less than whether the score is derived from source data, since SparseGPT and Fisher Information Matrix scores perform comparably to Wanda.

Real-world applications:

• Localizing AI assistants for speakers of languages such as Nepali, Kyrgyz, Amharic, Hausa and Igbo without commissioning costly native-language instruction datasets.
• Deploying safety-aligned chat systems in new markets while keeping the original safety behavior intact.
• Building translation, summarization and question-answering tools for underrepresented languages from raw text corpora alone.
• Preserving coding and reasoning ability in localized models, which matters when the same model serves both general and technical queries.

Industry relevance. The method needs no labeled target-language instruction data, which is often unavailable or prohibitively expensive to create, and machine-translated data is not consistently effective. Because SSU is a static masking approach, it fits into existing training pipelines and can be combined more easily with orthogonal techniques such as regularization and replay. It also reduces the need to re-run safety alignment after adaptation.

Future Directions

• Scaling beyond the tested settings. The main experiments cover five languages and 7B and 13B OLMo 2 models; generalization to OLMo 3 is reported in Appendix D.4, but broader language and model-family coverage remains open.
• Tuning the source–target trade-off. The freezing-ratio sweep reveals a clear tension between preserving source abilities and acquiring target-language proficiency, and the authors only offer loose guidance (50% balanced, at least 60% for source priority, at most 40% for target priority).
• Combining SSU with other mitigation families. The paper notes that static methods integrate more easily with orthogonal approaches like regularization and replay, which is flagged as a promising direction rather than tested.
• Broadening baseline coverage. The Limitations section states that the main comparison is against HFT and GMT, with additional comparisons to LoTA and S2FT in Appendix D.3, and mentions source data mixing plus model merging and post-hoc pruning as strategies not fully explored.

Target Audience

NLP and machine learning researchers working on multilingual adaptation, continual learning, and parameter-efficient or selective fine-tuning; engineers responsible for localizing LLM products into low-resource languages; and practitioners who need to expand a model's language coverage without sacrificing instruction-following, safety, or reasoning ability.

Authors’ abstract

Expanding the linguistic diversity of instruct large language models (LLMs) is crucial for global accessibility but is often hindered by the reliance on costly specialized target language labeled data and catastrophic forgetting during adaptation. We tackle this challenge under a realistic, low-resource constraint: adapting instruct LLMs using only unlabeled target language data. We introduce Source-Shielded Updates (SSU), a selective parameter update strategy that proactively preserves source knowledge. Using a small set of source data and a parameter importance scoring method, SSU identifies parameters critical to maintaining source abilities. It then applies a column-wise freezing strategy to protect these parameters before adaptation. Experiments across five typologically diverse languages and 7B and 13B models demonstrate that SSU successfully mitigates catastrophic forgetting. It reduces performance degradation on monolingual source tasks to just 3.4% (7B) and 2.8% (13B) on average, a stark contrast to the 20.3% and 22.3% from full fine-tuning. SSU also achieves target-language performance highly competitive with full fine-tuning, outperforming it on all benchmarks for 7B models and the majority for 13B models.