Continual Adaptation for Pacific Indigenous Speech Recognition

AI summary

Overview

Research area: Automatic speech recognition (ASR), specifically cross-lingual and continual adaptation of speech foundation models to low-resource, linguistically distant languages.

Technical level: Intermediate. The paper assumes familiarity with ASR metrics (word error rate, character error rate), fine-tuning versus parameter-efficient adaptation (LoRA and its variants), and the idea of catastrophic forgetting in neural networks.

Scope: An empirical study that fine-tunes Whisper-Small on a newly curated three-language Pacific Indigenous speech corpus and measures not only recognition accuracy but also internal representational drift and forgetting of previously learned languages.

What This Paper Is About

Speech foundation models such as Whisper and wav2vec 2.0 are widely assumed to be broadly adaptable to new languages, but that assumption has mostly been tested on well-documented languages that resemble their pretraining data. This paper asks whether adaptation to Pacific Indigenous languages that are both low-resource and structurally distant instead forces models to rewrite their internal representations, causing catastrophic forgetting. The authors build a new corpus of Bislama, Nafsan, and Lelepa, adapt Whisper-Small to each, and then track both accuracy and internal representation change across languages learned in sequence.

Key Contributions

1. A new Pacific Indigenous speech corpus. The authors curate three underrepresented languages from material held by the Pacific and Regional Archive for Digital Sources in Endangered Cultures (PARADISEC): Bislama, Nafsan, and Lelepa, totalling 23,843 audio samples and 32.13 hours of transcribed speech (13.75, 14.83, and 3.55 hours respectively).

2. A cross-lingual transfer benchmark under controlled supervision levels. Whisper-Small is fine-tuned with progressively increasing data (0.5, 1.0, 2.0, 5.0, and 10.0 hours where available) using two strategies, full fine-tuning and LoRA on the encoder and decoder, evaluated by character error rate (CER) and word error rate (WER).

3. A layer-wise representational drift analysis. The paper computes cosine distance between pre- and post-adaptation hidden states across all twelve encoder and twelve decoder layers, min-max normalised per language, to show where inside the model the structural change occurs.

4. A continual learning evaluation with parameter-efficient variants. Sequential Nafsan to Lelepa learning is tested with full fine-tuning, LoRA, DoRA (Weight-Decomposed Low-Rank Adaptation), and O-LoRA (Orthogonal Low-Rank Adaptation), alongside a component-specific ablation isolating encoder-only, decoder-only, and encoder-plus-decoder updates.

Main Findings

• More data helps, but the rate of improvement is language-dependent. Under full fine-tuning, Bislama reaches its lowest reported WER of 19.64 at 10.0 hours and already improves quickly at 1.0 hour (WER 32.95). Nafsan stays unstable between 0.5 and 2.0 hours (test WER 71.96, 69.33, 72.18) and only shows a clear gain at 5.0 hours (WER 52.05), reaching 47.84 at 10.0 hours.

• Full fine-tuning usually beats LoRA, except under extreme scarcity on the most distant language. For Bislama and Nafsan, full fine-tuning achieves lower error rates than LoRA at every time setting shown. For Lelepa, full fine-tuning is better at 0.5 hours (WER 79.16 vs 96.50) and 1.0 hour (WER 84.23 vs 92.48), but LoRA overtakes it at 2.0 hours with a WER of 75.66 versus 84.10.

• Adaptation induces severe internal representational drift. Bislama and Nafsan change mostly in the later encoder layers, suggesting the model reuses early acoustic features and only updates higher-level phonetic representations. Lelepa, by contrast, shows significant drift in the early encoder layers, meaning its basic acoustic properties differ strongly from pretraining data.

• Decoder drift profiles differ sharply. Bislama and Nafsan show active adjustment across most middle and late decoder layers, while Lelepa shows almost no drift between layers 2 and 10 with a sudden peak at the final layer.

• Catastrophic forgetting occurs even for languages seen during pretraining. After adapting to Lelepa, full fine-tuning (updating 244M parameters) raises English test WER from 15.68 to 26.24, Chinese test CER from 34.62 to 45.02, and French test WER from 24.27 to 40.26. LoRA (updating 20.1M parameters) forgets less but still noticeably: English WER 18.89, Chinese CER 36.71, French WER 27.82.

• There is a strict trade-off between target adaptation and source retention. Decoder-only LoRA minimises forgetting (English WER 18.27) but fails to learn Lelepa (CER 34.68). Encoder-only LoRA improves Lelepa recognition to 29.87 CER but pushes English WER to 31.26, worse than full fine-tuning. Encoder-plus-decoder LoRA gives Lelepa CER 28.41 and English WER 21.24, while encoder-plus-decoder full fine-tuning gives Lelepa CER 27.27 and English WER 26.24.

• In sequential learning, full fine-tuning wins on average but fails on the new language. In the Nafsan to Lelepa sequence, full fine-tuning keeps Nafsan WER at 45.67 after learning Lelepa but reaches only 83.72 WER on Lelepa, for an average WER of 64.70 percent, the lowest of all strategies tested. LoRA reaches 68.62 WER on Lelepa but Nafsan degrades to 84.42 (average 76.52 percent), DoRA reaches Lelepa 70.82 and Nafsan 84.80 (average 77.81 percent), and O-LoRA reaches Lelepa 69.08 and Nafsan 87.42 (average 78.25 percent).

• Current methods leave the plasticity-stability dilemma unresolved. Parameter-efficient methods provide better immediate plasticity on a new Pacific language but fail to protect historical knowledge, while full fine-tuning preserves history but cannot learn the new language well.

Methodology in Plain English

The authors gathered recordings with transcriptions for three Pacific languages from the PARADISEC archive. Bislama is an English-based creole and the national common language, combining English vocabulary with Melanesian grammar; Nafsan and Lelepa are Austronesian Indigenous languages that are still spoken but lack text resources.

They took Whisper-Small, a multilingual speech model pretrained mostly on high-resource languages, and adapted it to each language. To keep the comparison fair, each language was split 80 percent training, 10 percent validation, and 10 percent test, stratified by speaker where possible. Audio was resampled to 16 kHz mono. Training used AdamW with a peak learning rate of 1e-4, batch size 16, a 500-step linear warm-up, and cosine decay. The Whisper vocabulary was extended with characters seen in the three languages' transcripts, and the new token embeddings were initialised as the average of the pretrained vocabulary. Mild augmentation (random time-shifts and additive in-domain background noise) was applied. Every language, data-budget, and strategy combination was trained three times with different random seeds, and averaged CER and WER are reported.

Two comparisons were run: full fine-tuning of every parameter versus LoRA applied to the encoder and decoder. For the drift analysis, a separate held-out evaluation subset of balanced utterances from all three languages was passed through each checkpoint in teacher-forced mode; hidden states from every encoder and decoder layer were cached and compared with the original pretrained checkpoint using cosine distance, then min-max normalised per language. For the continual learning experiments, the model learned languages in sequence (for example Nafsan then Lelepa), and after each stage it was tested on the newly learned language, the previously learned Pacific languages, and English to expose forgetting.

Not reported in the provided content: the number of distinct speakers, speaker demographics, and per-language speaker counts.

Why This Matters

Impact on research. The paper challenges the common assumption that pretrained multilingual speech representations are largely language-agnostic and universally adaptable. It argues that for linguistically distant languages, adaptation is not smooth refinement of shared features but substantial internal reorganisation, and that final WER alone hides this. It also shows that the encoder-versus-decoder binary choice is insufficient for unseen Pacific languages.

Real-world applications.

• Deploying speech interfaces in Pacific education, public services, and digital communication, where current systems fail to support these languages.
• Supporting language documentation and revitalisation work for endangered languages held in archives such as PARADISEC.
• Building transcription tools for communities that lack large-scale annotated corpora or standardised orthographies.
• Informing multilingual deployment in settings where a model must serve several languages without losing the ones it already knew.

Industry relevance. Any organisation adding languages to an existing ASR product faces the same stability-versus-plasticity problem. The results indicate that parameter-efficient methods alone are not a safe default for sequential language addition: they adapt quickly but degrade prior languages, and full fine-tuning is costly in parameters (244M updated versus 20.1M for LoRA) while often failing to learn the new distant language well.

Future Directions

• Developing adaptation methods that resolve the plasticity-stability dilemma rather than trading one for the other, since neither full fine-tuning nor LoRA, DoRA, or O-LoRA solves it here.
• Exploring dynamic architectures, which the authors explicitly name as a direction for future work.
• Handling unbalanced data across languages, since Bislama has substantially more data than Nafsan and this imbalance drives overfitting.
• Designing approaches that accommodate unique linguistic features of underrepresented languages and go beyond choosing between encoder and decoder updates.
• Extending the analysis to more Pacific languages and to other linguistically distant, low-resource families to test how general the drift and forgetting patterns are.

Target Audience

Researchers and practitioners in low-resource and multilingual speech recognition, especially those working on continual learning, parameter-efficient adaptation, or catastrophic forgetting. It is also relevant to computational linguists and language documentation teams working with endangered or Indigenous languages, and to engineers evaluating whether LoRA-style adaptation is safe for incrementally adding languages to a deployed ASR system.

Authors’ abstract

Speech foundation models struggle with low-resource Pacific Indigenous languages because of severe data scarcity. Furthermore, full fine-tuning risks catastrophic forgetting. To address this gap, we present an empirical study adapting models to real-world Pacific datasets. We investigate how data volume and linguistic features affect adaptation success. Specifically, we evaluate strategies including Full Fine-Tuning and Low-Rank Adaptation (LoRA). Additionally, we analyze a continual learning framework for sequentially acquiring multiple languages. We demonstrate that adapting to these distant languages causes severe internal representational drift. Consequently, these models face a strict plasticity and stability dilemma. While LoRA adapts well initially, it suffers from catastrophic forgetting during sequential learning. Ultimately, this study highlights the urgent need for robust adaptation strategies tailored to underrepresented languages.