Multimodal In-context Learning for ASR of Low-resource Languages

AI summary

Overview

Research area: Low-resource automatic speech recognition (ASR), speech large language models, and multimodal in-context learning.

Technical level: Advanced. The paper assumes familiarity with speech LLM architectures, in-context learning, LoRA fine-tuning, perplexity/WER evaluation, and attention analysis.

Scope: The paper studies whether speech LLMs can learn endangered, unseen languages from a handful of paired audio–text demonstrations, how cross-lingual fine-tuning changes that ability, and whether in-context learning can be used to re-rank acoustic model hypotheses for ASR.

What This Paper Is About

ASR systems cover only a small fraction of the world's more than 7,000 languages because labelled speech data is scarce. In-context learning lets a model adapt from a few demonstrations rather than from supervised training, but prior work on low-resource languages is mostly text-only and mostly tested on languages the model already saw in training. This paper asks whether speech LLMs can learn genuinely unseen languages from multimodal demonstrations that pair audio with text, and whether that ability can be turned into a working ASR system.

Key Contributions

1. A systematic study of multimodal in-context learning (MICL) for unseen languages. The authors define three prompt designs that isolate modality contributions — T-ICL (text-only demonstrations, no target audio), ICL (text-only demonstrations plus target audio), MICL (paired audio–text demonstrations plus target audio) — plus a no-demonstration ASR baseline, and evaluate them on two speech LLMs (Phi4 and Qwen3-Omni) across three endangered languages.
2. Evidence that cross-lingual instruction fine-tuning transfers to unseen languages. The authors fine-tune Phi4 on a multilingual MICL instruction dataset covering 143 languages that excludes the three evaluation languages, and show this improves both perplexity and ASR hypothesis selection.
3. An interpretability analysis of MICL through attention weights. They quantify how much attention flows to demonstration audio versus demonstration text, and how that allocation changes across layers and across the number of in-context samples.
4. A MICL-based ASR hypothesis selection system. Since direct prompting of speech LLMs produces poor ASR on unseen languages, they combine a fine-tuned MMS acoustic model with a speech LLM that re-ranks the acoustic model's 10-best hypotheses using a combined acoustic and LM score.

Main Findings

• MICL is effective for languages not seen in training. Both Phi4 and Qwen3-Omni show lower perplexity with in-context demonstrations than in the no-context ASR setting. Increasing the number of in-context samples generally lowers perplexity, and this holds even with prompts containing up to 100 samples.
• The target audio matters. Comparing T-ICL to ICL, adding the target audio consistently lowers perplexity, indicating the models use the acoustic modality and understand the ASR task beyond text prompting.
• The two models respond differently to paired audio–text demonstrations. For Qwen3-Omni, MICL outperforms ICL across the full range of context sizes evaluated, up to 100 samples. For Phi4, MICL tends to help mainly in low-shot settings, consistently outperforming ICL when the context contains at most three samples. The authors attribute the difference to training scale: Phi4 was trained on 2.3M hours of audio and supports eight languages, while Qwen3-Omni was trained on 20M hours of audio, supports speech understanding in 19 languages, and can process audio recordings up to 40 minutes long.
• Cross-lingual fine-tuning improves unseen-language performance. Fine-tuning Phi4 on 143 non-target languages improved performance on all three evaluation languages. On Kichwa, cross-lingual fine-tuning came close to target-language fine-tuning even though Kichwa was excluded from the fine-tuning data. Improvements from fine-tuned MICL prompts were observed with up to ten demonstrations, matching the maximum number of in-context samples (randomly selected from 1 to 10) used during fine-tuning.
• Attention is imbalanced toward text but shifts with more samples. On the Khinalug dataset with Phi4, demonstration attention split was 30.5% audio / 69.5% text at 1 sample, 31.9% / 68.1% at 3, 32.9% / 67.1% at 5, and 35.0% / 65.0% at 10. Audio representations were roughly three times longer than text representations in the Phi4 setting, so text receives disproportionate attention per unit of representation.
• Attention allocation is layer-dependent. Averaged across heads, the first and last layers attend more to audio, while intermediate layers attend more to text, a pattern the authors observed consistently across different numbers of in-context samples. They offer this as a hypothesis about acoustic integration, semantic processing, and re-attention to audio for final prediction, not as proven causal mechanism.
• Attention reflects actual context use. Replacing one in-context demonstration's transcription with gold text reduced perplexity from 68 to 9; replacing both audio and text reduced it to 2; replacing only the audio yielded almost no change (67). After the text replacement, attention to the corresponding audio sample increased (audio share rose from 35.0% to 43.3%), suggesting the model links paired text and audio.
• Prompt-based ASR alone is unusable on these languages. WERs remained high across models. Even the target-language fine-tuned MICL configuration, which performed relatively better on Mboshi (38.6 WER, the language with the most supervised data), still lagged the acoustic model.
• MICL hypothesis selection beats the acoustic baseline. WERs for the plain acoustic model were 42.1 (Khinalug), 17.3 (Kichwa), and 31.4 (Mboshi). MICL selection with Phi4 ASR-FT reached 41.5, 17.4, and 29.9; XFT reached 41.0, 17.1, and 29.6; TFT reached 40.8, 16.6, and 28.6. Qwen3-Omni selection reached 40.7, 17.2, and 30.0. Oracle selection (best hypothesis against ground truth) was 36.5, 12.4, and 22.1.
• Multimodal selection beats text-only language models. The text-only baselines were Ngram-LM (39.6, 17.7, 30.6), Trans-LM (41.6, 18.9, 30.9), and Llama-3-8B-Instruct (41.9, 17.0, 30.2). The multimodal speech LLMs outperformed Llama across all three languages. The authors note these comparisons are not fully controlled because of architecture and pre-training differences.
• More fine-tuning languages help. Perplexity on unseen languages decreased as language coverage grew. For Khinalug: 233 with no fine-tuning, then 156 (8 languages), 93 (16), 76 (32), 68 (64), 68 (All). Kichwa: 31, 33, 20, 16, 15, 11. Mboshi: 75, 49, 36, 33, 27, 28.
• Joint decoding outperforms hypothesis selection in this setup. With an n-gram LM, joint decoding gave 34.2 (Khinalug), 15.4 (Kichwa), and 27.3 (Mboshi), versus 40.8, 16.6, and 28.6 for hypothesis selection. Joint decoding even exceeded the hypothesis-selection oracle on Khinalug, the language with the least labelled data.
• The acoustic model is stronger than a fine-tuned speech LLM. When fine-tuned directly on ASR, MMS scored 42.1, 17.3, and 31.4 WER, while Phi4 scored 77.9, 40.3, and 42.3.
• Inference cost is measured but not trivial. Loading Phi4 requires 11 GB of GPU memory, and hypothesis selection adds another 11 GB for a total of 22 GB. Average inference time was 3 seconds per item on a single NVIDIA RTX 6000 GPU. The acoustic model used is approximately 20 times smaller than the speech LLM.

Methodology in Plain English

The researchers chose three endangered languages from different families and with different audio sources — Khinalug (spontaneous speech, 2.14 h train / 0.49 h dev+test), Kichwa (radio, 3.05 h / 0.77 h), and Mboshi (reading, 3.93 h / 0.53 h) — because such languages are likely to be absent from the training data of large speech models, making them a genuine test of learning from demonstrations rather than recalling prior training.

They built prompts that vary only in what the demonstrations contain: text alone without target audio, text alone with target audio, or paired audio and text with target audio, alongside a plain ASR prompt with no demonstrations. Demonstrations were chosen by embedding candidate samples and the target's predicted transcript with SONAR and picking the top-N by cosine similarity.

To test cross-lingual transfer, they fine-tuned Phi4 on MICL-format data from 143 ML-SUPERB 2.0 languages, none of which matched the three evaluation languages. Fine-tuning used LoRA on the decoder with the rest of the model frozen, with a loss computed only on target transcription tokens and the demonstrations treated as conditioning context.

Because running WER for the full configuration space was too expensive, they used perplexity — computed in teacher-forcing mode over gold transcription tokens only — as the search signal, then reported WER on selected settings. They caution that perplexity is a selection heuristic, not an analytical substitute for WER, and that the matched systems they report show lower perplexity generally coinciding with lower WER.

For ASR, they fine-tuned MMS per target language and decoded 10 hypotheses with beam search, then re-ranked them by combining the acoustic score with an LM score from the speech LLM under MICL conditioning. Text-only baselines (an n-gram LM, a transformer LM trained on target-language text, and Llama-3-8B-Instruct) used the same combined scoring setup. They also compared this against joint decoding with an n-gram LM.

Why This Matters

Impact on research. The paper shifts in-context learning for low-resource speech from text-only, high-resource settings to genuinely unseen languages with multimodal context. It provides the first layer-resolved picture of how a speech LLM splits attention between demonstration audio and demonstration text, and it shows that cross-lingual instruction fine-tuning on 143 unrelated languages can substitute, to a degree, for target-language supervision. The finding that text-only in-context language modelling can match or beat a target-language-trained LM without any target-language training data is a notable data point for languages with no text corpora at all.

Real-world applications:

• Documentary and revitalisation work on endangered languages, where only a few hours of recorded speech and transcriptions exist and building a conventional ASR system is infeasible.
• Field documentation workflows in which a linguist records and transcribes a small sample and wants immediate draft transcriptions to speed up annotation.
• Speech interfaces for communities speaking languages outside the coverage of mainstream commercial assistants.
• Rapid adaptation of existing ASR pipelines to new languages or dialects without collecting large labelled corpora, by re-ranking a fine-tuned acoustic model's output.

Industry relevance. The hypothesis-selection design is modular: it adds a speech LLM on top of an existing production acoustic model without retraining it, and the paper quantifies the cost of doing so (22 GB memory, 3 seconds per item on a single RTX 6000 GPU). That makes it a plausible path for vendors wanting to extend coverage into low-resource languages cheaply, though the paper's own comparison shows joint decoding is stronger where it is computationally affordable. The results also give model developers a concrete signal: broader language coverage during pre-training and fine-tuning, plus larger multilingual audio training scale, yields better in-context learning on languages the model never saw.

Future Directions

1. Bring MICL into joint decoding. The authors note that joint decoding beat hypothesis selection in their setup and suggest that integrating MICL into joint decoding could bring the best performance, while acknowledging it is substantially more expensive because acoustic and LM scores are combined at every decoding timestep.
2. Scale up cross-lingual instruction fine-tuning. Since improvements appeared only up to the ten demonstrations seen during fine-tuning, and the fine-tuning data was limited in size and diversity, the authors hypothesise that scaling both data volume and diversity would improve speech context understanding and MICL effectiveness.
3. Redesign models to attend more to audio. The observed text bias in attention, mirroring findings in vision-based LLMs, suggests a concrete design target: making speech LLMs allocate more attention to the audio modality.
4. Broaden the evaluation. The limitations section calls for testing MICL on additional speech and multimodal tasks beyond ASR, on more than three languages, and on closed-source LLMs such as Gemini or GPT — which the current approach cannot do because hypothesis selection requires access to model parameters.
5. Investigate the anomalous sample-selection failures. The abnormally high perplexity for Khinalug with three samples appeared across settings and is attributed to imperfect sample selection, an open problem for retrieval-based demonstration choice.

Target Audience

Researchers and engineers working on low-resource and endangered-language speech technology, speech LLM adaptation, and in-context learning. It is also useful for practitioners designing multilingual ASR deployment pipelines who need to know whether to fine-tune an acoustic model, prompt a speech LLM, or re-rank one with the other. Readers interested in multimodal interpretability will find the layer-wise attention analysis and the gold-replacement intervention relevant, while those without a background in speech LLM architectures and ASR evaluation should expect a steep learning curve.

Authors’ abstract

Automatic speech recognition (ASR) still covers only a small fraction of the world's languages, mainly due to supervised data scarcity. In-context learning (ICL) with large language models (LLMs) addresses this problem, but prior work largely focuses on high-resource languages covered during training and text-only settings. This paper investigates whether speech LLMs can learn unseen languages with multimodal ICL (MICL), and how this learning can be used to improve ASR. We conduct experiments with two speech LLMs, Phi-4 and Qwen3-Omni, on three diverse endangered languages. Firstly, we find that MICL is effective for unseen languages, leveraging both speech and text modalities. We further show that cross-lingual transfer learning improves MICL efficiency on target languages without training on them. Moreover, we analyze attention patterns to interpret MICL mechanisms, and we observe layer-dependent preferences between audio and text context, with an overall bias towards text. Finally, we show that prompt-based ASR with speech LLMs performs poorly on unseen languages, motivating a simple ASR system that combines a stronger acoustic model with a speech LLM via MICL-based selection of acoustic hypotheses. Results show that MICL consistently improves ASR performance, and that cross-lingual transfer learning matches or outperforms corpus-trained language models without using target-language data. Our code is publicly available.