POWSM: A Phonetic Open Whisper-Style Speech Foundation Model

AI summary

Overview

Research area: Speech processing and phonetics within natural language processing — specifically multilingual phone recognition, automatic speech recognition, and grapheme-phoneme conversion.

Technical level: Intermediate. The paper assumes familiarity with encoder-decoder speech architectures, CTC loss, and phonetic transcription conventions (IPA, PanPhon), but its core argument — that four phone-related tasks can share one model — is accessible to a general NLP audience.

Scope: The paper presents POWSM, a fully open-source 350M-parameter attention-based encoder-decoder speech foundation model trained from scratch on roughly 17,000 hours of multilingual speech to jointly perform phone recognition, ASR, audio-guided grapheme-to-phoneme conversion, and audio-guided phoneme-to-grapheme conversion.

What This Paper Is About

Speech tasks that revolve around phones — automatic speech recognition (ASR), phone recognition (PR), grapheme-to-phoneme conversion (G2P), and phoneme-to-grapheme conversion (P2G) — are conceptually related but have traditionally been built as separate systems with task-specific architectures and datasets. POWSM asks whether a single attention-based encoder-decoder model, trained with task and language tokens over shared data, can do all four at once while matching or beating specialized phone recognition models of comparable size. The goal is a unified phonetic foundation model that converts seamlessly between audio, text (graphemes), and phones, with all data, code, and checkpoints released openly.

Key Contributions

1. A unified phonetic foundation model. POWSM is described as the first framework capable of jointly performing PR, ASR, audio-guided G2P, and audio-guided P2G within one architecture, enabling seamless conversion between speech, graphemes, and phones.

2. State-of-the-art phone recognition with multitask support. POWSM achieves the lowest average Phonetic Feature Error Rate (PFER) on the in-domain IPAPack++ test sets among compared models, while additionally supporting G2P, P2G, and ASR.

3. Systematic analysis of what makes the model work. The paper examines the effect of encoder target granularity (Unicode code points vs. PanPhon, with and without suprasegmentals), CTC loss weighting, the balance between speech and text prompts in G2P, and the role of language tokens in shaping phonotactic output.

4. Full open release. Data preparation scripts, evaluation scripts, model checkpoint, and code are all released on GitHub (espnet) and Hugging Face (espnet/powsm) to foster open science.

Main Findings

• Best in-domain phone recognition average. On IPAPack++, POWSM (350M) reaches an average PFER of 2.62 across eng, deu, nld, fra, ita, spa, por, pol, tam, kaz, and cmn, beating ZIPA-CR-Large (2.99) and ZIPA-CR-NS-Large (2.70). It is not best on every language — for example, on eng it reports 2.85 versus 1.40 for ZIPA-CR-NS-Large — but wins on the average. The paper attributes this to the decoder's language modeling capability and notes that its English data cleaning may have hurt Germanic-language PFER through a train/test mismatch.

• Strong generalization to unseen languages. On DoReCo (45-language subset), VoxAngeles, and Tusom2021, POWSM averages 18.71 PFER, better than ZIPA-CR-Large (19.54) and ZIPA-CR-NS-Large (19.01), and it does so while handling three additional tasks. It trails Wav2Vec2Phoneme (21.02 average on unseen languages) is not the case here — actually Wav2Vec2Phoneme scores 21.02, worse than POWSM's 18.71 on unseen languages, but Wav2Vec2Phoneme is better on socio-phonetic variation (average 12.89 versus POWSM's 14.40 on Buckeye, DRC-SE, L2-ARCTIC, EpaDB, and SO762). The authors attribute this to Wav2Vec2Phoneme's self-supervised pre-training on over 60k hours of speech from wav2vec 2.0.

• Competitive low-resource ASR from phone prompting. On eight low-resource FLEURS languages (afr, orm, aze, pan, tgk, mkd, bos, slv), POWSM 0.35B with PR-predicted phones as text prompts (PR-P2G) is often comparable to or better than web-scale models of similar size, such as OWLS 0.5B. For example, on mkd PR-P2G reports 48.6 WER versus 54.2 for OWLS 0.5B.

• Speech-guided G2P preserves variation; text prompts normalize it. On Buckeye, G2P with only speech scores 12.71 PFER, close to PR (12.63); adding both speech and text prompts degrades to 16.38 and produces more standardized pronunciations; using only the text prompt drops sharply to 23.44.

• P2G outperforms ASR on low-resource languages. Using gold phone labels for P2G yields much lower WER than ASR on the same languages (for example, 31.8 versus 62.8 on tgk), though the paper notes the comparison is not entirely fair because P2G receives gold phones.

• The language token shifts phonotactics. POWSM reaches 92.3% language identification accuracy on seen languages in FLEURS. On unseen data, forcing the language token to English performs worse than the automatically detected language, but marking the language as unknown performs best of all (17.11 on VoxAngeles and 21.96 on Tusom2021 versus 19.91 and 24.21 for <eng>).

• Encoder design choices matter. In 1k-hour pilot experiments, PanPhon tokenization without suprasegmentals produced the earliest drop in validation PER, suggesting that aligning CTC targets with decoder units and dropping length/break marks accelerates convergence. Higher CTC decoding weights improved out-of-domain PFER but hurt in-domain PFER, echoing a trade-off between in-domain fit and generalization.

• No clear evidence ASR helps PR. Across scales of data and model size, the paper finds no consistent trend showing that multitask learning improves PR, and no clear evidence that ASR benefits PR under the current setup.

Methodology in Plain English

One dataset, four formats. The team starts from IPAPack++, an open corpus of roughly 17,000 hours of multilingual speech with paired orthographic and phonetic transcriptions. Each training utterance is reused once per task, reformatted with a text prompt, a language token, a task token, and a target output. For PR and ASR the text prompt is left blank (<na>). For G2P the graphemes are supplied as the prompt and phones are the output; for P2G the phones are the prompt and graphemes are the output.

Cleaning the phones. Utterances longer than 300 phones are filtered out. IPA sequences are normalized to Unicode NFD, and English G2P output gets rule-based corrections for voice-onset-time issues, lateral velarization, and vowel nasalization. IPA tokens are split with diacritics and modifiers attached using a greedy trie search over PanPhon entries and wrapped in slashes, so IPA symbols are never confused with graphemes.

Architecture and training. POWSM follows the OWSM v3.1 design: an E-Branchformer encoder and a Transformer decoder, trained from scratch with ESPnet using a hybrid CTC/attention loss (Equation 1) with the CTC weight set to 0.3. The encoder runs at a 40 ms stride; inputs are 16 kHz and padded to 20 seconds; the global batch size is 256; the vocabulary is 40k tokens including about 6k phone tokens plus language, timestamp, and BPE orthography tokens. The model has roughly 350M parameters with 9 layers each in the encoder and decoder, and was trained for about 200 GPU hours on H100s. CTC targets use a simplified phone sequence stripped of break and length diacritics to speed convergence.

Evaluation. The headline metric is Phonetic Feature Error Rate (PFER), an edit distance computed over PanPhon articulatory features rather than exact phone matches; each feature contributes 1/24 of a distance unit, and insertions or deletions cost 1 unit. PFER is normalized by the number of reference phones, measuring the proportion of feature errors per phone. Decoding uses CTC weight 0.3 and beam size 3 unless otherwise stated, with greedy decoding (CTC 0.0, beam 1) for the fair ASR comparison. Evaluation covers in-domain IPAPack++ sets, unseen-language sets (DoReCo, VoxAngeles, Tusom2021), English variety and L2 sets (Buckeye, DoReCo South-England, L2-ARCTIC, EpaDB, SpeechOcean762), and FLEURS for ASR and P2G.

Why This Matters

Impact on research. POWSM argues that phone-related tasks should not be studied in isolation. By showing that one 350M-parameter model can handle PR, ASR, G2P, and P2G while matching specialized phone recognizers, it provides both a competitive open baseline and a testbed for questions about cross-lingual phonetic representation. The released scripts and checkpoints make those comparisons reproducible.

Real-world applications:

• Atypical speech analysis, including L2 (second-language) speech and pathological speech, where phone-level detail matters more than word-level transcriptions.
• Endangered and low-resource language documentation, where the model must handle languages it never saw in training, such as Tusom2021.
• Code-switched text-to-speech, where reliable grapheme-to-phone conversion across languages is a prerequisite.
• Cross-lingual transfer in speech-to-text, where phone-level supervision can bootstrap ASR for languages with little transcribed audio.

Industry relevance. The model is released openly on GitHub and Hugging Face with permissive intent, giving commercial and academic teams a foundation they can adapt without depending on a closed corpus. Its low-resource ASR results suggest phone-level pre-training as a practical route to extending speech products to languages and accents that large web-scale systems serve poorly. The paper also flags the risks: normalizing socio-phonetic variation can amount to prescriptivism and threaten linguistic diversity, and accurate modeling of that variation can enable demographic inference.

Future Directions

• Using the decoder as a phonotactic language model. The authors propose that linguists could use POWSM's decoder to investigate hypotheses about phonetic universals and phonotactics.

• Adapting to socio-phonetic variation. Candidate approaches include unsupervised test-time adaptation, in-context learning, and mechanistic interpretability.

• Early exiting to preserve phonetic detail. Since prior work found that earlier Whisper encoder layers retain more phonetic detail, the authors suggest early exiting could counter the decoder's tendency to normalize variation.

• Addressing architectural and data limitations. The paper names three open problems: the model is neither strictly phonemic nor phonetic and lacks allophone-level data; it still favors high-resource languages; and the AED design makes inference slower than encoder-only models and does not easily support tone modeling, limiting use for tonal languages.

Target Audience

Speech and phonetics researchers who need a multilingual phone recognizer or a controlled setting for studying phonetic representation; NLP engineers building low-resource or multilingual speech systems who want an open, adaptable checkpoint; and linguists interested in using model outputs to probe phonotactics, phonetic universals, and cross-linguistic variation. Readers unfamiliar with CTC, IPA transcription, or encoder-decoder speech models will need background reading, but the task framing and analysis sections are written to be followed without deep architectural knowledge.

Authors’ abstract

Recent advances in spoken language processing have led to substantial progress in phonetic tasks such as automatic speech recognition (ASR), phone recognition (PR), grapheme-to-phoneme conversion (G2P), and phoneme-to-grapheme conversion (P2G). Despite their conceptual similarity, these tasks have largely been studied in isolation, each relying on task-specific architectures and datasets. In this paper, we introduce POWSM (Phonetic Open Whisper-style Speech Model), the first unified framework capable of jointly performing multiple phone-related tasks. POWSM enables seamless conversion between audio, text (graphemes), and phones, opening up new possibilities for universal and low-resource speech processing. Our model outperforms or matches specialized PR models of similar size (Wav2Vec2Phoneme and ZIPA) while jointly supporting G2P, P2G, and ASR. Our training data, code and models are released to foster open science.