PRiSM: Benchmarking Phone Realization in Speech Models

AI summary

Overview

Research area: Speech processing and natural language processing, specifically phone recognition (the transcription of speech into phonetic units such as IPA symbols) and the evaluation of speech models.

Technical level: Intermediate. The paper assumes familiarity with terms like CTC, encoder-decoder architectures, self-supervised speech models, and downstream probing, though it explains its own metrics clearly.

One-sentence scope: The paper introduces PRiSM, the first open-source benchmark for phone recognition systems that combines intrinsic transcription evaluation with extrinsic downstream-utility evaluation across clinical, educational (L2), and multilingual settings.

What This Paper Is About

Phone recognition (PR) systems convert speech into phonetic units that capture how sounds are physically realized, independent of language-specific phonological constraints. Existing evaluations of these systems only measure surface-level transcription accuracy, and they are difficult to compare across studies because models differ in language coverage, phone inventories, and metrics. PRiSM addresses this by standardizing transcription-based evaluation and adding downstream probes that test whether a system's phonetic output and internal representations are actually useful in real applications.

Key Contributions

1. The first open-source benchmark for phone recognition systems. PRiSM (Phone Realization in Speech Models) releases code, evaluation recipes, and datasets where licensing permits, covering both intrinsic (transcription) and extrinsic (downstream) evaluation.

2. A standardized intrinsic metric, Phonetic Feature Error Rate (PFER). Unlike Phone Error Rate, which treats each phone as a token, PFER computes edit distance over articulatory features such as roundness or voicing, so it measures the percentage of incorrect features rather than incorrect symbols.

3. A dual-probe extrinsic evaluation framework. A transcript probe (TP) feeds predicted phonetic transcriptions into a text-based bi-GRU, while a representation probe (RP) uses final hidden-layer representations with temporal attention pooling and a Multi-Layer Perceptron, following the setup in Turian et al. (2022).

4. A broad empirical comparison across nine PR systems spanning Wav2Vec2 fine-tunes, from-scratch encoder-CTC and encoder-decoder models, and two Large Audio Language Models (Gemini 2.5 Flash and Qwen3-Omni-Instruct), plus WavLM and Whisper as representation-probing baselines.

Main Findings

• Diverse language exposure during training is key to PR performance. For MultiIPA, English appears during pretraining but not fine-tuning, showing the importance of language coverage in PR data. The authors conclude that seen-language variation benefits from outputs grounded in familiar patterns, while unseen languages benefit from multilingual training and learned phonological patterns.

• Encoder-CTC models are the most stable, but encoder-only design alone does not guarantee stability. ZIPAs are encoder-only Zipformer models trained with consistency-regularized CTC (CR-CTC), and their behavior in the masking experiment suggests structural stability comes from the loss as much as the architecture.

• Specialized PR models still outperform Large Audio Language Models. On unseen languages, the average PFER is 18.7 for POWSM and 19.0 for ZIPA-CTC-NS, versus 53.8 for Gemini 2.5 Flash and 105.4 for Qwen3-Omni-Instruct. On DoReCo, 86% of Gemini 2.5 Flash utterances and 91% of Qwen3-Omni-Instruct utterances remain below PFER 50, while only 1% and 6% exceed 100, meaning the high averages are driven by a small number of unstable generations involving long insertions or repetitions.

• A text-based G2P model achieves a PFER of 10.2 on PR-saa, beating Gemini 2.5 Flash, even though the G2P model only models canonical pronunciations.

• Whisper excels at representation probing. Its aggregate RP score of 68.5 is the highest reported, ahead of ZIPA-CTC at 62.9 and ZIPA-CTC-NS at 62.7, suggesting large-scale ASR pretraining produces representations that retain phonetic information.

• A trade-off emerges between the two probes among specialized PR models. Wav2Vec2Phs achieve strong TP results on L2 speech but limited gains on RP, whereas ZIPAs underperform on TP yet excel on RP. Pathological speech benefits more from RP, L2 speech falls in the middle, and multilingual tasks tend to favor TP.

• Language diversity can matter as much as data volume. IPAPack++ is 17k hours, much smaller than the roughly 160k hours used for Wav2Vec2Phs, but the larger number of languages in supervised training (88 versus about 40) leads to better recall for ZIPAs.

• Phone inventory induction for unseen languages favors encoder-only architectures. POWSM-CTC is the strongest model, and the gap between POWSM-CTC and POWSM, which differ only in architecture, points to the encoder-only design. Most models show high recall (above 70) and low precision (below 50), meaning predictions are high-entropy.

• Transcript probes beat representation probes on Hindi dialect geolocation, with an average error of 146 km versus 253 km, against data spread over 1478 km East-West and 1703 km North-South. The authors attribute this partly to the RNN in the transcript probe preserving phone order.

• Broad and diverse language coverage improves precision. ZIPA-CTC-NS, trained further on large-scale pseudo-labeled data from ZIPA-CTC, is more precise than ZIPA-CTC on unseen languages, suggesting noisy pseudo-labels can improve precision for new languages.

• LALMs lack fine-grained phonetic perception. On GEO-v, Qwen3-Omni-Instruct predicts New Delhi for nearly all inputs, while Gemini 2.5 Flash reaches only 6.5% hit@1 accuracy, with roughly 65% of predictions concentrated in 3-4 coordinate clusters near New Delhi (28.6°N, 77.2°E). On L1-eda, LALMs show bias toward the Romance accent cluster: 25.8% of Slavic/Balkan and 28.5% of South Asian accents are misclassified as Romance. Enabling thinking mode drops the F1-score from 32.7% to 24.9%, with 87% of erroneous Romance predictions mentioning "Spanish/Italian/Portuguese" and 65% citing "syllable-timed rhythm."

• Few-shot prompting helps pathological speech tasks but shows mixed effects elsewhere, with no consistent gains as the number of examples increases.

Methodology in Plain English

The researchers built a benchmark with two arms.

For the intrinsic arm, they gathered speech datasets that fall into two groups. "Variation of seen languages" covers regional and non-native English speech: TIMIT (six regional varieties of American English, 6,300 test utterances), L2-ARCTIC Perceived (3,599 manually annotated utterances from non-native speakers), and the Speech Accent Archive (3,019 utterances from non-native English speakers across 391 L1 languages). "Unseen languages" tests language-agnostic phonetic knowledge: DoReCo (18,734 utterances across 45 languages), VoxAngeles (5,445 utterances across 95 languages), and Tusom2021 (2,255 utterances of the low-data Tangkhulic language Tusom, with tones removed since no model supports them). They score all of these with PFER, and note that for closed LALMs strict verification of the unseen-language condition is impossible because full training corpora are not public.

For the extrinsic arm, they use two probes on nine downstream tasks. The transcript probe takes predicted transcriptions as text into a bi-GRU. The representation probe takes the model's final hidden-layer representations and applies temporal attention pooling plus an MLP. Because transcripts and hidden representations differ so much in form, the authors compare models mostly within each probe type rather than across them. The downstream tasks span pathological speech (dysarthria intelligibility prediction on EasyCall and UASpeech, child speech disorder detection on UltraSuite), L2 speech (L1 classification on EdAcc and on a combination of L2-ARCTIC and CMU ARCTIC, proficiency assessment on Speechocean762), and multilingual speech (language ID on FLEURS-24, geolocation on Vaani Hindi dialects, phone inventory induction on DoReCo). Results are reported as mean plus or minus standard deviation across 5 random seeds where applicable.

They also ran targeted analyses: a phone-masking experiment on TIMIT where a percentage of phones is replaced with silence and PFER is computed against the remaining phones, with a flat curve indicating pure acoustic reliance; a zero-shot phone inventory induction study on DoReCo using PanPhon-based phone tokenization followed by set union; integrated-gradient attribution maps on the transcript probe for geolocation; and a bias analysis of LALM geographic predictions.

Why This Matters

Impact on research. Before PRiSM, PR evaluations were not standardized across studies, and the link between transcription accuracy and downstream performance was assumed rather than empirically tested. PRiSM shows that the two do not always track each other: intrinsic evaluation alone may not capture phonetic capabilities, and the relative ranking of models on the transcript probe versus the representation probe is task-dependent. This gives the field a common, reproducible basis for comparison and a released toolkit, recipes, and datasets.

Real-world applications.

• Clinical speech assessment. Faithful phonetic transcriptions can help a clinician understand the nature of a speech sound disorder and design interventions, and PRiSM measures dysarthria intelligibility prediction and child speech disorder detection directly.
• Language learning. Phonetic cues enable pronunciation feedback and accent classification; PRiSM evaluates L2 proficiency scoring on Speechocean762 and L1-influence classification.
• Multilingual and dialect-aware speech technology. Phone inventory induction supports building transcription systems for new languages, and the geolocation task shows that transcript-based probes can distinguish Hindi dialects at an average error of 146 km.
• Speech model selection. Because PRiSM evaluates transcription and downstream utility together, practitioners can choose models based on the task at hand rather than on a single error rate.

Industry relevance. LALMs are increasingly used for general speech tasks despite limited evaluation of their phonetic abilities, and PRiSM provides evidence that they lag specialized PR models on phone recognition and related tasks. For teams deciding between a specialist encoder-CTC system and a general audio-language model, these results are directly actionable, particularly in non-English settings where the performance gap is largest.

Future Directions

• Improve benchmark coverage. The authors state that coverage of languages, dialects, accents, and speaking styles remains incomplete and may reflect biases in the underlying corpora, so expanding curated datasets is a natural next step.
• Address the subjectivity of phonetic ground truth. Transcription depends on annotation guidelines, annotator judgments, and phone inventory choice, and the IPA-based interface may normalize away language-specific or gradient phenomena.
• Refine representation probing. Phonetic information may be distributed across different layers, and RP performance can depend on the chosen fusion or pooling strategy; the authors explore layer-aggregation variants and note layer aggregation could be further improved.
• Investigate LALM phonetic perception. The analysis of bias toward higher-resourced dialects and the effect of thinking mode on accent classification raise open questions about how to make LALMs sensitive to sociophonetic variation.
• Extend the interpretability work. The attribution analysis of doubled phones in the Bangru dialect of Haryanvi is explicitly left for future work, and decoding hyperparameters and prompts were kept at defaults, so task-specific tuning remains unexplored.

Target Audience

This paper is most useful to researchers and engineers building or evaluating speech recognition and phonetic transcription systems, particularly those working on multilingual, low-resource, or non-standard speech. It also serves clinical and educational technology practitioners who need to know whether a model's phonetic output is trustworthy, and linguists interested in computational approaches to phone inventories and dialect geolocation. Readers comparing large audio-language models against specialized speech models will find the intrinsic-versus-extrinsic distinction the paper's most distinctive methodological contribution.

Authors’ abstract

Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis. Despite prolonged efforts in developing PR systems, current evaluations only measure surface-level transcription accuracy. We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception through intrinsic and extrinsic evaluation of PR systems. PRiSM standardizes transcription-based evaluation and assesses downstream utility in clinical, educational, and multilingual settings with transcription and representation probes. We find that diverse language exposure during training is key to PR performance, encoder-CTC models are the most stable, and specialized PR models still outperform Large Audio Language Models. PRiSM releases code, recipes, and datasets to move the field toward multilingual speech models with robust phonetic ability: https://github.com/changelinglab/prism.