Phoneme- and Word-Level Metrics Using Self-Supervised Speech Representations for Forced Alignment Evaluation

AI summary

Overview

Research area: Speech processing and natural language processing — specifically forced alignment (mapping speech audio to transcript units like words and phonemes) and the evaluation of alignment quality.

Technical level: Intermediate. The paper builds on self-supervised speech models (MMS, XLSR), mutual information, and dynamic time warping. Readers with basic familiarity with speech representation learning will follow it comfortably; the core ideas are explained without heavy mathematical prerequisite.

Scope: The paper proposes two reference-free, corpus-level metrics — Phoneme-Cluster Mutual Information (PCMI) and Word Acoustic Consistency Score (WACS) — that evaluate forced alignment quality without requiring manually annotated timestamps, and validates them on synthetic perturbations, 85 FLEURS languages, 45 DoReCo languages, and two phonologically complex low-resource languages (Archi and Rutul).

What This Paper Is About

Forced alignment systems are normally judged by comparing their output timestamps against manually annotated gold timestamps. Those annotations are expensive to produce and exist mostly for English, which forces researchers to either evaluate only English or measure their aligners against the output of an existing aligner such as the Montreal Forced Aligner (MFA). The authors address this bottleneck by designing two metrics that score alignment quality using only the speech signal and the aligner's own output, relying on the phonetic and lexical structure that self-supervised speech models already encode.

Key Contributions

1. Two reference-free metrics. The paper introduces PCMI (Phoneme-Cluster Mutual Information), which measures agreement between aligned phoneme labels and clusters induced from SSL speech representations, and WACS (Word Acoustic Consistency Score), which measures whether repeated occurrences of the same word are acoustically more similar to each other than to unrelated words, using dynamic time warping (DTW) over representation sequences.

2. Robustness and adversarial analysis. The authors characterize how both metrics behave under random boundary perturbations and under systematic adversarial perturbations — vowel absorption and silence absorption — of manually annotated Buckeye alignments.

3. Large-scale multilingual validation. The metrics are analyzed across alignments from multiple systems on 85 languages from FLEURS, validated against manually annotated alignments from 45 languages in DoReCo, and tested on two phonologically complex low-resource languages, Archi and Rutul (the full evaluation spans 132 languages).

4. Released resources. Multilingual phoneme recognition models trained as a byproduct (MMS-300M-IPA and Wav2Vec2-IPA for FLEURS, MMS-300M-DORECO for DoReCo) and an open-source Python package of the metrics are released.

Main Findings

• Both metrics degrade under random perturbation. When Gaussian noise is added to word boundaries in the Buckeye corpus (perturbations generated with target scales of 50–2000 ms, producing Average Accumulated Shift values of roughly 45–210 ms after constraint enforcement), PCMI and WACS both fall consistently. The decline is approximately linear at lower perturbation levels and then plateaus.

• Middle transformer layers work best. Layer-wise analysis of MMS and XLSR shows that layers approximately between 5 and 16 give the greatest separation between clean and severely perturbed alignments; layer 15 was selected for all subsequent experiments.

• PCMI is stable across cluster counts. Varying the number of K-means clusters from 25 to 100 changed the clean-versus-perturbed separation only from about 0.26 to 0.25 for MMS and from 0.24 to 0.22 for XLSR, so n_cluster = 50 was used.

• WACS is vulnerable to silence absorption. Because DTW is relatively insensitive to leading or trailing silence, systematically merging preceding silence into neighboring words produces weaker degradation than random perturbation at comparable shift levels, making silence absorption a prominent adversarial failure mode for WACS. The metrics cannot be artificially inflated, however: PCMI approaches 0 as phoneme-label entropy approaches 0, and WACS approaches 0 if the representation model collapses all word occurrences to identical embeddings.

• MFA fails bimodally and often. On FLEURS, PCMI and WACS distributions for MFA are strongly bimodal, separating failed from successful alignments. MFA training or decoding failed for approximately 12k out of 65k utterances (~19%), whereas the IPA-based CTC aligners produced only 11 word-level failures overall. CTC-based aligners gave substantially more stable cross-language distributions despite slightly lower peak PCMI values than the successful MFA cluster (for example, MFA on MMS: PCMI 0.25 ± 0.11, WACS 0.14 ± 0.08; MMS-300m-IPA: PCMI 0.24 ± 0.03, WACS 0.19 ± 0.02; the evenly spaced baselines: PCMI 0.09 ± 0.02, WACS 0.02 ± 0.01).

• Gold alignments set a useful reference point. On DoReCo, manually annotated gold alignments score approximately 0.33 PCMI and 0.13 WACS.

• Strong negative correlation with AAS. Across 45 DoReCo languages and four alignment conditions (180 alignment sets), Pearson correlations with Average Accumulated Shift were −0.7769 (MMS PCMI vs AAS), −0.6257 (MMS WACS vs AAS), −0.7761 (XLSR PCMI vs AAS) and −0.6686 (XLSR WACS vs AAS), all with p < 0.001.

• CER predicts alignment quality better than WER. Character error rate shows consistent moderate negative correlations with the alignment metrics (roughly −0.5 with WACS and −0.3 to −0.5 with PCMI), while word error rate shows a similar but weaker trend. This supports the view that the metrics track alignment quality rather than merely mirroring phoneme recognition performance.

• Silence handling explains metric–AAS divergence. For Archi, removing trailing silence eliminated roughly 44 seconds of excess silence and cut AAS from 88.5 ms to 46.3 ms, while PCMI and WACS changed only slightly. Only about 5 seconds were removed for Rutul, producing smaller AAS changes. The authors attribute the difference to the faster speech rate in the Rutul spontaneous narratives versus the Archi read-out recordings.

• AAS has its own weaknesses. The authors argue that AAS is highly sensitive to silence absorption and to speech-rate differences across languages, which complicates cross-lingual comparison, whereas PCMI and WACS are comparatively rate-invariant by design.

• XLSR generalizes despite limited pretraining coverage. XLSR (pretrained on 53 languages) remained effective even for FLEURS languages and most DoReCo languages absent from its training data, with MMS providing slightly stronger separation overall.

• Cluster and label entropy remain high. Normalized cluster entropy stayed around 0.95–0.99 and normalized phoneme-label entropy around 0.82–0.95 across languages and embedding models. MFA showed substantially larger variance in these statistics and in the metric scores, consistent with its bimodal failure behavior.

Methodology in Plain English

The core idea. Rather than checking whether predicted timestamps match gold timestamps, the authors check whether the content inside each predicted interval is internally consistent with what speech models know about phonetics and words.

PCMI. Frames of audio are converted into vectors using a self-supervised speech model. Each frame is labelled with whatever phoneme the aligner says occupies that moment. The vectors are then grouped into clusters with K-means. If the aligner's phoneme labels line up with the natural clusters in the representation space, the normalized mutual information between labels and clusters is high. If the aligner's boundaries drift, phonemes get contaminated by their neighbours, the clusters and labels disagree, and PCMI drops.

WACS. Every aligned word occurrence becomes a short sequence of frame vectors. The authors compare occurrences of the same word (positive pairs) with occurrences of different words (negative pairs) using dynamic time warping with cosine similarity between frames — DTW handles the fact that the same word can be spoken at different speeds. WACS is the average positive-pair similarity minus the average negative-pair similarity. Good boundaries mean a word's interval contains that word and nothing else, so same-word pairs look alike and the score rises.

Evaluation protocol. The metrics were first stress-tested on the Buckeye corpus (40 speakers, segmented into 2,935 utterances of 3–20 s, totalling about 7.5 hours), where boundaries were deliberately corrupted with Gaussian noise and with targeted vowel- and silence-absorption edits. Layer sensitivity was probed across MMS and XLSR transformer blocks, and MFCCs (25 ms window, 10 ms hop, 20 coefficients) served as a classical baseline. For the multilingual study, FLEURS transcripts were converted to IPA (numerals normalized with num2words; grapheme-to-phoneme conversion via Epitran and XPF), and multilingual IPA phoneme recognizers were fine-tuned with language-specific adapters, with alignments produced by CTC segmentation. Alignments were compared against per-language MFA models, an evenly spaced phoneme baseline, and Qwen3-ForcedAligner-0.6B on its 10 supported languages. DoReCo (X-SAMPA transcriptions, gold word and phone timestamps) allowed direct correlation against AAS. Subsampling keeps the metrics cheap: PCMI uses up to 50 utterances and at most 10,000 frames with MiniBatch K-means (batch size 1000); WACS uses up to 200 utterances and up to 200 unique word forms appearing at least three times, with at most 10 positive and 5 negative pairs per form. Results were averaged over 5 independent samples. The full 85-language FLEURS benchmark takes roughly 40 minutes per embedding model, about 30 seconds per language.

Why This Matters

Impact on research. Timestamp-based evaluation has kept forced alignment research concentrated on English and on a handful of well-annotated corpora. Metrics that need no gold timestamps remove that dependency, allow aligners to be compared across dozens of languages at once, and give low-resource projects a way to screen alignments before investing in manual annotation. The paper also documents a practical pitfall in current practice: because MFA failed on roughly 19% of FLEURS utterances, using MFA output as a stand-in reference is unreliable in low-resource scenarios.

Real-world applications:

• Subtitle generation — automatically flagging badly aligned segments before subtitles are published.
• Speech corpus segmentation — validating alignments over endangered or newly recorded language corpora where no gold timestamps exist.
• Acoustic-phonetic analysis — giving phoneticians confidence that measured segment boundaries reflect real phonetic structure rather than aligner drift.
• Speech synthesis and retrieval — ensuring training data and search indices are segmented so that units contain the right audio, including correct silence handling.

Industry relevance. Any pipeline that scales speech data processing across many languages — speech recognition training, voice assistants, media localization — needs a cheap quality gate on alignment output. Corpus-level metrics computable in about 30 seconds per language provide exactly that gate, and the released Python package and phoneme recognition models lower the barrier to adopting it.

Future Directions

• Reduce silence sensitivity in WACS. Silence absorption remains WACS's clearest adversarial failure mode, and the authors show it is only partly mitigated by post-processing that strips silence before segment extraction.
• Capture fine-grained phonological mergers. Expert analysis of Archi and Rutul found systematic errors such as consonant releases merging into following onsets; the perturbation results suggest the metrics are less sensitive to these, and residual AAS differences remain unexplained by silence removal alone.
• Extend G2P coverage and ablate phonemization. The experiments are limited to datasets with transcriptions and phoneme conversion pipelines, and the authors explicitly did not ablate the effect of different grapheme-to-phoneme systems because alternatives rarely exist for the evaluated languages.
• Add utterance-level or local diagnostics. The metrics are corpus-level only and cannot point at individual utterances or specific boundary errors, which limits their use as a debugging tool.

Target Audience

Speech and language technology researchers working on forced alignment, multilingual speech processing, or low-resource and endangered language documentation; phoneticians who need to assess alignments at scale; and engineers building speech pipelines that must process many languages without access to gold timestamp annotations. Readers interested in self-supervised speech representations and in evaluating whether those representations encode phonetic and lexical structure will also find the perturbation and layer-wise analyses directly useful.

Authors’ abstract

Forced alignment evaluation typically requires manually annotated timestamps, limiting large-scale and multilingual analysis. We introduce two corpus-level metrics based on self-supervised (SSL) speech representations for reference-free forced alignment evaluation: Phoneme-Cluster Mutual Information (PCMI) and Word Acoustic Consistency Score (WACS). PCMI measures agreement between aligned phoneme labels and clusters induced from SSL-speech representations, while WACS measures consistency of repeated word realizations using dynamic time warping similarity between word representation sequences. Using both random and systematic perturbations, we show that PCMI and WACS degrade consistently under alignment perturbations. We further analyze the metrics across multiple alignment systems on 85 languages from FLEURS, validate them against manually annotated alignments from 45 languages in DoReCo, and evaluate them on two phonologically complex low-resource languages. The metrics effectively separate high- and low-quality alignments and correlate strongly with timestamp-based alignment quality measures. Our results demonstrate that SSL-speech representations enable scalable, reference-free forced alignment evaluation. The metrics are available as an open-source Python package at https://github.com/mahesh-ak/forced-aligner-metrics.