Multi-granularity Interactive Attention Framework for Residual Hierarchical Pronunciation Assessment

AI summary

Overview

• Research area: Automatic pronunciation assessment (APA) for computer-assisted pronunciation training (CAPT), specifically multi-aspect, multi-granularity scoring of read-aloud speech using deep neural networks.
• Technical level: Advanced. The paper assumes familiarity with Transformer encoders/decoders, self-attention and cross-attention, GOP (Goodness of Pronunciation) features, and hierarchical sequence modeling.
• Scope: The paper proposes HIA, a residual hierarchical interactive framework that adds bidirectional attention-based interaction across phoneme, word, and utterance granularities, and evaluates it on the speechocean762 corpus.

What This Paper Is About

Pronunciation assessment systems typically score speech at a single level (phoneme, word, or utterance) or pass information in one direction from phonemes up to words and then to utterances. The authors argue that this one-way flow misses the fact that a word's stress can change depending on the utterance it appears in, so information should also flow downward. The goal of the paper is to build a single model that lets phoneme, word, and utterance representations interact in both directions while also preventing the initial encoded acoustic features from being forgotten as the hierarchy deepens.

Key Contributions

1. The authors identify that prior methods perform poorly on word stress, since the same English word can be stressed differently across utterances, and introduce the HIA framework to address this limitation.
2. They design an Interactive Attention Module that enables bidirectional interaction across phoneme, word, and utterance levels using a single self-attention operation over concatenated multi-granularity queries, followed by cross-attention back into the acoustic feature space.
3. They propose a residual hierarchical structure that adds the Transformer encoder's acoustic embeddings to each granularity's interactive attention head (and passes scores between adjacent granularities), mitigating forgetting of the initially encoded features.
4. They use 1-D convolutional layers to strengthen extraction of local contextual cues at each granularity, and report extensive experiments and analyses on the speechocean762 dataset showing state-of-the-art results on all metrics.

Main Findings

• Bidirectional interaction helps all granularities. In the ablation over interactive attention heads (Table 2), removing all granularity heads gives phoneme PCC 0.626, word stress 0.335, word total 0.605, and utterance total 0.748. Adding the corresponding head at each level improves its own metrics, and using heads at all three levels (the full HIA model) gives the best results: phoneme 0.657, stress 0.436, word total 0.628, utterance total 0.764.
• Word-level attention heads drive the word stress gain. The paper singles out word stress as the metric that improves most from the interactive attention design, which the authors present as validation of bidirectional modeling.
• Residual connections and hierarchy both matter. Removing the residual structure (-Res) lowers phoneme PCC to 0.647, stress to 0.382, word total to 0.603, and utterance total to 0.748. Removing score passing between adjacent granularities (-Hi) lowers them to 0.645, 0.374, 0.593, and 0.753, respectively (Table 3).
• One convolutional layer is optimal. Compared with no convolutional layers (phoneme 0.638, stress 0.415, word 0.601, utterance 0.754), one layer gives the best scores (0.657, 0.436, 0.628, 0.764). Two layers (0.646, 0.427, 0.618, 0.759) and three layers (0.645, 0.421, 0.617, 0.755) decline; the authors attribute this to the dataset not being large enough to optimize the extra parameters.
• Modest model size is best. An embedding size of 48 outperforms 24 (0.657/0.436/0.628/0.764 versus 0.649/0.420/0.613/0.752) and 96 (0.654/0.432/0.611/0.762). One attention head outperforms 2 and 4 heads, again attributed to limited data and harder optimization.
• Benchmark comparison on speechocean762. HIA reaches phoneme MSE 0.076 and PCC 0.657, word accuracy 0.613, word stress 0.436, word total 0.628, utterance accuracy 0.743, completeness 0.354, fluency 0.778, prosodic 0.784, and utterance total 0.764, ahead of GOPT (0.085, 0.612, 0.533, 0.291, 0.549, 0.714, 0.155, 0.753, 0.760, 0.742), HiPAMA (0.084, 0.616, 0.575, 0.320, 0.591, 0.730, 0.276, 0.749, 0.751, 0.754), Gradformer (0.079, 0.646, 0.598, 0.334, 0.614, 0.732, 0.318, 0.769, 0.767, 0.756), and LSTM (0.089, 0.591, 0.514, 0.294, 0.531, 0.720, 0.076, 0.045, 0.747, 0.741).
• Comparison with human expert ratings. HIA outperforms the reported human evaluation results on all metrics except utterance-level completeness. Human reference scores are phoneme PCC 0.555, word accuracy 0.589, word stress 0.212, word total 0.602, utterance accuracy 0.618, completeness 0.658, fluency 0.665, prosodic 0.651, and utterance total 0.675. The authors attribute the completeness gap to distributional bias: 4975 out of 5000 sentences in the dataset have completeness scores of 10.
• Single-granularity methods. The paper reports that single-granularity baselines include RF (MSE 0.130, PCC 0.440), SVR (0.160, 0.450), UOR (0.120, 0.520), Mixup-pretrain (0.610), Deep feature (0.720), Wav2vec2-based (0.725), and LAS (0.766). The authors state HIA holds an advantage over these single-granularity methods on all metrics except the total score at the utterance level; LAS's reported utterance total of 0.766 is the higher number on that single metric.
• Granularities are correlated in the data. A correlation matrix (Figure 4) shows relatively high correlations among phoneme accuracy, word mean scores, and utterance-level scores, which the authors use to justify bidirectional and hierarchical modeling.

Methodology in Plain English

The model takes GOP features extracted with a Librispeech acoustic model that has 42 pure phones, producing an 84-dimensional vector per phone (42 log phone posterior values plus 42 log posterior ratio values), consistent with the GOP formulation of Hu et al. (2015). These are added to projected canonical phoneme embeddings and a trainable positional embedding, then encoded by a Transformer encoder to produce acoustic embeddings.

For each granularity, the model projects a query vector. The three queries (phoneme, word, utterance) are concatenated and passed through a single self-attention operation, so every level can see every other level, giving bottom-up and top-down exchange at once. The resulting self-attention output then acts as a query in a cross-attention operation over the acoustic embeddings, mapping the interaction features back into the acoustic space, and a feed-forward network produces per-granularity interactive attention heads.

Scoring is then carried out hierarchically. At the phoneme level, the encoder output is added to the phoneme attention head and passed through a convolutional layer, with each phoneme having its own 48x1 linear regression head with layer normalization to output an accuracy score (0-2 scale). At the word level, the encoder output, the phoneme scores, and the word attention head are summed, processed by an aspect attention mechanism, and passed through convolution and regression heads to produce word accuracy, stress, and total scores. At the utterance level, learnable aspect queries are used in a Transformer decoder over the sum of the encoder output, the word scores, and the utterance attention head, followed by convolution and regression heads.

Training uses MSE loss, with the total loss being the sum over granularity levels of the average loss over that level's aspects. The optimizer is Adam with an initial learning rate of 1e-3, halved every 5 epochs after epoch 20, up to 100 epochs; the checkpoint with the lowest phoneme-level MSE is kept. Transformer encoder and decoder each have 3 layers with embedding dimension 48, the interactive attention query dimension is 48, self-attention, cross-attention, and Transformer heads are all set to 1, dropout is 0.1, and convolutional kernel size is 5 with stride 1. Every experiment is run five times with different random seeds, and means and standard deviations are reported.

Why This Matters

• Research impact: The paper reframes hierarchical pronunciation assessment as a bidirectional interaction problem rather than a one-way pipeline, and shows that the largest gain lands on word stress, a metric where earlier multi-granularity systems (GOPT 0.291, HiPAMA 0.320, Gradformer 0.334) lagged well behind their other scores. It also offers a simple residual remedy for feature forgetting in deep hierarchies, which is a general architectural idea.
• Real-world applications:
  • Language-learning apps that give learners immediate phoneme-, word-, and sentence-level feedback, which is the stated purpose of CAPT systems.
  • Automated scoring of read-aloud test items, where accuracy, fluency, prosody, and completeness must be reported together.
  • Word stress and prosody coaching, the specific weakness the interactive attention module targets.
  • Classroom or self-study pronunciation drills that need one model to serve several feedback levels at once.
• Industry relevance: The model is deliberately small (48-dimensional embeddings, 3 encoder and 3 decoder layers, 1 head), so it is a realistic candidate for latency- and cost-sensitive deployment in commercial language learning products, where serving many learners per request matters. The paper also references open-response assessment scenarios such as IELTS as a related but separate setting.

Future Directions

• Extend to open-response speech. The paper's related work distinguishes open-response assessment (spontaneous speech without pre-specified text, e.g. IELTS, tackled by MultiPA) from the read-aloud setting the model is trained on, leaving that setting untested here.
• Test on larger and more diverse datasets. The authors attribute the degradation from larger embeddings and multiple heads, and from 2-3 convolutional layers, to limited data size; more data could let the extra capacity pay off.
• Generalize beyond speechocean762. The paper evaluates only on speechocean762, described as currently the only open-source standard dataset designed specifically for read-aloud pronunciation assessment, so generalization to other languages, corpora, or L2 populations is not reported.
• Reduce dependence on GOP features and forced alignment. The pipeline relies on a Librispeech acoustic model for GOP features and forced alignment; the paper's related work notes ASR-free and self-supervised alternatives, which could be combined with the interaction module.
• Extend granularity range. The framework currently covers phoneme, word, and utterance; sentence, paragraph, or discourse-level interaction is not reported.

Target Audience

This paper is most useful to researchers and engineers working on automatic pronunciation assessment, CAPT system development, and speech-scoring models, particularly those already familiar with Transformer architectures and GOP-based pipelines. It is also relevant to practitioners building multi-task speech models who are interested in cross-level attention and residual hierarchical designs, and to graduate students studying multi-granularity sequence modeling for speech.

Authors’ abstract

Automatic pronunciation assessment plays a crucial role in computer-assisted pronunciation training systems. Due to the ability to perform multiple pronunciation tasks simultaneously, multi-aspect multi-granularity pronunciation assessment methods are gradually receiving more attention and achieving better performance than single-level modeling tasks. However, existing methods only consider unidirectional dependencies between adjacent granularity levels, lacking bidirectional interaction among phoneme, word, and utterance levels and thus insufficiently capturing the acoustic structural correlations. To address this issue, we propose a novel residual hierarchical interactive method, HIA for short, that enables bidirectional modeling across granularities. As the core of HIA, the Interactive Attention Module leverages an attention mechanism to achieve dynamic bidirectional interaction, effectively capturing linguistic features at each granularity while integrating correlations between different granularity levels. We also propose a residual hierarchical structure to alleviate the feature forgetting problem when modeling acoustic hierarchies. In addition, we use 1-D convolutional layers to enhance the extraction of local contextual cues at each granularity. Extensive experiments on the speechocean762 dataset show that our model is comprehensively ahead of the existing state-of-the-art methods.