When Can You Prune Your Network? A Study of Intermediate Neurons in Multilingual Speech Parsing

AI summary

Overview

• Research area: Natural Language Processing — specifically end-to-end speech parsing (joint automatic speech recognition and dependency parsing), multilingual and low-resource speech processing, and neural architecture design.
• Technical level: Advanced. The paper assumes familiarity with dependency parsing, CTC-based ASR, self-supervised speech encoders (Wav2Vec2, data2vec), and representation-similarity analysis (Centered Kernel Alignment).
• Scope: The paper tests when intermediate neural network units between a pretrained encoder and a parsing model are actually useful, and proposes a simplified end-to-end speech parser evaluated on Naija, Slovenian, and French spoken treebanks.

What This Paper Is About

End-to-end speech parsing takes raw audio and outputs both a transcription and a syntactic dependency tree. Prior architectures such as Wav2tree connect the pretrained speech model to the parsing model through intermediate neural network units (a feedforward network, an LSTM, and a bidirectional LSTM), which add parameters and training time but have never been systematically studied for parsing.

This paper asks what those intermediate units actually do, introduces a simplified architecture that removes them, and shows that they only add value under one specific condition: when the pretrained encoder is frozen and its representations no longer match the downstream parsing task.

Key Contributions

1. A simplified end-to-end speech parsing architecture (Simplified W2T). The authors remove all three intermediate NN units from Wav2tree and instead feed averaged Wav2Vec2 frame representations directly to the parser. This reduces the parameter count by 12% relative to Wav2tree under the hyperparameters used by Pupier et al. (2024), while matching or improving performance on ASR and parsing.

2. A characterization of the "representational gap." The authors define the representational gap as the mismatch between an encoder optimized for its pretraining objective (when frozen) and the parsing model, and demonstrate experimentally that intermediate NN units reduce this gap.

3. A multilingual evaluation of spoken dependency parsing on Naija-NSC, Slovenian SST, and French ParisStories, including the first end-to-end speech parsing experiments for Naija and Slovenian, plus comparisons against a pipeline approach and a gold-transcription upper bound, and an analysis of using intermediate Wav2Vec2 layers as input.

4. A data-scaling study and language-specific analyses. Using Naija and the much larger French Orféo treebank (122.1h train), the authors assess how much speech data the parser needs, and separately examine the usefulness of Naija syntax-prosody segmentation annotations for speech parsing and release code to produce a filtered UD_Slovenian-SST v2.16 with improved speech-transcription alignment.

Main Findings

• Removing intermediate units usually helps or is neutral: After removing the three units, LAS (Labeled Attachment Score) increased by 1.2% absolute on Slovenian and 2.5% on French (ParisStories), and decreased by 0.95% on Naija. On Naija, Naija's UPOS improved from 77.8 to 79.1 and WER went from 31.4 to 31.8. For Slovenian, LAS moved from 37.7 (baseline) to 38.9; for ParisStories, from 50.9 to 53.3. ASR performance remained consistent across baseline and simplified architectures for all three datasets. The improvements for Slovenian and ParisStories are reported as statistically significant.

• Intermediate units matter only when the encoder is frozen (H2): In text-parsing experiments with gold transcriptions, LAS was always higher without an intermediate LSTM when the text encoder was jointly fine-tuned (Naija 83.9 vs 81.7; Slovenian 71.3 vs 66.1; ParisStories 72.0 vs 69.8). When the text encoder was frozen, LAS dropped to 51.5, 40.9, and 44.8 respectively without an LSTM, but recovered to 74.6, 55.4, and 62.9 with one — an increase of 15-20% across all three treebanks.

• Representation similarity confirms the gap-closing effect: Using linear Centered Kernel Alignment, similarity between the fine-tuned encoder representation and the frozen encoder representation was 17% (Naija), 21% (Slovenian), and 23% (French) without an intermediate unit, and rose to 44%, 32%, and 34% with one.

• Intermediate encoder layers work for parsing: Using the intermediate Wav2Vec2 layer that scored best on development data, LAS remained stable or improved on all three treebanks (Naija 61.3 with 21 layers; Slovenian 40.7 with 18 layers; ParisStories 54.2 with 11 layers), corresponding to a 12%-24% reduction in parameters and a 5-9% speedup in training time.

• End-to-end beats pipeline except for Slovenian: The pipeline approach (ASR followed by a text parser) had LAS of 61.2 on Naija, 50.9 on Slovenian, and 48.7 on ParisStories, and uses 2.75 times more parameters than the Simplified architecture due to requiring two pretrained models. The end-to-end Simplified model beat the pipeline on Naija and ParisStories but not on Slovenian.

• Gold transcriptions remain far ahead: The GOLD upper bound (parsing gold transcriptions) reached LAS of 83.9 (Naija), 71.3 (Slovenian), and 71.9 (ParisStories), showing a large remaining gap for end-to-end parsing.

• Parsing gains diminish after 9-10 hours of data: With a pre-fine-tuned ASR model held fixed, parsing accuracy gains diminish beyond 9-10 hours of training data. In the Baseline (jointly fine-tuned) setup, WER decreased and LAS improved with more data on both Naija and Orféo. In the Frozen ASR setup, LAS increased only up to 80% of the training data on Naija and stayed relatively stable across all partitions on Orféo.

• Naija segmentation annotations do not help speech parsing: LAS dropped by 4.1% absolute when punctuation/segmentation marks were present in the evaluation data. As the number of markups in the ground-truth transcript increased, ASR predicted fewer of them and LAS degraded further — for example, for trees with 13 or more markups, LAS was 44.0 with markups versus 54.0 without. The authors attribute this to these marks not corresponding to identifiable sounds in the audio.

• English bias in the Naija speech encoder: WER was 24.5% for the 8,991 Naija tokens found in an English word lexicon and 52.5% for the 1,269 tokens not found in it, which the authors link to the English bias of wav2vec2-xls-r-300m and limited Naija-related pretraining data.

• Statistical testing: Using the Wilcoxon signed rank test on LAS, the null hypothesis of no difference between simplified and baseline could not be rejected for Naija (p = 21.6%), but was rejected for Slovenian (p = 0.2%) and ParisStories (p = 3.6%), favoring the simplified architecture as better.

Methodology in Plain English

The authors start from Wav2tree, an end-to-end model that takes raw speech, runs it through a pretrained speech encoder, tags each frame with a character via CTC to produce a transcription and word boundaries, then builds word-level vectors and passes them to a dependency parser. Three intermediate units sit between these stages.

To test whether those units earn their keep, the authors build a stripped-down version. They apply CTC directly to the encoder output, average all frames inside each predicted word to get a word vector, and send that sequence straight to the parser — no feedforward network, no word-level LSTM, no bidirectional LSTM. They compare this against the original on the same data and hyperparameters.

Because the same pruning cannot be tested directly on speech (Wav2Vec2 must be fine-tuned for the target language), they replicate the question in text: parse gold transcriptions with a pretrained text encoder, once with the encoder fine-tuned jointly and once frozen, each with and without an LSTM. They also quantify how similar the fine-tuned and frozen representations are using Centered Kernel Alignment.

For scaling, they split training data into 20%, 40%, 60%, 80%, and 100% subsets and run two setups: the standard simplified model, and a version where the ASR model is pre-fine-tuned and frozen, with an LSTM added to bridge it to the parser.

Models and data: the speech encoder is wav2vec2-xls-r-300m for Naija and Slovenian, and Speech_Large_fr_114K (a data2vec model trained on 114k hours of French speech) for ParisStories. The text encoder is xlm-roberta-large. The parser is a biaffine graph-based parser. Metrics are WER for ASR and UPOS, UAS, and LAS for parsing.

Why This Matters

Impact on research: The paper gives architecture designers a concrete rule: if you fine-tune your encoder jointly with the downstream task, intermediate bridge modules are redundant, but if you freeze it — for cost or stability reasons — you need them. It also provides the first end-to-end speech parsing baselines for Naija and Slovenian, and a reusable filtered version of UD_Slovenian-SST v2.16.

Real-world applications:

• Data-driven linguistics and syntactic study of spontaneous speech, where spoken treebanks are much scarcer than written ones.
• Computational language documentation, where automatically parsed speech can support annotation of endangered or under-documented languages.
• Speech corpus annotation pipelines that need both transcription and syntactic structure from audio in one pass.
• Efficient multilingual NLP deployment, since the simplified architecture cuts parameters and training time while keeping accuracy.

Industry relevance: A 12%-24% parameter reduction and 5-9% training speedup matter directly for training cost and deployment footprint, especially for multilingual products where separate ASR and parsing models (as in the pipeline, which uses 2.75 times more parameters) are expensive. The finding that parsing gains flatten after 9-10 hours of transcribed speech is also a practical budgeting signal for teams deciding how much labeled audio to collect.

Future Directions

• Explain why the pipeline outperforms end-to-end speech parsing on the Slovenian treebank.
• Explain why parsing performance stays stable when representations from an intermediate layer of the speech encoder are used as input.
• Extend the analysis of intermediate neural network units to neural architectures beyond parsing.
• The authors note that hyperparameter search was limited to the learning rate due to computational constraints, and that they could not evaluate on the English Switchboard corpus because its audio recordings are not publicly available — leaving both as open limitations.

Target Audience

Researchers and practitioners working on speech parsing, joint ASR-and-parsing architectures, and multilingual or low-resource spoken language processing. It is also useful for engineers designing efficient NLP pipelines who want evidence on when bridge modules between a pretrained encoder and a task head are needed, and for computational linguists interested in dependency parsing of spontaneous speech in Naija, Slovenian, and French.

Authors’ abstract

End-to-end speech parsing, a task recently proposed, consists in predicting both the transcription and the syntactic tree for a spoken utterance. Existing architectures for speech parsing often utilise intermediate neural networks. In this work, we examine the effectiveness of intermediate neural networks (NN) for parsing, and, specifically, what role do they play. We introduce a simpler end-to-end architecture for speech parsing, where we remove these intermediate NN units, reducing the parameters by 12%, while achieving comparable or better performance than prior method on both automatic speech recognition (ASR) and parsing. We demonstrate that intermediate NN units help reduce the representational gap when the pre-trained encoder is frozen. We do a comprehensive evaluation of speech parsing on French, and medium-low resource languages Slovenian and Naija. We further investigate the impact of the training data size and intermediate layers of the pretrained speech encoder on speech parsing.