Do speech foundation models really learn words?

AI summary

Overview

Research area: Speech representation learning / interpretability of self-supervised speech models (Natural Language Processing & Speech Processing).

Technical level: Intermediate — assumes familiarity with self-supervised speech models, layer-wise linear probing, and basic linear algebra, but the core argument is accessible.

Scope: This paper tests whether self-supervised speech models like HuBERT and wav2vec 2.0 encode word identity as a genuinely word-level property, or whether their apparent "word knowledge" is just a byproduct of how well they encode phonemes.

What This Paper Is About

Previous work has shown that self-supervised speech models can be probed to identify words from their internal representations, which has been taken as evidence that these models "learn words." But a word is simultaneously a string of sounds (phonemes) and a meaning-bearing unit — so a model that merely encodes phonemes well could appear to know words without representing anything word-specific. This paper asks whether word information survives after phoneme information is mathematically removed from the model's representations.

Key Contributions

1. A residualization method for isolating non-phonological word information. The authors fit a ridge regression that predicts each frame's representation from its phoneme (or diphone, or triphone) label, then subtract the prediction. What remains is a representation with local phonological content linearly removed.

2. A clean dissociation between word and phoneme encoding. They show that word identity can still be decoded at frame level from later transformer layers of HuBERT and wav2vec 2.0 even after phoneme, diphone, or triphone information is removed — evidence that word encoding is not reducible to phonological form.

3. Validation that the intervention actually works. A parallel phoneme-identity probe confirms that residualization substantially damages a linear classifier's ability to recover phonemes, while standardization of features before residualization is shown to be necessary for the method to work.

4. A demonstration that this matters downstream. Applying phoneme residualization to HuBERT layer 9 improves unsupervised word segmentation and clustering (better normalized edit distance, token F1, and R-value), showing the technique has practical as well as interpretive value.

Main Findings

• Word identity survives phoneme removal. After residualizing out phoneme labels, linear probes still classify words at frame level with accuracy well above baseline in later layers — peaking around layers 9–10 for HuBERT and 7–8 for wav2vec 2.0, at a little above 90% for raw representations.

• Removing triphones hurts, but does not erase word information. Triphone residualization causes a substantial drop in accuracy on words of length 3–6, yet performance stays far above the modal-word-by-length baselines (14.9%, 0.4%, 0.1%, and near-zero for lengths 3, 4, 5, and 6) and above the 50.3% triphone baseline. Some word information therefore exists independently of short phonological sequences.

• The convolutional layer knows almost nothing about words. Word probe accuracy is low in the final conv layer and rises sharply through the transformer stack, indicating that word-level structure is built up by contextual layers.

• Phoneme residualization improves word discovery. On HuBERT layer 9, residualizing phonemes improved normalized edit distance, token F1, and R-value in unsupervised segmentation and clustering across multiple hyperparameter settings — but residualizing diphones or triphones degraded segmentation performance, likely because phoneme transition statistics are themselves useful for boundary detection.

• Standardization before residualization is essential. Without centering and scaling, phoneme information remains easily recoverable after residualization (near-100% in early layers), presumably because per-phoneme distribution shapes leak identity information.

• Residualization is not perfectly complete. Phoneme accuracy after residualization stays above the 11.6% modal-phoneme chance level for layers 1–10, because the authors deliberately excluded frames where a phoneme, diphone, or triphone exactly matched a whole word from the regression fit.

• The paper does not claim semantics. The results show the models treat words as units and integrate enough context to identify them locally, but they say nothing decisive about whether lexical meaning or syntax is encoded.

Methodology in Plain English

The researchers took two widely used pretrained speech models and ran English audio from LibriSpeech through them, capturing the internal representation at every layer for every short time frame (about every 20 ms). Because the audio has human-verified alignments, each frame is known to fall inside a specific phoneme and a specific word.

For each frame, they then asked a simple question: can a linear classifier read the word off the representation? They did this first on the untouched representations. Then they did the same thing again, but this time they first "subtracted out" phoneme information. That subtraction works by fitting a linear model that predicts the representation from a one-hot label of which phoneme is being spoken, and then subtracting the model's prediction from the actual representation. What remains is a version of the representation with the phoneme-level signal removed. This is essentially subtracting the average representation for each phoneme.

They repeated this with diphones (a phoneme plus its left or right neighbor) and triphones (both neighbors), to test whether word knowledge was hiding in slightly longer sound sequences. To verify the subtraction actually worked, they also trained a probe to predict phonemes from the residualized representations and checked that it dropped in accuracy.

Finally, in a downstream test, they plugged the residualized HuBERT layer-9 representations into an existing unsupervised word discovery pipeline that detects word boundaries and clusters them into word categories, measuring whether removing phoneme information helped or hurt.

Why This Matters

Impact on research. Probing studies have repeatedly reported that speech foundation models "encode words," but those results have been hard to interpret because a word probe could succeed purely by decoding its sound shape. This paper provides a way to disentangle the two, and its results strengthen the claim that contextual layers do build word-level units. It also argues that probing alone is insufficient — the authors show that a task where the raw representations already ace the probe (word identity) reveals nothing new, while a more complex downstream task reveals a real benefit from the intervention.

Real-world applications.

• Low-resource speech technology — word discovery without transcripts is a core problem for languages with little labeled data, and improving segmentation/clustering quality directly helps documentation and lexicon bootstrapping.
• Speech tokenization for language models — speech-aware LLMs rely on discrete tokens from these models; understanding whether those tokens carry word-level meaning informs tokenizer design.
• Keyword spotting and spoken term detection — if word identity is recoverable from later-layer frames, frame-level word detectors can be built without pooling.
• Language documentation and linguistic fieldwork — improved unsupervised word segmentation supports automatic transcription and analysis of unwritten languages.

Industry relevance. Companies building speech LLMs, voice assistants, and multilingual ASR systems care about whether their speech encoder is producing tokens aligned with meaningful units or merely with sounds. A residualization step that strips phonological information is cheap and, as shown here, can improve word-level clustering without retraining the model — attractive as a lightweight post-processing or feature-engineering technique. The authors caution, though, that the method requires phoneme alignments that are not usually available in production settings.

Future Directions

1. Removing the need for gold alignments. The biggest practical obstacle is that residualization requires precise phoneme labels. Approximate or automatically inferred phoneme labels (for example, from an ASR system or unsupervised unit discovery) could make the technique broadly usable.

2. Testing longer context windows. The authors constrained residualization to n-grams of length 3 to avoid unstable regression fits. Whether word information survives removal of longer phonological contexts, and whether that reflects genuine context-independent word representations, remains open.

3. Investigating what is actually encoded. The study shows words are treated as units but says nothing about semantics or syntax. Probing residualized representations for part-of-speech, argument structure, or meaning would clarify how far beyond sound shape these models go.

4. Connecting to the text-vs-speech gap. Speech models still lag text models on semantic tasks. Understanding why self-supervised learning recovers word identity so effectively from raw audio — and where its limits lie — is a central open question for closing that gap in textless language modelling.

Target Audience

Researchers and graduate students working on self-supervised speech representation learning, speech model interpretability, and probing methodology. It is also valuable for practitioners building speech tokenizers or speech-aware language models who want to know what their encoder's representations actually contain. Readers unfamiliar with linear probing or ridge regression will need to consult background material, but the core argument — that word knowledge in these models is not just phoneme knowledge — is stated clearly enough for a general NLP audience.

Authors’ abstract

Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.