Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder

AI summary

Overview

Research area: Self-supervised speech foundation models, utterance-level speech representation learning, and multi-task teacher–student distillation (speech–text alignment, speaker verification).

Technical level: Intermediate. The paper assumes familiarity with self-supervised speech encoders (wav2vec 2.0, HuBERT, w2v-BERT), knowledge distillation, and standard retrieval/verification metrics, but the core idea is described in accessible terms.

Scope: The paper proposes a unified post-training framework in which one shared speech encoder produces several different utterance-level representations — demonstrated here for semantic and speaker attributes — and shows that both can be learned at once with little loss to either.

What This Paper Is About

Speech foundation models normally produce frame-level acoustic representations, but recent post-training methods (SENSE, SAMU-XSLR, SONAR) instead align a speech encoder with a text embedding space so that whole utterances become semantically meaningful vectors. The problem is that optimizing only for semantics can suppress paralinguistic information such as speaker identity, emotion, or speaking style. This paper asks whether a single speech encoder can simultaneously produce multiple types of utterance-level representations, one per attribute, and tests this by jointly learning semantic and speaker representations.

Key Contributions

1. A general multi-task teacher–student framework for learning multiple utterance-level attribute representations from a shared speech encoder, where each attribute gets its own task-specific projection branch and supervision signal.
2. A demonstration that semantic and speaker representations can be learned jointly without significantly degrading the performance of either attribute.
3. A layer-weighting mechanism (scalar importance scores per layer, converted to softmax interpolation weights) that lets each attribute branch select which encoder layers to use — a mechanism the authors note is not present in the SENSE framework.
4. An analysis of layer usage across tasks, showing that semantic and speaker information are distributed differently within the shared encoder.

Main Findings

• Multi-task training preserves semantic retrieval: On VoxPopuli speech-to-speech translation retrieval, Att(sem+spk) stays close to the single-task semantic model Att(sem) across all evaluated language pairs and consistently outperforms SONAR. For example, on fr-de the multi-task model scores 93.83 R@1 versus 96.55 for Att(sem) and 92.15 for SONAR.
• Large gains over SONAR on the hardest pairs: SONAR collapses on several cross-lingual pairs in VoxPopuli (hu-da 3.32 R@1, da-hu 4.31, hr-en 52.16, en-hr 52.36, hr-cs 52.93, cs-hr 53.65), while Att(sem+spk) reaches 91.07, 90.79, 95.9, 95.79, 94.01 and 93.91 respectively. The paper attributes part of this contrast to SONAR using 37 language-specific encoders while the proposed models use a single multilingual speech encoder.
• Speech-to-text retrieval on MTEDx: Att(sem+spk) remains close to Att(sem) on most language pairs and stays above SONAR on the majority of them (for example it-en: SONAR 89.01, Att(sem) 90.69, Att(sem+spk) 90.10).
• Low-resource generalization on FLEURS: Att(sem+spk) stays close to Att(sem) and consistently outperforms SONAR, and it slightly improves over Att(sem) on the my-en pair (16.38 vs 14.11), suggesting semantic generalization is preserved in low-resource conditions.
• Speaker verification is nearly on par with the teacher: On VoxCeleb1-O, Att(sem+spk) achieves 0.91% EER and 0.1253 MinDCF₀.₀₁, versus 0.90% EER / 0.1104 MinDCF for the ECAPA-TDNN teacher and 0.93% EER / 0.1285 MinDCF for the single-task speaker branch Att(spk).
• Joint training may even help the speaker branch: Att(sem+spk) slightly outperforms Att(spk) on speaker verification, which the authors read as joint optimization potentially benefiting speaker representations.
• The two branches use different layers: Learned layer-interpolation weights show the semantic branch concentrating weight on a narrow band of middle layers with a strong peak around layers 13 and 14, while the speaker branch spreads weight more broadly across the encoder, gradually increasing toward the highest layers and peaking at layers 23 and 24.

Methodology in Plain English

The approach builds directly on SENSE, an open-source framework derived from SAMU-XSLR and close to Meta's SONAR, in which a frozen text embedding model acts as a teacher and a self-supervised speech encoder acts as the student.

In SENSE, the encoder's frame-level outputs are pooled into a single utterance vector and trained so that its cosine similarity to the teacher's sentence embedding is maximized, transferring semantic structure from text to speech. This paper generalizes that idea: for each target attribute, a separate branch is attached to the same shared encoder. Each branch projects the encoder's layer representations through an attribute-specific linear layer, assigns a learnable scalar importance score to each layer, converts those scores into normalized weights with a softmax, combines the projected layers with a weighted sum, applies layer normalization, and aggregates the result with attribute-specific attention pooling. An optional linear projection can follow, and the final embedding is ℓ2-normalized and aligned with its corresponding teacher embedding via a cosine similarity objective. All parameters, including the speech encoder itself, are jointly optimized across attributes.

For the experiments, the semantic branch aligns with BGE-M3 embeddings and the speaker branch aligns with embeddings from a pretrained ECAPA-TDNN speaker verification model (trained on VoxCeleb 1 and VoxCeleb 2). Both teachers stay frozen. The speech encoder is initialized from w2v-BERT 2.0, the implementation is built in SpeechBrain on top of the open-source SENSE recipe, and training uses the Common Voice 19 dataset covering the 83 languages supported by BGE-M3, corresponding to 8,250 hours of speech. Only the validated training dataset from Common Voice is used, with weighted sampling to handle language imbalance. The shared encoder is fine-tuned with Adam at a learning rate of 10⁻⁵, while task-specific modules use Adadelta with an initial learning rate of 1.5. Batch size is 20 for both training and validation, and training runs for 350K iterations on 8 H100 GPUs.

Evaluation covers two fronts. Semantic quality is measured by multilingual and multimodal translation retrieval using Recall@1 with mean-subtraction normalization, on VoxPopuli (speech-to-speech), MTEDx (speech-to-text) and FLEURS (speech-to-text, including unseen and low-resource languages). Speaker quality is measured by speaker verification on the VoxCeleb1-O protocol, reported as EER and minimum normalized Detection Cost Function with P_target = 0.01 and C_FA = C_Miss = 1, using cosine similarity between trial embeddings.

Why This Matters

Research impact. The paper shows that the "align speech to text semantics" recipe, which dominates recent utterance-level speech representation work, does not have to be a one-way trade-off against paralinguistic information. It also introduces a layer-selection mechanism that is absent from SENSE and provides evidence that different attributes naturally occupy different regions of a shared encoder — a useful signal for anyone designing multi-task speech systems or probing what speech foundation models encode where. The reported collapse of SONAR on several language pairs, contrasted with a single multilingual encoder, is also a notable data point about multilingual encoder design.

Real-world applications.

• Cross-lingual speech and speech-to-text search, where a user speaks in one language and retrieves matching spoken or written content in another.
• Speaker verification and speaker-aware voice interfaces built on the same encoder used for semantic tasks, avoiding a second model.
• Speech translation and translation search systems that need language-agnostic meaning representations.
• Multilingual content indexing and archival, where low-resource languages (the FLEURS and VoxPopuli conditions) are often the weakest link for existing systems.

Industry relevance. Consolidating semantic and speaker capabilities into one shared encoder reduces the number of models to train, deploy and maintain for products that need both meaning and identity, such as voice assistants, call analytics, and multilingual media platforms. The framework is designed to be extensible to arbitrary attributes, which makes it a template that can be adapted to a company's own set of target tasks.

Future Directions

• Extending the framework to additional attributes such as emotion, language, and accent, as the authors explicitly plan, toward richer representations from one unified encoder.
• Determining how many attributes a single shared encoder can absorb before interference between tasks degrades performance, given that the paper demonstrates only two.
• Investigating why joint semantic and speaker training slightly improves speaker verification over the single-task speaker branch (0.91 vs 0.93 EER), which the paper observes but does not fully explain.
• Exploring whether the learned layer-selection patterns generalize to other attribute pairs and other speech foundation models, and whether they can be used to guide which layers to adapt or share.

Target Audience

Researchers and engineers working on speech foundation models, self-supervised speech representation learning, speech–text multimodal alignment, multilingual retrieval, and speaker verification. It is most useful to readers already familiar with knowledge distillation and utterance-level embedding training who want to understand how to extend a single-task alignment pipeline to multiple attributes without retraining separate encoders.

Authors’ abstract

Speech foundation models trained with self-supervised learning produce generic speech representations that support a wide range of speech processing tasks. When further adapted with supervised learning, these models can achieve strong performance on specific downstream tasks. Recent post-training approaches, such as SAMU-XSLR and SONAR, align speech representations with utterance-level semantic representations, enabling effective multimodal (speech-text) and multilingual applications. While speech foundation models typically learn contextual embeddings at the acoustic frame level, these methods learn representations at the utterance level. In this work, we extend this paradigm to arbitrary utterance-level attributes and propose a unified post-training framework that enables a single speech foundation model to generate multiple types of utterance-level representations. We demonstrate the effectiveness of this approach by jointly learning semantic and speaker representations and evaluating them on multilingual speech retrieval and speaker recognition tasks.