The Pulse
Apple Researchers Find Telling Languages Apart Helps Speech Models
A study of English-French HuBERT models found that training them to distinguish languages improved several speech and language measures while preserving cross-language sharing. The strongest results varied by intervention, and the experimen
AI.info Team · Written with AI ·
“We show that strengthening the model’s ability to discriminate languages during pretraining reduces and, on some measures, closes this multilingual gap on continuous phonetic and higher-level linguistic measures, while preserving substantial cross-language sharing.”
The authors of the study, Apple researchers Maureen de Seyssel, Jie Chi and Zakaria Aldeneh, write.
Apple researchers submitted the study to arXiv on September 27, 2026. Their experiments suggest that multilingual speech models may learn more effectively when training encourages them to tell languages apart, rather than treating cross-language sharing as the only route to better representations.
The finding addresses a trade-off: multilingual models can share useful patterns across languages, but may still trail single-language models when both receive the same total amount of training data. The researchers tested two ways to increase language discrimination in HuBERT, a self-supervised speech model, using English and French. They report gains in several measures while retaining cross-language sharing.
Two changes to English-French HuBERT
The team trained 95-million-parameter HuBERT-Base models from scratch. In the main comparison, the bilingual model received 500 hours of English speech and 500 hours of French speech; each monolingual reference model received 1,000 hours in its language. The setup held the total pretraining budget constant, making it possible to compare multilingual training with monolingual training at equal overall data volume.
One intervention added an auxiliary language classifier during HuBERT’s first training iteration. The other replaced shared targets in the second iteration with separate, language-specific k-means targets. The classifier was removed before the later stage, and neither approach required language labels at test time.
Different interventions led on different measures
The bilingual baseline recorded 11.6% error on continuous-feature phone discrimination, compared with 10.8% for the 1,000-hour monolingual reference. With the classifier intervention, that error fell to 10.4%. The same classifier condition raised the English-only lexical ProsAudit score from 68.9% to 72.9%, just above the monolingual reference of 72.6%.
The separate-target intervention produced the strongest reported sWUGGY lexical score: 56.7%, up from the bilingual baseline’s 52.1%, and closer to the monolingual reference’s 58.5%. Those headline results do not all come from one model condition: the 10.4% phone error and 72.9% ProsAudit score belong to the classifier condition, while the 56.7% sWUGGY score belongs to the separate-target condition. The paper reports averages across three training seeds.
Stronger discrimination was not always better
The paper keeps three properties apart. Language identity could already be decoded almost perfectly in the bilingual baseline, with linear-probe error under 1%; what was weak was discrimination, measured as language ABX error of 27.0%. Segregation, the clustering of representations into language-specific neighborhoods, was low. Both changes cut the discrimination error sharply, to 5.2% with the classifier and 2.9% with separate targets, while segregation stayed low.
The paper also found that timing mattered: the largest gains on most linguistic measures came when the classifier appeared in the first training iteration. Applying it later, or across both iterations, gave even stronger discrimination but smaller gains, together with more language-wise segregation; the authors suggest that segregation may partly offset the benefit by reducing cross-language sharing.
The authors caution against reading the results as a production-system benchmark. The experiments use a controlled English-French pairing, read-speech datasets and a specific HuBERT training recipe. Their evidence supports a causal benefit from stronger language discrimination in that setting, but does not establish that the same changes will improve every language pair or speech model.
A narrow result with a clear trade-off
The study’s practical contribution is a training signal that can help a multilingual model distinguish language identity without simply splitting its internal representations into isolated language groups. That distinction matters because discrimination may limit interference between languages, while shared representations can still carry useful information from one language to another.
For the tested English-French models, the clearest gains appear in continuous phonetic performance and lexical or prosodic measures, with different interventions producing the strongest result on different tasks. The work measures those gains in HuBERT-Base trained on 500 hours per language—not in a deployed Apple speech product.