The Pulse
Apple Tests a Whisper-Based Detector for Mixed-Language Speech
Apple researchers report that a detector using Whisper-small’s speech representations reached a mean F1 of 0.798 across 14 code-switched language pairs. The benchmark result depends on a small labeled sample for calibration, and the team ha
AI.info Team · Written with AI ·
Apple researchers report a mean F1 score of 0.798 for a detector that identifies speech mixing two languages, without first transcribing it. The result comes from Whisper-small evaluated on 14 English-and-other-language pairs in the CS-FLEURS benchmark. The paper was submitted to arXiv on October 13, 2026, and says the approach could help researchers find code-switched recordings in large speech collections.
Whisper-small finds language cues in its hidden states
The method looks for dimensions in a speech encoder’s internal representations whose activity differs between two languages. Researchers identify those dimensions using monolingual recordings, then track their frame-level activations to check whether an utterance contains evidence of both languages. The detector works on speech directly, avoiding the need to produce a multilingual transcript before screening an audio collection.
For Whisper-small, the team uses eight dimensions—about 1% of a 768-dimensional encoder layer—and combines their signals across short audio windows. A small set of labeled code-switched examples tunes the decision rule. The detector flags an utterance as mixed-language; it does not transcribe the words or locate every language boundary.
The 0.798 score uses oracle tuning
The headline result is an oracle score: researchers choose the decision-rule settings using the evaluation set itself. Under that setup, Whisper-small’s mean F1 of 0.798 tops a CNN baseline at 0.776 and a HuBERT linear classifier at 0.751. F1 balances precision and recall, but the oracle comparison does not show how the detector performs when its settings must be chosen in advance.
In a more practical calibration test, the team used 10% of the labeled CS-FLEURS examples to tune the rule and evaluated on the remaining 90%. Whisper-small’s mean F1 fell to 0.784, compared with 0.760 for the CNN and 0.733 for the HuBERT linear baseline. The paper notes that the calibration sample can be very small for some pairs: Italian-English, for example, has 176 code-switched examples in the benchmark, leaving about 18 in a random 10% sample.
Conversation brings a different test
The researchers also evaluated on SEAME, a corpus of spontaneous Mandarin-English speech from Singapore and Malaysia. Whisper-small scored 0.733 in the calibrated test, while the HuBERT linear baseline reached 0.769. The paper’s strongest overall result therefore comes from read benchmark speech, not from the conversational data.
Across the 14 CS-FLEURS pairs, Whisper-small performed best on 11, but the leading model varied by language pair. MMS-1B, for instance, reached the highest single-pair score, 0.918, for Telugu-English; Whisper-small ranged from 0.754 for German-English to 0.870 for Telugu-English. The averages conceal meaningful differences between languages.
Corpus mining remains untested
The researchers frame the detector as a pre-filter for finding candidate recordings, not as a finished speech product. Their motivation is that multilingual training collections contain too few examples of people switching languages within an utterance, while gathering and labeling such speech is difficult. As the authors write, “Our findings suggest that pretrained speech encoders already encode the information needed to identify CS utterances, and they provide a lightweight tool for building more representative multilingual corpora.” Jie Chi et al., “Code-Switching Detection via Language-Selective Dimensions in Speech Foundation Models”
The paper’s limits are direct: the detector has not been tested as part of a mining pipeline on unlabelled, in-the-wild audio, and it still needs some labeled code-switched examples to tune its decision rule. Whether it can reliably screen large, messy audio collections without that labeled set remains an open research question.