The Pulse
Apple Reports 2.8× Compression for a Streaming Audio Encoder
Apple researchers describe distilling an on-device Dictation audio encoder, with a student model that uses fewer parameters and stays close to its teacher on five of six tested pairs.
AI.info Team ·
Compressing the encoder used by Dictation
Apple researchers report a method for compressing a streaming neural audio encoder used by system-wide Dictation on Apple devices. In the study, a distilled student encoder is 2.8 times smaller than its teacher. It stays within 1.9% relative word error rate (WER) of the teacher in five of six teacher–student comparisons, without fine-tuning.
The encoder, which Apple calls a tokenizer, maps short windows of audio into representations read by a foundation model. It produces one vector every 80 milliseconds of audio. The paper says the tokenizer and language model share memory, and the tokenizer’s parameter count affects the memory available to the model, as well as power and latency. The research page describes the study; it does not announce a product deployment.
Distilling a shared representation
Rather than training the student to reproduce the teacher’s discrete tokens or output distribution, the researchers use the teacher’s representation immediately before quantization as the target. That latent representation is shared by the two token interfaces discussed in the paper: a discrete interface, which maps vectors to codebook indices, and a continuous interface, which projects vectors for the language model. The student encoder is trained to match the teacher’s representation frame by frame using squared error. An affine layer accounts for the difference in width between teacher and student.
The teacher remains frozen during training; only the student encoder and the affine layer receive gradients. The student inherits the teacher’s quantizer or bridge, as applicable. The researchers say the affine layer accounts for less than 1% of the student encoder and can be folded into the downstream interface or removed for inference. They also report a 2.8-fold reduction in multiply-accumulate cost, alongside the parameter reduction.
The method is tested at two points in the tokenizer’s training: after pretraining the tokenizer alone, and after training it jointly with a language model. The experiments use unlabeled audio, since the student learns from the teacher’s activations rather than transcripts. For the pretrained stage, WER is evaluated on LibriSpeech, TED-LIUM, VoxPopuli, AMI and Earnings-22. The joint-model tests use a different suite, including LibriSpeech, VoxPopuli, MLS, Common Voice and FLEURS.
Results vary across the six comparisons
Three of the six teacher–student pairs come from the pretrained stage. In those tests, the distilled students are within 1.9% relative WER of their teachers; one student is 0.4% worse, another 1.9% worse, and the third 0.8% worse. On some individual test sets, students also score better than their teachers.
The other three comparisons use teachers jointly trained with language models, and their results are less uniform. One student is within 1.5% relative WER of its teacher. Another improves on its teacher across every reported test set, with average WER of 11.68% versus 12.33%. The remaining student is 7.7% relatively worse, the largest degradation reported. That result is the exception to the five-of-six summary.
At equal capacity, the distilled tokenizer also outperforms an independently trained tokenizer by 3.9% relative WER, according to the paper. The authors report this as evidence that the teacher signal contributes beyond using a smaller architecture. The findings describe evaluations of the research models; they do not establish that the compressed encoder has been deployed in Dictation.