X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation

AI summary

Overview

Research area: Speech processing / speech large language models (cs.SD), specifically post-training compression of the audio encoder in an ASR-capable LLM.

Technical level: Advanced. The paper assumes familiarity with Transformer block pruning, knowledge distillation, LoRA adapters, and ASR error metrics (CER/WER).

Scope: A single paper-length study of one compression recipe (progressive layer pruning plus three-stage recovery) applied to one model family (Qwen3-ASR-0.6B), evaluated on ten public Chinese–English ASR benchmarks plus proprietary development suites.

What This Paper Is About

Speech LLMs rely on a deep audio encoder that must process every input frame, which makes encoder depth a direct contributor to first-token latency. Removing whole Transformer blocks is attractive because it produces a regular, deployment-friendly model, but deleting blocks changes the audio embeddings fed to the language-model decoder and can cause large deletion errors and premature end-of-sequence (EOS) predictions. The paper's goal is to find which combinations of audio-encoder layers can actually be removed and recovered under a fixed training budget, and to design a recovery recipe that repairs both the hidden-state mismatch and the errors caused by the student's own decoding history.

Key Contributions

1. A progressive recovery framework that pairs behavioral candidate screening with cross-scale hidden-state and logit supervision, scheduled student-policy training, and LoRA finetuning, while keeping the pretrained decoder backbone frozen.
2. A matched teacher-scale comparison showing that a 1.7B teacher reduces mean error to 5.55%, versus 8.45% for self-distillation from the unpruned 0.6B model under the same recipe.
3. Layer-pair probes that expose non-additive interactions: the {6,8} pair combines the two strongest single removals but underperforms {5,6} by 0.85 pp after matched recovery.
4. Two practical operating points: a 16-layer model at 5.27% macro error (better than the 5.61% baseline), and a 14-layer model at 5.75% macro error with 20.7% fewer audio-tower parameters, 21.4% lower measured encoder latency on an in-vehicle accelerator and 11.4% lower on an H800.

Main Findings

• The 16-layer model beats the unpruned baseline: Macro-average error falls from 5.61% to 5.27%, an absolute change of −0.34 pp and a 6.1% relative reduction. Four benchmarks improve: AISHELL-1, CommonVoice zh, CommonVoice en, and WenetSpeech-meeting. The largest gains are CommonVoice zh (−1.83 pp), CommonVoice en (−1.85 pp), and WenetSpeech-meeting (−1.30 pp); the largest degradation is 0.44 pp on Tedlium.
• The 14-layer model is a second operating point with a small average cost: 5.75% macro error versus 5.61% for the baseline (+0.14 pp, +2.5% relative). CommonVoice zh improves by 1.59 pp and LibriSpeech test-clean by 0.03 pp; Fleurs-en degrades the most at 0.93 pp. Relative to the 16-layer model, seven benchmark changes stay within 0.3 pp, with CommonVoice en (+1.99 pp) and WenetSpeech-meeting (+1.72 pp) accounting for most of the gap.
• Cross-scale supervision matters more than self-distillation: At Stage 1 best checkpoints, the 1.7B teacher gives 5.55% mean error versus 8.45% for the self-teacher, and is better on all ten benchmarks. Relative to the unpruned baseline, the cross-scale checkpoint improves macro error by 1.1% while the self-teacher checkpoint increases it by 50.6%. The largest gaps are on Tedlium (11.28 vs. 3.92), LibriSpeech test-clean (6.00 vs. 2.73), and LibriSpeech test-other (9.89 vs. 5.90).
• Progressive pruning beats direct pruning: Progressive 18→16→14 reaches 5.75% mean error versus 6.73% for a direct 18→14 run with the same nominal recovery and data budget, and is better on all ten benchmarks. Largest direct-minus-progressive gaps: CommonVoice en (+2.73 pp), CommonVoice zh (+1.43 pp), Tedlium (+1.15 pp).
• Layer recoverability is not additive: Main-text Section 5.4 reports a single-layer sweep spanning 6.29–8.90% TER, with L6 the strongest single removal at 6.29%, L5 at 6.42%, and L8 at 6.52%. The selected {5,6} pair reaches 6.93% while {6,8} reaches 7.78%, giving interaction penalties of 0.58 pp and 1.38 pp respectively. Appendix A.4 reports different constituent values (L8=5.97%, L6=6.11%, L5=6.15%) and interaction penalties of 1.74 and 0.80 pp; the paper does not reconcile the two sets of numbers.
• Adjacency is not a reliable rule: The adjacent pairs {5,6}, {6,7}, and {8,9} rank ahead of the non-adjacent {6,8} and {3,6}, but the adjacent {14,15} pair performs worst of all at 9.93%.
• Stage 2 improves every entry over Stage 1: Stage 0 TER falls from 10.88% at step 500 to 7.80% at step 2500, then 8.38% at the first Stage 1 evaluation near step 4000. Stage 1 reaches its minimum of 5.76% at step 47,000, and Stage 2 cuts a further 0.40 pp to reach 5.36% at step 28,000.
• Premature EOS is partially mitigated: In a 2×2 ablation on prune-16, the no-mitigation condition shows five windows with a nonzero empty ratio (maximum 0.3%, mean 0.018%) and best TER of 6.86%; training the tied output head gives 6.83% with three events; minimum-token gating gives 6.97% with zero events; combining both gives 6.75% with zero events. Gating converts some premature terminations into longer or degenerate rollouts: configurations C and D trigger 2113 and 2162 filter rejections versus 20 and 11 for A and B. Mean rollout length is 15.06 tokens for D and 11.05 for A.
• Results are single-run: All runs use seed 42, with no repeated seeds, no bootstrap utterance-level confidence intervals, and no significance tests. Boldface in the tables marks the best observed number, not statistical significance.

Methodology in Plain English

The recipe starts from an already-pretrained ASR model (Qwen3-ASR-0.6B, with an 18-block audio Transformer and a bridge that feeds audio embeddings into the Qwen3 decoder) rather than training a small encoder from scratch.

1. Filter the training data by agreement. Two strong ASR systems transcribe each utterance, and the largest pairwise edit rate among the source transcript and the two hypotheses places each record into one of nine consistency tiers. All reported runs use the top tier (class 1). The Stage 2 finetuning stage keeps that same threshold but reweights sources toward cockpit and other target-domain data.
2. Prune in two hops with behavioral probes. The first hop removes original layers {1, 18} (18→16). For the second hop (16→14), every candidate starts from the same recovered 16-layer checkpoint and gets the same 0.3-epoch LoRA warm-up. Candidates are scored by the lowest aggregate TER on a fixed five-benchmark development suite, first as single removals and then as a fixed set of adjacent and non-adjacent pairs. This is more expensive than a static importance score but measures post-removal behavior directly.
3. Recover in three stages, per hop. Stage 0 (first 5% of the distillation epoch) aligns intermediate-layer and bridge representations plus logit and cross-entropy losses; teacher layers are divided uniformly into M ordered groups and student layer m aligns to the last teacher layer in group m; because teacher and student hidden widths are 2048 and 1024, a learned two-layer MLP with a 256-dimensional bottleneck projects teacher features into student space. Stage 1 (the remaining 95%) keeps bridge alignment, teacher-forced logit KD, and gold-transcript CE, and after 20% of the stage, every fifth optimizer step uses student-policy supervision: the student greedily generates a prefix and student and teacher are compared on that same generated context over the union of each model's top-512 support. Rollout filters reject degenerate prefixes, and if at least half of a microbatch is rejected it falls back to the teacher-forced objective, so the realized on-policy fraction can be below the ~20% target. Stage 2 runs one epoch of gold-transcript CE with target-domain source reweighting and no teacher loss.
4. Keep the decoder mostly frozen. Decoder base weights stay frozen; rank-32 LoRA adapters on q/k/v/o attention projections and, during Stages 0–1, the tied output embedding are trainable; the tied lm_head/embedding is frozen in Stage 2.

Training uses 32 accelerators, per-device batch 8, gradient accumulation 2, global batch 512, AdamW with weight decay 0.01 and gradient clipping 1.0 in bf16. Learning rates are 2×10⁻⁵ for the audio tower, 10⁻⁴ for decoder-side LoRA and the tied output embedding during distillation, and 5×10⁻⁶ for all trainable parameters in Stage 2. Checkpoints are chosen by the lowest selection-suite macro TER on five development subsets capped at 25 utterances per benchmark, and final numbers come from the corresponding single-run checkpoint on the full ten-benchmark public suite, where the macro mean weights benchmarks equally rather than by utterance count.

Why This Matters

For research, the paper argues that recoverability is a property of a combination of layers rather than an additive score per layer, and it reports evidence that progressive pruning outperforms the direct drop of the same four layers (5.75% vs. 6.73%). It also isolates the teacher as a variable in a matched comparison, which prior ASR compression work on encoder-layer removal and LoRA recovery did not do in this exact setting.

Real-world applications suggested by the paper's motivation:

• Streaming ASR, where encoder depth directly affects first-token latency.
• Mobile on-device speech recognition, where a regular pruned architecture is easier to deploy than an irregular sparse one.
• In-vehicle voice interfaces, the setting the authors measure on a PPU accelerator, where latency fell 21.4% relative.
• Server-side multilingual Chinese–English transcription, where the same checkpoint serves ten heterogeneous benchmarks and the 14-layer model cuts audio-tower parameters by 20.7%.

Industry relevance: the work comes from XPeng Inc. and targets commercial speech-stack deployment, where a 10.35% (16-layer) or 20.70% (14-layer) reduction in audio-tower parameters (186.376M → 167.085M → 147.794M) translates into concrete hardware savings. The paper is explicit that only the audio tower is compressed, so autoregressive decoding limits the end-to-end latency reduction.

Future Directions

• Repeated trials and uncertainty quantification. The primary results are single runs with seed 42, without paired utterance-level confidence intervals or significance tests, which leaves small differences such as the 0.14-pp gap between the 14-layer model and the baseline as descriptive observations. The paper calls for repeated trials.
• Broader model families and layer candidates. The study covers one model family and a limited candidate set; layer interactions, projection-based alignment, and EOS behavior may differ in Whisper, Qwen2-Audio, or other speech–language architectures.
• Controlled data-mixture experiments. The pipeline defines nine consistency tiers, but every reported run uses class 1 and changes only source weights in Stage 2; experiments across controlled data mixtures would clarify which parts of the recipe generalize.
• Deployment-grade efficiency studies. The latency averages come from one retained benchmark output over more than 50 utterances without run-to-run variability, so the authors call for repeated hardware measurements under a fixed software stack reporting latency distributions rather than averages. They also note that causal activation analysis and a larger factorial candidate set are needed to test the untested explanation that removing a contiguous sub-block creates one residual-stream discontinuity whereas dispersed removal creates two.

Target Audience

Researchers and engineers working on speech LLM efficiency, encoder pruning, and knowledge distillation for ASR, particularly those who need post-training compression of an already-deployed model rather than training a compact model from scratch. It is also relevant to practitioners building latency-sensitive speech systems for streaming, mobile, or automotive deployment, and to readers interested in how data-quality filtering and cross-scale teacher supervision interact with structural pruning. The paper states its single-run, single-family scope clearly, so readers looking for statistically established rankings should treat the reported comparisons as operating points rather than confirmed effects.

Authors’ abstract

Reducing audio-encoder depth lowers the inference cost of speech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause deletion and premature end-of-sequence errors. We introduce X-AuT, a progressive framework that selects layer combinations through short behavioral probes and restores the pruned model through representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning. The language-model backbone remains frozen, while attention LoRA adapters and the tied output embedding adapt during distillation. Training uses the highest-agreement tier from a transcript-consistency pipeline, followed by source reweighting during finetuning. On ten public Chinese--English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduces macro-average error from 5.61% to 5.27%. The 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters. Under the matched recipe, the 1.7B teacher yields 5.55% mean error, compared with 8.45% for self-distillation, and progressive 18$\rightarrow$14 pruning outperforms direct pruning (5.75% vs. 6.73%). These single-run results establish two practical operating points and show that the accuracy effects vary across benchmarks. Project website: https://xpeng-ai.github.io/x-aut