Bridging the gap: A comparative exploration of Speech-LLM and end-to-end architecture for multilingual conversational ASR

AI summary

Overview

• Research area: Multilingual conversational automatic speech recognition (ASR), specifically the integration of large speech encoders with large language models (Speech-LLM), parameter-efficient fine-tuning (LoRA), and speech encoder fusion.
• Technical level: Advanced.
• Scope: This paper systematically compares a parallel-speech-encoder Speech-LLM system against carefully fine-tuned end-to-end Whisper models on the INTERSPEECH 2025 MLC-SLM multilingual conversational ASR task.

What This Paper Is About

The authors want to know two things about Speech-LLM systems for multilingual conversational ASR: whether richer fusion methods than simple feature concatenation can better exploit complementary information from parallel speech encoders, and how large the real performance gap is between LLM-based ASR and end-to-end encoder-decoder ASR trained on identical data. They build on their earlier SHNU-mASR system (Whisper plus mHuBERT encoders feeding an LLM) and test several cross-attention fusion mechanisms, projector designs, and training schedules. Their system reaches a CER/WER of 10.69% on the official MLC-SLM evaluation set, but still does not match a fine-tuned end-to-end Whisper model.

Key Contributions

1. Analyzes Whisper Large-v3 encoders under both LoRA-based and full fine-tuning strategies on the MLC-SLM ASR task.
2. Proposes and compares five cross-attention-based fusion mechanisms for the parallel-speech-encoder architecture: Direct Feature Concatenation (DFC), Unidirectional Cross-Attention Fusion with Residual Connection (Res-Uni-CAF), Bidirectional Cross-Attention Fusion with Residual Connection (Res-Bi-CAF), Gated Res-Bi-CAF (Res-Gated-Bi-CAF), and Res-Gated-Bi-CAF with DFC (Res-Gated-Bi-CAF©DFC).
3. Investigates the impact of projector design (linear projector versus Q-Former) between the speech encoders and the LLM on MLC-SLM ASR performance.
4. Provides direct comparisons between the Speech-LLM system and well fine-tuned end-to-end Whisper models under the same training data, offering empirical guidance for future Speech-LLM design. Code is released at https://github.com/1535176727/MLC-SLM.

Main Findings

• Full fine-tuning wins in-domain: On the in-domain Dev and Eval sets, full fine-tuning of Whisper Large-v3 achieves the best end-to-end performance (10.99% on Dev, 10.07% on Eval), while LoRA fine-tuning also improves substantially over vanilla Whisper (11.40% on Dev, 10.71% on Eval versus 16.04% and 15.96% for vanilla Whisper).
• Fine-tuning hurts out-of-domain robustness: On the out-of-domain CV-TestSet, vanilla Whisper shows the lowest WER at 9.86%, whereas LoRA fine-tuning rises to 11.47% and full fine-tuning to 13.11%, with full fine-tuning suffering the most. The authors describe this as a trade-off between in-domain adaptation and cross-domain generalization.
• Encoder complementarity: The large performance gap between mHuBERT (CTC) and Whisper (29.67% versus 11.40% on Dev in Table 1) suggests the two encoders may provide complementary information. (Note: Table 4 lists 19.99% for mHuBERT (CTC) on Eval, while Table 1 lists 25.65% for the same system; the paper does not explain this difference.)
• Linear projector beats Q-Former: In the parallel-speech-encoder setting with direct concatenation and a LoRA-finetuned Whisper encoder, the Linear projector outperforms the Win-level Q-Former on all sets (Dev 11.91% vs 12.52%, Eval 11.05% vs 11.51%, CV-Test 14.91% vs 15.64%), indicating a simple linear mapping is sufficient, more robust, and more efficient.
• Gated fusion helps mainly in Stage 1: During Stage 1 (projector-only training), gated fusion consistently outperforms DFC and Res-Uni(Bi)-CAF, with the largest gains on Dev and Eval and strong robustness on OOD CV-Test. In Stage 2 (joint LLM LoRA and projector optimization), these gains diminish substantially, suggesting that once the speech encoders are sufficiently task-specifically fine-tuned, the choice of fusion mechanism matters less.
• Two-stage training beats tri-stage: Comparing the DFC rows against the SHNU-mASR rows, the two-stage strategy proves more effective on Dev and CV-Test in Stage 1 and achieves better performance across all three test sets in Stage 2.
• LoRA-finetuned encoders generalize better inside Speech-LLMs: Using a fully fine-tuned Whisper encoder inside the Speech-LLM brings gains only on the Eval set, offers almost no benefit on Dev, and causes a significant performance drop on the OOD CV-Test set.
• Competitive result with less data: The proposed Speech-LLM achieves 11.74% on Dev, 10.69% on Eval, and 15.26% on CV-Test, using only 1,500 hours of baseline training data and no external resources, while systems such as NTU-Speechlab (11.57% / 10.58%) and Seewoo (12.73% / 11.57%) used large-scale external datasets. The paper states that the system ranks on par with the top-ranked Track 1 systems.
• A gap remains: The final LLM-based ASR still does not match the fine-tuned end-to-end Whisper model (10.69% versus 10.07% on Eval; 15.26% versus 13.11% on CV-Test), indicating inherent limitations in current Speech-LLM design.

Methodology in Plain English

The system uses two speech encoders running in parallel. Whisper captures broad multilingual acoustic and linguistic patterns, while mHuBERT provides a different, complementary view of the speech. Their outputs are combined using one of five fusion methods, ranging from simply stacking the two feature sets side by side, to letting one encoder "look at" the other through cross-attention, to a bidirectional version, and finally to a version with learnable gates that decide how much of the attended information to keep. A projector then maps the combined features into the LLM's embedding space; the authors compare a simple linear projector (1D convolutions plus an MLP) against a Q-Former that uses learnable queries to summarize the speech.

Training is streamlined into two stages instead of the earlier three. First, each speech encoder is fine-tuned on its own: Whisper with either LoRA or full-parameter updates, and mHuBERT with a CTC objective. The tuned encoders are then connected to the LLM. In Stage 1 only the projector is trained; in Stage 2 the projector and the LLM's LoRA parameters are updated jointly while the speech encoders stay frozen. Experiments use the 1,500-hour MLC-SLM dataset covering 11 languages, with an internal validation set of roughly 2 hours per language for model selection, the official Dev and Eval sets for evaluation, and an out-of-domain set drawn from CommonVoice 21.0 (2 hours per language, except Korean at 0.6 hours and Vietnamese at 1.4 hours). The LLM is Qwen2.5-7B, LoRA on Whisper uses rank 32 and alpha 64, training runs for 6 epochs per stage on four NVIDIA A100 80G GPUs in bfloat16, and scoring uses CER for Japanese, Korean, and Thai and WER for the other languages via the meeteval toolkit.

Why This Matters

Impact on research: The paper supplies a rare apples-to-apples comparison on identical training data, showing that a fine-tuned end-to-end Whisper still outperforms a comparable Speech-LLM. It also shows that expensive fusion machinery adds little once the encoders are well adapted, and that a simple linear projector beats a Q-Former — findings that redirect effort away from architectural complexity and toward training-strategy design. The release of code makes these comparisons reproducible.

Real-world applications:

• Multilingual contact centers and customer-support lines, where conversational speech in many languages must be transcribed accurately.
• Meeting and conference transcription systems serving mixed-language participants.
• Voice assistants and captioning tools that must operate in code-switched or non-Latin-script languages such as Japanese, Korean, and Thai.
• Deployment scenarios requiring robustness to domain shift, where the paper's out-of-domain CommonVoice results are directly informative.

Industry relevance: The finding that a LoRA-finetuned Whisper encoder is sufficient and preferable for generalization inside a Speech-LLM has direct cost implications, since parameter-efficient adaptation demands far less compute than full fine-tuning. The result that a 1,500-hour baseline training set can approach the performance of systems trained on large-scale external datasets is also relevant for organizations that cannot assemble massive proprietary corpora.

Future Directions

• Close the remaining gap between Speech-LLM systems and fully fine-tuned end-to-end Whisper models trained on the same amount of speech data, which the authors attribute to limitations in model design and optimization.
• Reconsider fusion mechanism design, since gains from gated fusion largely disappear after joint optimization with the LLM — the question is whether fusion matters at all once encoders are well adapted, or whether it should be redesigned for the joint-training regime.
• Explore more robust methods for inserting fully fine-tuned encoders into Speech-LLMs, given the observed out-of-domain degradation.
• Investigate why the simple linear projector outperforms the Q-Former, and whether alternative cross-modal alignment designs can beat a plain linear mapping.

Target Audience

Speech and language researchers working on Speech-LLM integration, multilingual ASR, and speech encoder fusion; engineers building production multilingual transcription systems who need guidance on fine-tuning strategy and architecture choices; and graduate students or practitioners with some background in ASR who want a concrete, empirically grounded comparison between end-to-end and LLM-based approaches.

Authors’ abstract

The INTERSPEECH 2025 Challenge on Multilingual Conversational Speech Language Models (MLC-SLM) promotes multilingual conversational ASR with large language models (LLMs). Our previous SHNU-mASR system adopted a competitive parallel-speech-encoder architecture that integrated Whisper and mHuBERT with an LLM. However, it faced two challenges: simple feature concatenation may not fully exploit complementary information, and the performance gap between LLM-based ASR and end-to-end(E2E) encoder-decoder ASR remained unexplored. In this work, we present an enhanced LLM-based ASR framework that combines fine-tuned Whisper and mHuBERT encoders with an LLM to enrich speech representations. We first evaluate E2E Whisper models with LoRA and full fine-tuning on the MLC-SLM ASR task, and then propose cross-attention-based fusion mechanisms for the parallel-speech-encoder. On the official evaluation set of the MLC-SLM Challenge, our system achieves a CER/WER of 10.69%, ranking on par with the top-ranked Track 1 systems, even though it uses only 1,500 hours of baseline training data compared with their large-scale training sets. Nonetheless, we find that our final LLM-based ASR still does not match the performance of a fine-tuned E2E Whisper model, providing valuable empirical guidance for future Speech-LLM design. Our code is publicly available at https://github.com/1535176727/MLC-SLM.