Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs

AI summary

Overview

• Research area: Natural Language Processing / speech-to-text translation (ST); specifically the comparison of SpeechLLMs (LLMs with native speech input) against cascaded and direct speech translation systems.
• Technical level: Intermediate. Readers need some familiarity with speech translation architectures, automatic metrics, and LLM terminology, but the paper's central comparison is easy to follow.
• Scope: A large-scale, reproducible benchmark suite ("Hearing to Translate") benchmarking 6 SpeechLLMs against 16 direct and cascade systems across 16 benchmarks, 13 language pairs, and 9 speech conditions.

What This Paper Is About

Large Language Models have recently been extended to accept speech directly, producing "SpeechLLMs" that translate spoken language end-to-end without a separate transcription step. The open question is whether this integration of speech into LLMs actually improves speech translation quality compared with long-established cascaded systems (speech recognition followed by a text translation model) or with direct, speech-translation-specialized models. The paper builds a comprehensive test suite to answer that question empirically rather than by assumption.

Key Contributions

1. The Hearing-to-Translate Suite: a released test suite (available at the project GitHub repository) that benchmarks speech translation across 9 explicitly defined conditions: GENERIC, GENDER BIAS, ACCENTS, CODE SWITCHING, DISFLUENCIES, NAMED ENTITIES, NOISE, EMOTION, and LONG-FORM.
2. A systematic comparison of 22 systems: 6 SpeechLLMs (Phi-4-Multimodal, Qwen2-Audio, Qwen3-Omni, DeSTA2, Voxtral, Spire) against 16 strong systems (4 direct and 12 cascade) built from four speech foundation models (Whisper, SeamlessM4T, Canary, OWSM) and three LLMs (Aya Expanse, Gemma, Tower+), all under 32 billion parameters.
3. A new noise-robustness benchmark (NoisyFLEURS), derived from FLEURS by adding two realistic noise types, babble (B) and ambient (A), from the MUSAN corpus, released under CC-BY-NC 4.0.
4. A small-scale human evaluation with automatic metric correlation analysis, using ESA and MQM protocols on CoVoST2 with five native speakers and 60 annotated items per language pair, plus releases of the human annotations under CC-BY 4.0.

Main Findings

• Cascades remain the most reliable overall: cascaded systems deliver the strongest and most consistent translation quality across languages, benchmarks, and acoustic conditions, although most recent SpeechLLMs can match or even outperform them in various settings.
• Some SpeechLLMs close the gap: Voxtral and Qwen3-Omni are the only SpeechLLMs that reliably close, and sometimes overturn, the gap with the best-performing cascades on the GENERIC benchmarks.
• Standalone speech foundation models lag behind: SFMs generally trail both cascades and SpeechLLMs, indicating that integrating an LLM — either inside the model or in a pipeline — is essential for high-quality speech translation. OWSM performs worst as a standalone SFM but recovers most of its gap when paired with an LLM.
• Gender bias is small on FLEURS but larger on WinoST: FLEURS gender speaker gaps range from 0.9 to –2.4, with gaps slightly larger when translating from English than into English. WinoST exposes substantially larger F1 gaps, and the choice of LLM matters: pairing ASR modules with Gemma3 produces substantial gaps, whereas Tower+ significantly mitigates them.
• Gender bias is tied to the LLM decoder: cascades using Tower+ cluster near zero on both the gender coreference gap and the stereotypical gap, while other systems degrade on anti-stereotypical roles. The two gaps are positively correlated (r = 0.54).
• Accent robustness is encoder-driven: Seamless, used directly or in a cascade, achieves the strongest performance on CommonAccent, outperforming the best cascades and SpeechLLMs by at least 1.5 xCOMET_S^QE on en–x. The most challenging accents include South Asian English, Austrian German, Rioplatense Spanish, and Basilicata–Trentino Italian; in ManDi, Standard Mandarin scores highest while Taiyuan performs worst.
• Code switching favors cascaded Whisper and especially Voxtral, while Whisper alone lags behind both paradigms, showing that both encoder and decoder matter.
• Disfluency robustness varies widely: Voxtral, DeSTA2, and Whisper cascades are most robust to stuttered speech, while Seamless, OWSM, and Phi-4-Multimodal show large degradations (43–75 Δdisfluency).
• Named entity and terminology accuracy follow the LLM: the highest translation quality on EuroParl-ST en–x comes from Canary-based cascades, but the best named entity accuracy comes from systems combining Tower+ (strongest with Seamless), followed by Qwen3-Omni.
• SpeechLLMs are the most robust under noise: all models degrade under both noise types, with babble noise causing extreme degradation (minimum 38 Δnoise), but all SpeechLLMs except Spire show equal or greater robustness than SFMs and cascades. Manual inspection found SFMs often hallucinate under noise while LLMs propagate or amplify those errors.
• Cascades handle emotion and long-form better: cascades are more robust than SFMs and SpeechLLMs (except Qwen3-Omni) on emotional speech, contradicting prior work that found direct systems better at capturing prosody. On long-form input, DeSTA2 and Qwen2-Audio show extreme degradation (Δlength ≈ 91–94), while cascades with OWSM, Canary, and Voxtral achieve near-zero Δlength. Voxtral is notable for re-concatenating all chunk representations before feeding them to the LLM, enabling real long-context translation.
• Human evaluation mirrors automatic metrics: average human scores were 81.66 for the cascade, 80.41 for the SpeechLLM, and 78.33 for the SFM. Mistranslation was the most common error type, omissions were twice as frequent in the SpeechLLM as in the cascade and SFM, models employing LLMs were more affected by overtranslation (doubling this error compared to the SFM), and wrong terminology accounted for 11.5–12.5% of identified errors.
• No paradigm dominates universally: robustness is phenomenon-dependent — cascades excel on emotion and long-form, SpeechLLMs are more resilient to noise and code switching, accent/dialect performance is primarily encoder-driven, and gender bias disparity and named entity accuracy are tied to the LLM decoder.

Methodology in Plain English

The authors defined nine categories of speech phenomena known to challenge translation systems and matched each category to one or more existing benchmarks, creating a new noise benchmark (NoisyFLEURS) where a suitable one did not exist. They then assembled 22 systems under a 32-billion-parameter size cap for reproducibility: four speech foundation models used directly, twelve cascades combining those foundation models with three LLMs, and six SpeechLLMs. All models were run with default decoding settings to reflect out-of-the-box performance, with the single exception of Spire, which required beam search (beam size 5) because its default outputs were unusable. Because most speech benchmarks lack reference translations and reference-based metrics are known to be unreliable, evaluation relied on two quality estimation metrics, xCOMET_S^QE and MetricX_S^QE, modified to penalize off-target outputs, with the maximal penalty applied to any translation identified as being in the wrong language. Beyond raw scores, the authors defined a unified "gap" formula measuring relative performance differences between contrasting subsets of the same benchmark (male vs. female speakers, standard vs. non-standard varieties, fluent vs. disfluent, clean vs. noisy, short vs. long), plus case-sensitive named entity and terminology accuracy. A small human evaluation on CoVoST2 with five native speakers, MQM and ESA protocols, and 60 items per language pair checked the reliability of the automatic metrics and characterized error types.

Why This Matters

This is the first study to systematically test, rather than assume, whether putting speech inside an LLM improves translation — a question that shapes how future multilingual speech systems are built. It shows that the answer depends on the phenomenon: no single architecture wins everywhere, and the bottleneck often lies in the language module rather than the speech encoder or the presence of an end-to-end pipeline.

Real-world applications:

• Live and broadcast translation, where the paper's long-form and named-entity results directly inform whether to use cascades (near-zero length degradation) or SpeechLLMs.
• Noisy environments such as call centers, field recordings, or public venues, where SpeechLLMs proved more resilient because they avoid propagating ASR hallucinations through a text stage.
• Multilingual and code-switched communication, relevant for spontaneous bilingual speech, where cascaded Whisper and Voxtral performed best.
• Speech translation for under-served accents and dialects, where the results show that strong standard-variety performance does not transfer reliably and motivate accent-aware training.

Industry relevance: the released suite, the new NoisyFLEURS benchmark, and the annotated human evaluation data give practitioners reusable evaluation infrastructure, and the conclusions favor architectures that integrate an LLM somewhere in the pipeline, which is directly actionable for teams choosing between cascade and SpeechLLM deployments under latency, cost, and robustness constraints.

Future Directions

1. Closing the long-form gap in SpeechLLMs: Voxtral's design of re-concatenating chunk representations before the LLM is highlighted as the reason it handles long context, raising the question of whether other SpeechLLMs can adopt similar strategies.
2. Reducing decoder-driven gender and named entity bias, since WinoST gaps and terminology accuracy traced back to the LLM rather than the speech encoder — testing whether translation-specialized LLMs like Tower+ can be more widely adopted.
3. Accent- and dialect-aware training, given that standard-variety quality does not transfer and that the paper calls for more diverse training strategies.
4. Extending the suite's scope: the evaluation is capped at models under 32 billion parameters and focuses on translation among a specific set of languages; whether the findings generalize to larger models, additional language pairs, and further phenomena is left open.

Target Audience

Researchers and engineers working on speech translation, speech-language model integration, or multilingual LLM systems; evaluation and benchmarking specialists interested in robustness across acoustic and linguistic conditions; and practitioners deciding between cascade and SpeechLLM architectures for deployed speech translation products. Readers focused on gender bias, accent robustness, or error analysis in translation will also find the per-phenomenon results and released annotation data directly useful.

Authors’ abstract

As Large Language Models (LLMs) expand beyond text, integrating speech as a native modality has given rise to SpeechLLMs, which directly process spoken language and enable speech-to-text translation (ST) and other downstream tasks, bypassing traditional transcription-based pipelines. Whether this integration improves ST quality over established cascaded architectures, however, remains an open question. We present Hearing to Translate, the first comprehensive test suite rigorously benchmarking 6 state-of-the-art SpeechLLMs against 16 strong direct and cascade systems that couple leading speech foundation models (SFM), with multilingual LLMs. Our analysis spans 16 benchmarks, 13 language pairs, and 9 challenging conditions, including disfluent, noisy, and long-form speech. Across this extensive evaluation, we find that cascaded systems remain the most reliable solution overall, but most recent SpeechLLMs can match or even outperform cascades in various settings while SFMs lag behind both, highlighting that integrating an LLM, either within the model or in a pipeline, is essential for high-quality speech translation.