Simultaneous Speech-to-Speech Translation Without Aligned Data

AI summary

Overview

Research area: Simultaneous (real-time) speech-to-speech and speech-to-text translation, spanning spoken language processing, neural audio codec modeling, and reinforcement learning for sequence generation. arXiv:2602.11072v2 [cs.CL], by Tom Labiausse, Romain Fabre, Yannick Estève, Alexandre Défossez and Neil Zeghidour (Kyutai, LIA / University of Avignon, and Gradium).

Technical level: Advanced.

Scope in one sentence: The paper introduces Hibiki-Zero, a multilingual-to-English simultaneous speech translation model that is trained without any word-level aligned translation data, using sentence-level alignments plus a GRPO-based reinforcement learning stage driven only by BLEU-derived process rewards.

What This Paper Is About

Simultaneous speech translation means listening to speech in one language and producing translated speech in another at the same time, which forces a model to decide when to listen and when to speak. The standard recipe needs word-level aligned data (knowing which target word corresponds to which source word and when), which barely exists for human interpretation and must be synthesized with language-specific heuristics. Hibiki-Zero removes that requirement: it learns from loosely aligned, sentence-level data and then uses reinforcement learning to tighten its translation timing without losing translation quality.

Key Contributions

1. A training pipeline that needs no word-level alignments. Hibiki-Zero trains on sentence-level aligned pairs, which the authors describe as much easier to construct and largely language-independent, replacing hand-crafted alignment heuristics.
2. A reinforcement learning procedure that optimizes quality and latency jointly using BLEU only. Using GRPO adapted to a multistream architecture, the method computes process rewards at intermediate instants and combines them into a single reward, avoiding the multiple weighted metrics and extra hyperparameters used by prior RL approaches.
3. State-of-the-art results across four X-to-English directions. The model reports top performance in translation accuracy, latency, voice transfer and naturalness on short-form (Europarl-ST) and long-form (Audio-NTREX-4L) test data.
4. A cheap new-language adaptation recipe plus released resources. An Italian-to-English model is built from less than 1000 hours (the adaptation experiment uses 850 hours) of finetuning data, and the authors release examples, model weights, inference code, and a 45h multilingual speech translation evaluation benchmark.

Note: the authors disclose a conflict of interest, as N.Z. and A.D. are employed by Gradium, which develops one of the three TTS models used to create the long-form evaluation dataset.

Main Findings

• Short-form quality and latency: On Europarl-ST, Hibiki-Zero reports BLEU scores of 35.0/34.6 (French), 33.8/33.9 (Spanish), 33.6/33.6 (Portuguese) and 28.7/28.6 (German) across the two BLEU columns, with End Offset of 2.1, 2.3, 2.4 and 1.9 seconds and LAAL

Authors’ abstract

Simultaneous speech translation requires translating source speech into a target language in real-time while handling non-monotonic word dependencies. Traditional approaches rely on supervised training with word-level aligned data, which is difficult to collect at scale and thus depends on synthetic alignments using language-specific heuristics that are suboptimal. We propose Hibiki-Zero, which eliminates the need for word-level alignments entirely. This fundamentally simplifies the training pipeline and enables seamless scaling to diverse languages with varying grammatical structures, removing the bottleneck of designing language-specific alignment heuristics. We first train on sentence-level aligned data to learn speech translation at high latency, then apply a novel reinforcement learning strategy using GRPO to optimize latency while preserving translation quality. Hibiki-Zero achieves state-of-the-art performance in translation accuracy, latency, voice transfer, and naturalness across five X-to-English tasks. Moreover, we demonstrate that our model can be adapted to support a new input language with less than 1000h of speech. We provide examples, model weights, inference code and we release a benchmark containing 45h of multilingual data for speech translation evaluation.