CRESiST: Self-Enhancing Explorative Policy Learning for Simultaneous Speech Translation

AI summary

Overview

Research area: Simultaneous Speech Translation (SimulST) — machine translation of a live audio stream while it is still being spoken — with a focus on LLM-based streaming policies and reinforcement-style policy learning.

Technical level: Advanced. The paper assumes familiarity with encoder-decoder speech translation, KV caching, RNN-Transducers, Actor-Critic training, and reward-based distillation.

Scope (one sentence): The paper proposes CRESiST, an interleaved read-write streaming framework built on an omni-modal LLM, trained with a self-enhancing Actor-Critic loop that discovers when to read audio and when to write translation without any external word alignment, and evaluates it on sentence-level (FLEURS, CoVoST2) and document-level (MuST-C, ACL 60/60) benchmarks.

What This Paper Is About

Most current LLM-based simultaneous speech translation systems work by repeatedly re-feeding the whole audio prefix plus new audio, which invalidates the Key-Value (KV) cache and forces the system to depend on external segmentation such as Voice Activity Detection. This causes quadratic computation, breaks cross-sentence context, and does not naturally handle infinite audio streams. CRESiST instead interleaves speech chunks and translated text tokens into one continuous sequence so the KV cache is fully reused, and it learns the read/write timing policy autonomously from offline parallel data rather than from rigid forced alignments.

Key Contributions

1. Unsupervised streaming policy inside an interleaved framework. Trained purely on offline parallel speech-text data, the model autonomously transitions to an online interleaved read-write paradigm, achieving full KV cache reuse and segmentation-free, infinite-length translation.
2. Fine-grained Actor-Critic policy discovery. A self-enhancement pipeline operating at a 400ms acoustic chunk size, in which an auxiliary RNN-T Actor samples read/write trajectories and an LLM-based Critic scores them with intrinsic semantic and latency rewards, transferring the preferred policy back to the Actor via reverse distillation — with no static forced alignments.
3. Multilingual generalization. A single unified CRESiST model translates Chinese and English inputs into six target languages (Chinese, English, German, French, Spanish, Italian), rather than requiring language-pair-specific models.
4. State-of-the-art quality-latency trade-offs. Experiments on sentence-level and unbounded document-level benchmarks show the model outperforming strong baselines including InfiniSST, establishing new quality-latency Pareto frontiers.

Main Findings

• New Pareto frontiers at sentence level: On CoVoST2 and FLEURS, sweeping the decision threshold (τ ∈ [0.3, 0.7]) shows CRESiST dominating both Seamless Streaming and AlignATT across all latency regimes, and degrading gracefully at extremely low latency (e.g., LAAL < 1.5).
• Beats a commercial system on FLEURS zh↔en: The streaming model strictly outperforms the Seed LiveInterpret 2.0 API on FLEURS Chinese-English bidirectional translation, with higher translation quality at comparable or superior latency.
• Segmentation-free streaming is nearly as good as oracle segmentation: In Table 1, unsegmented streaming trails oracle-segmented streaming by only 1–2 COMET points — e.g., MuST-C en-de 85.96 COMET / 0.76 LongYAAL without segmentation versus 87.75 / 0.80 with oracle segmentation.
• Strong document-level numbers without segmentation: CRESiST (τ=0.5) reaches 73.73 COMET / 1.14 LongYAAL (MuST-C en-zh), 85.96 / 0.76 (MuST-C en-de), 79.45 / 1.49 (ACL 60/60 en-zh), and 85.59 / 1.08 (ACL 60/60 en-de), versus InfiniSST at 70.66 / 1.88, 78.74 / 1.46, 68.62 / 1.80, and 63.52 / 1.46 respectively.
• Incremental offline-style baselines are unstable: StreamATT on ACL 60/60 en-de plummets from 80.83 COMET under oracle segmentation to 70.61 without segmentation, with latency inflation — a failure mode CRESiST avoids.
• InfiniSST behaves paradoxically: Although designed for infinite streams, InfiniSST degrades when evaluated sentence-by-sentence with oracle boundaries (e.g., ACL 60/60 en-zh falls from 68.62 unsegmented to 59.71 with oracle segmentation).
• Joint self-enhancement matters most: Direct SFT on RNN-T aligned data (similar to InfiniSST) gives the poorest trade-off and struggles to run at 400ms granularity; pipeline training without joint optimization improves it; the full CRESiST framework gives the best frontier, with the largest latency reduction on Chinese-English.
• Latency penalty weight α controls the trade-off: Large α aggressively compresses latency at the cost of translation quality, while overly small α fails to constrain the policy and produces latency spikes at specific decision thresholds.
• Axial attention blocks are essential in the Actor: A conventional RNN-T reaches PPL 22.3 and Average Lagging 8.03s, while the proposed axial-block model reaches PPL 6.03 and Average Lagging 1.81s (both trained for 600k steps).
• Complementary to RL post-training: GRPO post-training further improves CRESiST, whereas the same recipe cannot close the gap from a direct-SFT base (Appendix D.1).
• Domain cost: CRESiST shows an offline performance drop on the ACL60/60 dev set relative to the vanilla Qwen2.5-omni model, indicating domain distribution discrepancy and catastrophic forgetting.

Methodology in Plain English

The system turns streaming translation into a sequence of simple decisions: Read (take in the next 400ms of audio) or Write (emit the next text token). Because these two kinds of tokens are placed in one shared sequence, the model never has to recompute its cache, and it can keep going on an arbitrarily long stream without anyone deciding where sentences begin and end.

Two models share the work. A small, fast Actor — an RNN-Transducer whose audio encoder starts from Whisper-medium and whose text side starts from Qwen3-0.6B, augmented with 2D axial attention — guesses, at each lattice position, whether to read or write, and its guesses are used to build the interleaved training sequence. A large Critic — the Qwen2.5-omni-7B backbone with a two-layer MLP "Router" and a "Refiner" — looks at that interleaved sequence and scores it two ways: a consistency reward (how well the model's predictions match the ground-truth text, i.e., negative log-likelihood) and a latency reward (the expected number of consecutive writes it will emit, computed from the Router's write probabilities). The Critic's Router output is thresholded at τ to produce a preferred read/write target at each step, and a binary cross-entropy "reverse distillation" loss pushes the Actor to match those targets, while the standard RNN-T loss stays as a small auxiliary term.

Training runs in three stages: (1) pretrain the RNN-T Actor with ordinary RNN-T loss; (2) freeze the Actor, build interleaved sequences from its samples, and adapt the Speech LLM, Router and Refiner to that streaming input; (3) unfreeze everything and run joint self-enhancement, with K = 8 Critic steps per Actor step. The LLM is fine-tuned with LoRA (rank r = 16), and offline ST and MT tasks are trained jointly to preserve offline ability. Training data comes from Emilia, WenetSpeech and GigaSpeech, with pseudo-labels generated by Qwen3-235B-A22B into six languages; WenetSpeech and GigaSpeech clips are concatenated into segments of up to 30 seconds. The paper reports a local convergence analysis for this loop in an appendix (the supplied text truncates at the theorem statement).

Why This Matters

Impact on research. The paper reframes SimulST as a policy-learning problem decoupled from content generation, removing the dependence on forced alignments (e.g., MFA) and on external segmenters for unbounded audio. It offers an alternative to prefix-incremental generation whose KV-cache invalidation causes quadratic cost, and it shows a single multilingual model can cover many translation directions rather than one model per language pair.

Real-world applications:

• Live interpretation of conferences, lectures and meetings where speech arrives continuously and subtitles must appear with minimal delay.
• Multilingual live broadcast and streaming-media captioning, where speech runs for hours without sentence boundaries.
• Real-time multilingual customer support and telephony, where a single model must serve Chinese, English, German, French, Spanish and Italian.
• Assistive transcription/translation on long recordings where segmentation tools are unavailable or unreliable.

Industry relevance. Removing VAD and enabling full KV-cache reuse directly attacks inference cost and complexity, which matters for deploying omni-modal LLMs in real-time products. The comparison against a commercial API (Seed LiveInterpret 2.0) on FLEURS zh↔en positions the work as a competitive systems contribution, and the authors' affiliations (Alibaba Token Hub, Alibaba Group; Shanghai Jiao Tong University; University of Macau; University College London) reflect an industrial research setting.

Future Directions

• Better data and domain coverage. The authors state they did not perform rigorous data engineering; absolute quality is bounded by backbone capacity and offline SFT data, and they observed an offline drop on ACL60/60 dev versus vanilla Qwen2.5-omni.
• Document-level latency control. The decision threshold τ tunes sentence-level trade-offs well but has only marginal effect in continuous document-level streaming; the authors suggest dynamic penalty mechanisms such as a context-aware latency multiplier (as used in InfiniSST) instead of a static probability threshold.
• Combining with RL post-training. GRPO post-training improved CRESiST but could not rescue a direct-SFT base, raising the question of how to layer policy-optimization methods more systematically.
• Extending scale and languages. The current model handles Chinese/English inputs into six target languages over 10 directions; scaling the multilingual and multimodal scope is an open direction.

Target Audience

Researchers and engineers working on simultaneous speech translation, streaming speech LLMs, and speech-text interleaved modeling; practitioners building real-time multilingual captioning or interpretation products on omni-modal LLM backbones; and readers interested in Actor-Critic or reverse-distillation methods for learning non-differentiable output policies (here, read/write timing) without external alignment supervision. Beginners will find the setup accessible but the reward formulation and lattice sampling require intermediate background.

Authors’ abstract

Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from redundant Key-Value cache recomputation and necessitates external Voice Activity Detection to process unbounded streams. To overcome these bottlenecks, we propose an end-to-end interleaved read-write framework built upon an omni-modal LLM. By interleaving speech chunks and text tokens, our architecture achieves full KV cache reuse, natively supporting segmentation-free, infinite-length inference. Furthermore, to learn optimal streaming policies without rigid offline alignments, we introduce a novel Actor-Critic self-enhancement pipeline. An RNN-T Actor dynamically samples read-write trajectories, while an LLM-based Critic evaluates these paths via intrinsic semantic and latency rewards, distilling the optimal policy back to the Actor. Extensive experiments on sentence-level (FLEURS, CoVoST2) and document-level (MuST-C, ACL 60/60) benchmarks demonstrate that our model establishes new state-of-the-art quality-latency Pareto frontiers, substantially outperforming existing cascaded and static-alignment baselines.