Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment

AI summary

Overview

Research area: end-to-end code-switch (CS) speech translation (ST) using large language models, specifically the design of a Mixture-of-Experts speech projector and a multi-stage training scheme.

Technical level: Intermediate. The paper is readable for someone familiar with speech translation basics, but the MoE routing losses and four-stage training schedule require some background in LLM-based speech pipelines.

Scope: The paper proposes and evaluates a single LLM-based CS speech translation system that routes speech tokens to language-specific expert groups and is trained through four progressive stages, tested on the Fisher and NTUML2021 benchmarks.

What This Paper Is About

Code-switch speech translation means translating spoken audio in which the speaker alternates between two (or more) languages within a single utterance into text in a target language. This is hard for two reasons the authors identify: semantic modeling becomes complex when languages alternate, and high-quality CS speech translation data is scarce and expensive to annotate. The paper's goal is to improve CS speech translation without relying on costly manual annotation, by building language-aware expert routing into the speech-to-LLM connector and by reusing readily available ASR and monolingual ST data to train the model in stages.

Key Contributions

1. The authors state that this is the first exploration of LLMs for end-to-end CS speech translation, and they provide baseline results to support future work.
2. A Mixture-of-Experts speech projector in which experts are organized into language-aware groups, described as enabling fine-grained modeling of speech features from different languages so the model can capture semantic distinctions in CS speech.
3. A four-stage training paradigm that moves from ASR pretraining, to MoE construction and aggregation, to monolingual ST, and finally to CS speech translation, using a transition loss to smooth each domain shift. Two auxiliary losses — a language-specific loss and an intra-group load balancing loss — guide routing at the group level and within groups.
4. Empirical evaluation on monolingual and CS speech translation datasets (Fisher, NTUML2021), plus ablation studies and expert-level analyses covering routing behavior and per-expert semantic performance.

Main Findings

• Semantic space gap exists between languages. A preliminary study on Common Voice English, Spanish and Chinese ASR data, visualizing mean-pooled Whisper encoder features with t-SNE and a bivariate KDE plot, showed samples from the same language clustering tightly and different languages being clearly separated.

• Per-language projectors beat a shared projector. Fine-tuning only the projector on monolingual ASR data, the expert-projector setting scored WER/CER of 7.05 (En), 4.52 (Es) and 13.21 (zh-CN) on the validation set, versus 7.07, 6.01 and 14.26 for the shared projector.

• Overall gains over SeamlessM4T. Averaged across test sets, the approach reports improvements of 0.86 BLEU and 0.93 COMET over SeamlessM4T, with maximum improvements of 1.49 BLEU and 1.41 COMET.

• Best CS test scores. On CS data, the model reached 37.51 BLEU / 77.54 COMET on Fisher and 39.52 BLEU / 81.33 COMET on NTUML2021. Comparable baselines on the same CS settings: SeamlessM4T 36.89/76.13 (Fisher) and 38.03/79.93 (NTUML2021); Whisper 31.56/72.63 and 29.12/74.90; Whisper+LLaMA 29.67/70.99 and 34.85/79.08; LLaST 33.78/75.35 and 32.53/78.92.

• Competitive on monolingual ST too. Under the monolingual setting the model scored 35.87 BLEU / 80.14 COMET on Fisher and 36.27 BLEU / 81.65 COMET on NTUML2021, with Whisper+LLaMA obtaining the best COMET on the NTUML2021 monolingual setting (81.68).

• Hyperparameters chosen by tuning. Fixing top-k at 3 and varying the number of experts per group n over {3, 5, 7, 9, 11}, BLEU/COMET on the NTUML2021 CS validation set were 38.02/80.95 (3), 38.40/80.62 (5), 38.55/81.02 (7), 38.35/80.99 (9), 37.74/80.77 (11). With n fixed at 7 and varying top-k over {1, 3, 5, 7}, scores were 27.76/76.38 (top-1), 38.55/81.02 (top-3), 38.53/81.01 (top-5), 38.90/81.14 (top-7). The final configuration is n = 7 and top-7.

• Every component contributes. On the NTUML2021 CS test set, the full model scores 39.52 BLEU / 81.33 COMET. Removing the MoE speech projector (replacing it with a single MLP projector and dropping Stage 2 and both auxiliary losses) drops to 36.85/79.95; removing the language-specific loss gives 39.05/80.88; removing the balancing loss gives 38.86/81.03; removing the transition loss gives 39.21/81.24; substituting the conventional Fedus et al. balancing loss gives 38.68/80.79; removing Stage 1 gives 38.14/80.31; removing Stage 3 gives 37.74/80.02.

• Routing mostly lands in the correct language group. Using monolingual Common Voice speech, the share of top-k experts belonging to the matching language group was 97.62 percent (En) and 95.66 percent (Es) for top-1, and 96.72 percent / 95.76 percent for top-7 on the Fisher-trained model. For the NTUML2021-trained model it was 99.99 percent (En) and 85.21 percent (zh-CN) for top-1, and 99.98 percent / 81.03 percent for top-7. Without the language-specific loss these proportions fall sharply, to 54.42 percent / 51.16 percent / 45.72 percent / 69.28 percent (top-1) and 51.06 percent / 55.00 percent / 45.66 percent / 69.80 percent (top-7).

• The balancing loss stabilizes rank assignment. Mean and variance of the correct expert rank within the matching group were 4.25/0.69 (Fisher-trained) and 4.77/0.88 (NTUML2021-trained); without the balancing loss these became 4.14/1.05 and 4.67/1.21.

• Experts are complementary. Disabling the router and forcing all tokens to a single expert index, the routed model performed best across datasets, while fixed-index experts did not degrade substantially when used alone, which the authors read as evidence that experts capture complementary semantic aspects.

• Competitive against recent Audio-LLMs in zero-shot comparison. Against SALMONN-13B, Qwen2-Audio-7B-Instruct, Qwen3-Omni-30B-A3B-Instruct, Voxtral-Mini-3B-2507 and Voxtral-Small-24B-2507, the model reports the best BLEU in all settings and competitive COMET, underperforming in COMET in two cases against much larger models (for example Qwen3-Omni-30B-A3B-Instruct reaches 82.42 COMET on NTUML2021 monolingual).

Methodology in Plain English

The system has three parts: a speech encoder (Whisper-large-v3 encoder) that turns audio into a sequence of features, a projector that maps those features into the LLM's representation space, and the LLM itself (Llama-3-8B-Instruct) which generates the translation. The prompt text embeddings and the projected speech embeddings are concatenated and fed to the LLM.

The distinctive part is the projector. Instead of one shared projection network, the authors first train a separate projector for each language involved using that language's ASR data. Those per-language projectors then become the seed weights for separate groups of experts, and a randomly initialized router is inserted so the overall projector becomes a Mixture-of-Experts layer stack (three layers, ReLU activation, n experts per group, total experts N = n × m for m languages). Because the router starts random, experts within a group that share identical initial weights gradually specialize during training.

Two auxiliary losses steer the router. The language-specific loss penalizes sending a token to an expert group belonging to a different language, using the language labels available in the monolingual ASR data. The intra-group load balancing loss encourages tokens of a given language to be spread across experts inside the correct group rather than collapsing onto a few. Cross-entropy on the target text is the main objective.

Training proceeds in four stages. Stage 1 pretrains per-language projectors on ASR data to align speech and text. Stage 2 aggregates the projectors into the MoE projector and continues on the same ASR data with the two auxiliary losses. Stage 3 shifts toward translation by mixing ASR data with monolingual ST data, using a transition loss whose weight on the ST term increases linearly with the batch index. Stage 4 mixes monolingual ST with CS speech translation data the same way; because CS speech has no token-level language labels, the two auxiliary routing losses are dropped at this final stage. The LLM and the speech encoder are tuned with LoRA (alpha 256 for the LLM, 64 for the encoder) while the MoE projector is fully fine-tuned. Each stage runs for one epoch with AdamW at learning rate 2 × 10⁻⁴, per-device batch size 8 with 4 gradient accumulation steps (effective batch 32), and inference uses beam search with beam size 5, all on NVIDIA A100 GPUs, with a full training cycle taking about 2 days on 4 GPUs.

Why This Matters

For research, the paper reframes the CS speech translation problem as one of explicit language-aware routing rather than hoping an LLM resolves language mixing implicitly, and it contributes a training recipe that avoids expensive CS annotation by reusing ASR and monolingual ST corpora. The preliminary t-SNE/KDE evidence and the routing-proportion analysis provide concrete, measurable handles on a phenomenon that is usually discussed only qualitatively.

Real-world applications:

• Multilingual contact centers and customer support lines, where agents and customers switch languages mid-sentence (the Fisher data used here is telephone conversation speech).
• Lecture and classroom translation in bilingual education, matching the NTUML2021 setting of Chinese-English lecture speech translated to English.
• Meeting transcription and translation in multinational workplaces where participants mix languages.
• Media localization: subtitling, dubbing and captioning of broadcasts, interviews or podcasts that contain code-switched speech.

Industry relevance: the approach plugs into an existing, widely used architecture (pretrained speech encoder plus LLM plus projector), so it is a component-level and training-schedule change rather than a new model family. The reported gains are strongest on the hardest CS conditions while remaining competitive on monolingual ST, and the paper's comparison against recent Audio-LLMs (including Qwen3-Omni-30B-A3B-Instruct and Voxtral-Small-24B-2507) frames it as a viable small-model option. Code is released at https://github.com/XMUDeepLIT/CSST-SSA.

Future Directions

• Extend the method to a wider range of CS language pairs beyond the English-Spanish and Chinese-English settings studied here.
• Investigate more adaptive expert selection mechanisms, replacing or augmenting the current router.
• Generalize the framework to related CS tasks such as ASR and MT to test robustness and generality.
• Address the language imbalance visible in routing: Chinese tokens were routed to the correct group at a lower rate than English or Spanish tokens, which the authors attribute to data imbalance across languages.
• Handle the fact that oracle language labels are unavailable for CS speech, since the language-specific loss is dropped in Stage 4 and the routing analysis had to be run on monolingual ASR data instead.

Target Audience

Researchers and engineers working on speech translation, multilingual and code-switched speech processing, and LLM-based multimodal systems, particularly those interested in Mixture-of-Experts architectures or in training pipelines that substitute abundant ASR and monolingual ST data for scarce human-annotated CS parallel data. It is also useful

Authors’ abstract

Code-switching (CS) speech translation (ST) aims to translate speech that alternates between multiple languages into a target language text, posing significant challenges due to the complexity of semantic modeling and the scarcity of CS data. Previous studies mainly rely on the models themselves to implicitly learn semantic representations and resort to costly manual annotations. To mitigate these limitations, we propose enhancing Large Language Models (LLMs) with a Mixture-of-Experts (MoE) speech projector composed of language expert groups, where each group specializes in the semantic space of a specific language for fine-grained speech feature modeling. A language-specific loss and an intra-group load balancing loss are jointly introduced to guide efficient token routing across and within expert groups. Furthermore, we introduce a multi-stage training paradigm that utilizes readily available automatic speech recognition (ASR) and monolingual ST data, facilitating speech-text alignment and improving translation performance. To bridge the data gap for smooth domain transfer, a transition loss is employed to improve adaptation to CS scenarios. Extensive experiments on widely used datasets demonstrate the effectiveness and generality of our approach, achieving average improvements of $0.86$ BLEU and $0.93$ COMET over SeamlessM4T, with maximum improvements of $1.49$ BLEU and $1.41$ COMET across different test sets.