OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion

AI summary

Overview

• Research area: Multilingual and multimodal machine translation, specifically end-to-end simultaneous speech translation (SimulST), speech-and-image translation, and text-and-image translation.
• Technical level: Advanced. The paper assumes familiarity with speech translation pipelines, multimodal foundation models, LoRA fine-tuning, simultaneous decoding policies (Local Agreement), and translation quality metrics (XCOMET-XL, COMET-22, ChrF, Average Lagging).
• Scope: The paper proposes OmniFusion, a modular architecture that connects a frozen multimodal foundation model (Qwen2.5-Omni-7B) to a translation-specialized LLM (Seed-X-PPO-7B) through a lightweight token-wise gated fusion module, and evaluates it on simultaneous, offline, and visually grounded translation benchmarks.

What This Paper Is About

Open-source translation LLMs are strong multilingual translators but are unimodal, so speech translation typically requires a cascaded ASR-then-MT pipeline that adds latency and cannot directly use visual context. Conversely, pretrained multimodal foundation models (MMFMs) perceive audio and images well but lag behind specialized translation LLMs in multilingual generation quality. OmniFusion bridges the two: it keeps the MMFM frozen, taps its internal hidden states at selected layers, and fuses them into a translation LLM so that a single framework can handle speech-to-text, speech-and-image-to-text, and text-and-image-to-text translation.

Key Contributions

1. A modular fusion architecture (OmniFusion) that connects a frozen MMFM to a translation-specialized LLM, supporting speech-to-text, speech-and-image-to-text, and text-and-image-to-text translation within one framework. Code is released at https://github.com/saikoneru/OmniFusion.
2. Evidence that MMFM layer choice is critical: selected early and middle MMFM layers transfer better to the translation LLM than late layers, and gated fusion is important for effective transfer.
3. Simultaneous and offline translation results: with Local Agreement decoding, OmniFusion improves the SimulST latency–quality trade-off over a matched cascaded SimulST pipeline, while remaining competitive in offline ST and reducing severe (major and critical) XCOMET-XL errors.
4. Competitive visually grounded caption translation on the CoMMuTE benchmark, achieved without training from scratch and while additionally supporting audio input.

Main Findings

• Late MMFM layers transfer poorly. Adjacent-layer cosine similarity computed with Qwen-Omni 2.5 7B on the MCIF test set (en→de) shows a drastic shift at the first and last layers, indicating three distinct representation regions. The authors therefore select the first, middle, and last layers as candidate transfer points.
• Selected three-layer gated fusion beats all-layer fusion and attention fusion. On FLEURS with zero-shot source languages, the gated model over first/mid/last layers achieves ChrF 64.0 / COMET-22 87.12 for en→de, 44.5 / 81.82 for zh→de, 37.2 / 81.95 for ja→ru, 25.5 / 82.74 for fr→ko, and 12.5 / 65.60 for ur→zh — best across all evaluated pairs. Using all MMFM layers consistently degrades performance.
• OmniFusion outperforms the compared multimodal baselines on FLEURS directions. It exceeds SeamlessM4T v2 and Tower-Spire in all evaluated directions; Tower-Spire only supports English audio.
• Task ablations show OCR grounding matters. On MCIF, removing OCR gives the weakest result (87.12 en→de, 82.42 en→it, 79.75 en→zh); removing both OCR and self-cascading yields 89.19 / 84.35 / 82.14; training on all tasks yields 89.90 / 86.79 / 82.03; with self-cascade inference, the full model gives 90.05 / 87.20 / 82.47 — the best across all three pairs.
• SimulST gains hold under matched decoding. Evaluated with chunk sizes [1, 1.5, 2, 2.5, 3] seconds on MCIF for en→de and en→it, OmniFusion achieves better latency and quality than the fine-tuned cascade even when Local Agreement is applied to ASR. Image input gives further quality gains. OmniFusion achieves substantially higher XCOMET-XL scores than SeamlessM4T Large-v2 and StreamUni-Phi4 in the evaluated settings, though StreamUni-Phi4 obtains lower computation-aware latency because it is designed for streaming generation and reuses cached computation.
• The strongest cascade still wins on aggregate XCOMET-XL in offline ST, but OmniFusion cuts severe errors. Table 3 average over en→zh, en→de, en→it: best cascade (LoRA FT, with image) reaches 86.59 XCOMET-XL with 2024.3 total errors and 55.7 critical errors; OmniFusion with gated fusion, image input, and self-cascade inference reaches 86.57 with 2005.3 total errors and 55.3 critical errors. Directly LoRA fine-tuning the MMFM for speech translation performs substantially worse (82.30).
• Image context helps OmniFusion more than the cascade. The cascade shows only marginal gains from images (85.88 → 85.94 without fine-tuning), while OmniFusion with gated fusion improves from 83.98 (no image) to 86.24 (with image).
• Fusion strategy is task-dependent on CoMMuTE. Mid fusion slightly outperforms gated fusion (e.g., 87.49 vs. 87.58 for en→cs is mixed, but 85.69 vs. 84.62 for en→ru and 85.15 vs. 84.34 for en→zh favor mid fusion in some directions), indicating that middle-layer fusion suffices when images mainly disambiguate text, whereas gated fusion suits speech as the primary input.
• The learned gate adapts to modality. The gate consistently assigns low weight to the last MMFM layer; speech inputs place more weight on early representations, while image-only and speech+image inputs rely more on middle-layer representations.
• Forcing a single layer breaks the system. On MCIF en→de, forcing the first layer degrades quality (55.22 speech-only, 63.02 speech+image) but does not collapse; forcing the middle layer produces hallucinated transcripts and ASR-like outputs; forcing the last layer makes the model emit only the eos token. The unforced model scores 89.90 and 90.05.
• Manual disambiguation check is modest. In a manual annotation of the first 50 English-to-German CoMMuTE source sentences, each paired with two images (150 examples), OmniFusion correctly disambiguated 18 examples versus 23 for TowerVision; a text-only baseline would score 0.
• Training-time detail: LoRA on the MMFM degrades performance due to catastrophic forgetting, so the MMFM is frozen and only the translation LLM is fine-tuned.

Methodology in Plain English

The researchers avoid training a multimodal translation model from scratch. Instead, they take two existing 7B models: a multimodal foundation model that already understands audio and images, and a translation-specialized LLM that is good at multilingual generation. The multimodal model stays frozen and acts as a representation source.

To decide where to tap the multimodal model, they compute cosine similarity between adjacent layers of its hidden states as a diagnostic. The similarity shifts drastically at the first and last layers, suggesting three distinct representation regions, so they pick the first, middle, and last layers rather than using every layer. Using all layers would triple the multimodal sequence length and may add noise.

For each multimodal token, they concatenate the three chosen layer vectors and pass the concatenation through a small bias-free linear layer that produces three logits. A softmax turns these into per-layer importance weights, and the layer vectors are combined as a weighted sum. A three-layer MLP with GELU then projects the fused vector into the translation LLM's embedding dimension, and the resulting vectors are prepended to the source token embeddings as extra input tokens.

Training uses three tasks (speech translation, speech–image translation, text–image translation) plus self-cascading, where the model first generates an ASR transcript or OCR text before producing the translation. Self-cascading is applied stochastically during training with modality-specific prompts. The MMFM is frozen; the translation LLM is fine-tuned with LoRA. Validation uses CoVoST dev data, selecting the checkpoint with the lowest loss after 20k steps.

For simultaneous translation, the offline-trained model is adapted at inference time using Local Agreement decoding: audio arrives in fixed-size chunks, the model generates a new hypothesis at each step, the longest common prefix between consecutive hypotheses is committed, and committed tokens are force-decoded subsequently. The cascade baseline applies the same policy in two stages, first to ASR hypotheses and then to MT conditioned on the agreed ASR prefix. Training data comes from Europarl-ST, CoVoST, M3AV (with Paddle-OCR predictions), an augmented M3AV-Aug built by translating ASR transcripts with Seed-X-PPO-7B, and Multi30k for text–image translation.

Why This Matters

• Impact on research: The paper reframes speech translation, speech–image translation, and text–image translation as one unified multimodal translation problem rather than three separate tasks. It shows that the internal cross-modal representations of frozen MMFMs are an underused resource, and that a lightweight interface plus multitask objectives can extract them. It also provides a diagnostic (adjacent-layer cosine similarity) and a layer-forcing analysis that other researchers can reuse.
• Real-world applications:
  • Live simultaneous interpretation of talks and meetings, where lower latency matters and slides or visuals are available as context.
  • Subtitle and caption generation for video, including cases where an image or frame disambiguates the correct translation of an ambiguous word.
  • Presentation and lecture translation where audio is paired with slides containing OCR-readable keywords and names.
  • Translation of image-captioned content where visual grounding determines the correct lexical choice.
• Industry relevance: The approach reuses existing pretrained models rather than requiring from-scratch multimodal training, which lowers compute and data requirements. The finding that late MMFM layers transfer poorly, and that gated fusion over a few selected layers beats attention over all layers, gives concrete architectural guidance for teams building speech translation systems on top of open MMFMs. The removal of the ASR stabilization delay in the cascade is directly relevant to latency-sensitive deployment.

Future Directions

• Streaming-specific training and caching. OmniFusion is not trained with a streaming objective and does not reap the computation-aware latency benefits that StreamUni-Phi4 does; the authors call adding streaming-specific training and caching support a promising direction.
• Broader source-language coverage. Training data is primarily English-centric, and while zero-shot experiments on non-English source languages are encouraging, broader source-language training and evaluation are needed.
• Instruction-following behavior. The model is trained with fixed task-specific prompts across the three supported settings and does not exhibit general instruction-following; whether the fusion strategy can support more flexible, instruction-driven multimodal behavior remains unclear.
• Larger-scale human evaluation. The manual CoMMuTE analysis is small-scale (50 source sentences, single annotator) and the authors state that broader human evaluation is needed to fully assess visually grounded translation behavior.

Target Audience

Researchers and engineers working on multimodal machine translation, simultaneous speech translation, and speech–language models; practitioners who want to build speech or image-aware translation systems on top of existing open-source LLMs and MMFMs; and readers interested in representation-level questions about which layers of a multimodal foundation model carry transferable cross-modal information. Readers with a background in translation systems, LoRA fine-tuning, and evaluation metrics such as COMET and Average Lagging will get the most out of the experimental sections.

Authors’ abstract

There has been significant progress in open-source text-only translation large language models (LLMs) with better language coverage and quality. However, these models can be only used in cascaded pipelines for speech translation (ST), performing automatic speech recognition first followed by translation. This introduces additional latency, which is particularly critical in simultaneous ST (SimulST), and prevents the model from exploiting multimodal context, such as images, which can aid disambiguation. Pretrained multimodal foundation models (MMFMs) already possess strong perception and reasoning capabilities across multiple modalities, but generally lack the multilingual coverage and specialized translation performance of dedicated translation LLMs. To build an effective multimodal translation system, we propose an end-to-end approach that fuses MMFMs with translation LLMs. We introduce a novel fusion strategy that connects hidden states from multiple layers of a pretrained MMFM to a translation LLM, enabling joint end-to-end training. The resulting model, OmniFusion, built on Omni 2.5-7B as the MMFM and SeedX PPO-7B as the translation LLM, can perform speech-to-text, speech-and-image-to-text, and text-and-image-to-text translation. Experiments demonstrate that OmniFusion effectively leverages both audio and visual inputs, achieves a 1-second latency reduction in SimulST compared to cascaded pipelines and also improves the overall translation quality\footnote{Code is available at https://github.com/saikoneru/OmniFusion}.