Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data

AI summary

Overview

Research area: Natural Language Processing and multimodal AI — specifically omnimodal large models (OLMs) that handle text, images, video, audio and music in a single unified system, plus their generation counterparts (text, speech, images).

Technical level: Advanced. The paper assumes familiarity with Mixture-of-Experts (MoE) routing, rotary position embeddings (RoPE), reinforcement learning for LLMs (GSPO, DPO), diffusion transformers and speech tokenization.

Scope in one sentence: The paper describes the architecture, training recipe, data curation and 85-benchmark evaluation of Uni-MoE-2.0-Omni, an open-source omnimodal model built from the dense Qwen2.5-7B LLM that both understands and generates text, images and speech.

What This Paper Is About

Existing omnimodal systems tend to be biased toward either deep multimodal understanding and reasoning (often with no generative ability) or high-fidelity generation confined to a narrow set of modalities. Scaling dense transformers to cover dozens of task types and modal interactions is computationally prohibitive and unstable on heterogeneous data. The paper's goal is a single, computationally efficient, language-centric architecture that unifies multimodal understanding, reasoning and generation — without the usual trade-off between comprehension and generation quality.

Key Contributions

1. Dynamic-Capacity MoE. A new MoE framework that splits experts into three explicit roles — null experts (which let tokens skip computation), modality-specific routed experts (audio, visual, textual) that store modality knowledge, and compact shared experts only 1/8 the size of routed experts for cross-modal knowledge transfer. It adds routing gradient estimation (migrated from Grin-MoE) to make the Top-K expert selection differentiable, and Top-P-based dynamic capacity so each token can activate a variable number of routed experts (threshold P = 0.7).

2. Omni-Modality 3D RoPE. A position-encoding mechanism in the self-attention layers that decomposes rotary embeddings into temporal, height and width components, aligning speech, image, text and video tokens in a shared spatio-temporal frame — for audio, temporal IDs follow absolute time with 20 tokens as the minimum unit (3 seconds).

3. Progressive training recipe with iterative reinforcement. A staged pipeline — cross-modal alignment, expert warm-up, MoE fine-tuning and reinforcement learning, then generative training — plus an iterative GSPO-DPO method that reuses online RL rollouts, uses "LLM as a Judge" to pick positive examples, and calls on Gemini-2.5-Flash to generate reasoning demonstrations for samples that scored zero accuracy.

4. Hybrid language-centric training and open release. Multimodal understanding and generation tasks (image editing, image generation, speech synthesis) are unified inside a language generation framework. The authors open-source training and inference code, five model checkpoints, and data lists.

Main Findings

• Benchmark scale: Evaluation covers 85 multimodal benchmarks. Against Qwen2.5-Omni (trained on 1.2T tokens), the model surpasses it on over 50 of 76 benchmarks.

• Video understanding and reasoning: Reported as +7% average across 8 benchmarks in the abstract; the introduction states an average of +4% over Ming-Lite-Omni-1.5 on 8 benchmarks.

• Omnimodality comprehension: +7% average over Qwen2.5-Omni across 4 benchmarks, including OmniVideoBench and WorldSense.

• Audiovisual reasoning: +4% average over Qwen2.5-Omni on Speech-Image QA (Reasoning).

• Long-form speech understanding: WER reduced by up to 4.2%; specifically 3.5% lower WER than Qwen2.5-Omni on LibriSpeech-clean/other-long. Speech generation reports 1% lower WER on TinyStories-en.

• Image tasks: +0.5% over Ming-Lite-Omni on GEdit-Bench, and surpassing previous state-of-the-art Qwen-Image and PixWizard on 6 metrics; the model leads in low-level image processing and controllable generation across 5 metrics. The abstract reports this as state-of-the-art or highly competitive against leading OLMs.

• Model scale and efficiency: The MoE-LLM is a 26B-parameter MoE transformer. A routed expert is 5.7B parameters and a shared expert 712M, but the number of activated parameters ranges from a minimum of 1.5B (2 shared experts + 0 routed experts) to a maximum of 18B (2 shared experts + 3 routed experts).

• Module inventory: Audio encoder is Whisper-Large-v3 (637M); vision encoder is SigLIP-So400M (398M); MoE-TTS is a MoE transformer (1.2B total, 0.7B activated); Task-DiT is a dense transformer (1.5B); codec decoder is WavTokenizer-large-600-24k-4096 (442M); VAE decoder is SD-XL (49M).

• Data efficiency: The base model was trained on approximately 75B tokens of open-source multimodal data — much less than the 1.2T tokens reported for Qwen2.5-Omni.

• Not reported: The provided content does not report GPU hardware, training compute hours, the exact size or composition of each data subset, or the number of benchmarks in the full 85 (only the 76-benchmark comparison to Qwen2.5-Omni is quantified). The Appendix data lists and the formal gradient-estimation description in Appendix A.4 are referenced but truncated here.

Methodology in Plain English

The team started from one dense language model (Qwen2.5-7B) rather than building separate models per modality, on the view that language is a structured mediator between modalities and gives training stability plus easy scaling to new scenarios. Inputs are tokenized uniformly: audio is cut into 30-second clips represented by 200 tokens each (20 tokens per 3 seconds) and encoded via Whisper-Large-v3's encoder with a Qformer that uses the Whisper decoder layers and 200 query tokens; images are encoded by SigLIP with a sliding window where each 384×384 patch is tokenized independently; video frames are resized to the encoder's minimum resolution and sampled at one frame per second, then length-adjusted between lower and upper bounds. To keep the modalities aligned, 3D RoPE assigns temporal IDs from absolute time for audio and video, and spatial IDs for image height and width.

Inside the transformer, standard MLP layers are replaced with MoE layers. Every token always uses the small shared experts; a router picks modality-specific routed experts using Top-P sampling so simple tokens use few experts and complex ones use more; null experts let tokens be effectively skipped. Because picking experts is normally non-differentiable, the authors import a straight-through gradient estimator so the router still receives gradients.

Training proceeds in stages: cross-modal pretraining to map each modality into the LLM's language space; a warm-up that trains three dense specialists (speech comprehension, speech generation, visual comprehension); then fine-tuning the MoE mixture (initialized with two speech experts, one visual expert, one null expert plus two shared experts) on mixed data; an annealing phase on balanced data to close capability gaps; and finally reinforcement learning with cold-start initialization, online GSPO and DPO. For generation, the model writes special control tokens: speech tokens specify language and voice (fixed timbres Brain, Jenny, Xiaoxiao, or natural-language descriptions) and drive a MoE-TTS built on Qwen2.5-0.5B that splits long text at punctuation and uses the previous segment as context, producing audio beyond two minutes. Image tokens specify the task and content and drive a Task-Aware Diffusion Transformer whose image generator is frozen from PixWizard, steered through task, image and visual projectors rather than end-to-end fine-tuning, which avoids the interference that plagues jointly fine-tuned models.

Why This Matters

The work is a demonstration that a comparatively small data and compute budget (75B tokens, a 7B base LLM scaled with MoE) can produce an omnimodal model that is competitive with, and on many benchmarks better than, much more heavily trained systems such as Qwen2.5-Omni (1.2T tokens). It also offers a concrete architectural recipe — role-specialized experts, dynamic capacity, unified positional encoding — for the open-research community, which has had far fewer omnimodal artifacts than industrial labs. By treating generation as another language task rather than a separate model, it suggests a path to avoiding the understanding-versus-generation trade-off.

Real-world applications the paper's capabilities map onto:

• Human-computer interfaces and AI assistants that perceive audio, images and video jointly and reply with synthesized speech in a chosen voice.
• Content creation tools, including instruction-guided image generation and editing.
• Accessibility and media processing, such as long-form speech transcription with reduced word error rate and audiovisual (speech-plus-image) reasoning.
• Controllable speech synthesis for dialogue, narration and multi-style voice output over clips longer than two minutes.

Industry relevance: the model, checkpoints and code are fully open-source (CC BY-SA 4.0), lowering the barrier for product teams and academic groups that cannot train trillion-token omnimodal systems; the MoE activation range of 1.5B to 18B parameters also points to inference-cost control, which matters for deployment.

Future Directions

• Formalizing the routing gradient estimation. The paper defers the full mathematical treatment to Appendix A.4, leaving open how generally the straight-through/ODE-based estimator holds across MoE configurations beyond this one.
• Iterative RL scaling. The GSPO-DPO pipeline is described as supporting iterative refinement, and the truncated text points to VIPO-R1 for a comprehensive treatment; how far reasoning improves with more iterations, and whether the Gemini-2.5-Flash teacher remains necessary, is unresolved.
• Balancing the expert mixture. The final configuration uses two speech experts, one visual expert and one null expert; whether more experts, different modality allocations, or additional null experts improve coverage across the 10 cross-modal inputs is untested here.
• Reproducing and extending the data recipe. Only aggregate token counts and Appendix data lists are available in this content, so the composition, licensing and mixture ratios of the 75B-token corpus remain the key variable other groups would need to reconstruct.

Target Audience

Researchers and engineers working on multimodal and omnimodal LLMs, MoE sparse architectures, and unified understanding-plus-generation models will get the most from this paper, particularly those interested in open-source alternatives to industrial omnimodal systems. It is also relevant to practitioners building speech synthesis, image generation/editing, video understanding, or audiovisual reasoning pipelines, and to graduate students studying reinforcement learning for multimodal models. Readers without a background in MoE routing, RoPE variants, or diffusion transformers will find the architecture sections demanding.

Authors’ abstract

We present Uni-MoE 2.0 from the Lychee family. As a fully open-source omnimodal large model (OLM), it substantially advances Lychee's Uni-MoE series in language-centric multimodal understanding, reasoning, and generating. Based on the dense LLM, we build Uni-MoE-2.0-Omni from scratch through three core contributions: dynamic-capacity Mixture-of-Experts (MoE) design, a progressive training strategy enhanced with an iterative reinforcement strategy, and a carefully curated multimodal data matching technique. It is capable of omnimodal understanding, as well as generating images, text, and speech. Architecturally, our new MoE framework balances computational efficiency and capability for 10 cross-modal inputs using shared, routed, and null experts, while our Omni-Modality 3D RoPE ensures spatio-temporal cross-modality alignment in the self-attention layer. For training, following cross-modal pretraining, we use a progressive supervised fine-tuning strategy that activates modality-specific experts and is enhanced by balanced data composition and an iterative GSPO-DPO method to stabilise RL training and improve reasoning. Data-wise, the base model, trained on approximately 75B tokens of open-source multimodal data, is equipped with special speech and image generation tokens, allowing it to learn these generative tasks by conditioning its outputs on linguistic cues. Extensive evaluation across 85 benchmarks demonstrates that our model achieves SOTA or highly competitive performance against leading OLMs, surpassing Qwen2.5-Omni (trained with 1.2T tokens) on over 50 of 76 benchmarks. Key strengths include video understanding (+7% avg. of 8), omnimodallity understanding (+7% avg. of 4), and audiovisual reasoning (+4%). It also advances long-form speech processing (reducing WER by 4.2%) and leads in low-level image processing and controllable generation across 5 metrics.