Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings

AI summary

Overview

Research area: Multimodal representation learning — specifically omnimodal embedding models built on Omnimodal Large Language Models (Omni-MLLMs), covering text, image, video, and audio.

Technical level: Advanced. The paper assumes familiarity with LoRA adaptation, contrastive learning (InfoNCE), mixture-of-experts routing, and multimodal LLM backbones.

Scope: This paper proposes Syn-Omni, a framework that separates LoRA adaptation into shared and modality-specific expert paths and progressively blends their routing, evaluated on 81 embedding tasks across four modality groups.

What This Paper Is About

Existing omnimodal embedding models usually adapt a backbone with a single, fully shared LoRA applied to mixed-modality data, which learns a unified space but offers little structural separation between universal semantics and modality-specific characteristics. The authors argue that neither extreme works well: fully shared adaptation underuses modality-aware specialization, while fully isolated per-modality models cannot exploit cross-modal synergy.

The goal is a structured middle ground — one shared adaptation path plus modality-expert paths, with a training schedule that lets experts first specialize in their own modality and then progressively collaborate with experts from other modalities.

Key Contributions

1. A unified omnimodal adaptation framework that balances a robust shared semantic foundation against modality-aware specialization, addressing the structural limitations of both fully shared and entirely isolated adaptation.

2. Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path (always active) plus modality-expert LoRA paths (activated according to the modalities present), with an orthogonal penalty that reduces redundancy between shared and expert weight directions.

3. Progressive Synergy Routing (PSR), which schedules routing weights from hard, modality-indicator routing to learnable soft routing across three phases (Warmup, Blending, Synergy), combined with a Synergy Margin Loss (SML) that enforces a relative margin between present- and absent-modality experts rather than absolute suppression.

4. Evaluation across 81 tasks spanning image (36 tasks), video (23), audio (12), and audiovisual (10) modalities, with classification, retrieval, and QA task types, showing improvements over omnimodal baselines at both 3B and 7B scales.

Main Findings

• Overall benchmark performance: Syn-Omni reaches the best overall Hit@1 scores of 49.4% at the 3B scale and 51.6% at the 7B scale, compared with 47.7% and 50.7% for the Uni-Omni contrastive baseline.

• Statistical significance: The improvements over Uni-Omni are reported as statistically significant across the 81 tasks in a paired t-test (p = 0.0044 at 3B and p = 0.0189 at 7B).

• Modality-wise gains: At the 3B scale, Syn-Omni improves most on audio (+2.5%) and audiovisual (+3.0%) tasks, with more moderate margins on image and video. Per-modality 3B scores are 68.1 (image), 40.5 (video), 45.6 (audio), and 43.4 (audiovisual).

• Comparison to other baselines (3B): Omni-Embed-Nemotron 33.4, LCO-Emb 44.8, e5-omni 43.6 overall. At 7B: OmniEmbed-v0.1 35.0, OmniEmbed-Multivent 41.5, LCO-Emb 46.9, e5-omni 50.9, WAVE 38.5, Uni-Omni 50.7, Syn-Omni 51.6.

• Different baselines lead on different modalities: LCO-Emb leads on video tasks and e5-omni leads on image tasks at certain scales, but Syn-Omni is reported as stronger on audio and audiovisual tasks and higher overall.

• Component ablation (3B): Starting from the Uni-Omni baseline at 47.7 overall, adding the shared-expert structure with hard routing gives 48.7; adding the orthogonal penalty alone gives 48.6; soft routing alone gives 48.6; PSR with SML gives 49.0; the full model with the orthogonal penalty gives 49.4.

• Structural efficiency: Increasing a single shared path's rank from 16 to 36 yields only marginal gains (47.7 → 48.2, with 59.9M parameters), while OME-LoRA decomposing the same total rank of 24 into shared and expert paths reaches 48.7 at 44.9M parameters, and 49.4 with PSR at 48.3M parameters — surpassing the largest shared-only variant with fewer trainable parameters.

• Shared path matters more than experts alone: Expert-only adaptation falls below the shared-only baseline at the same parameter count (47.0 vs. 47.7 at 29.9M parameters).

• SML beats BCE: Replacing BCE with the Synergy Margin Loss consistently improves results under both standard soft routing (48.6 → 48.8) and progressive blending (48.8 → 49.4).

• Hyperparameter sensitivity: Best overall score at λ_ortho = 0.20, λ_SML = 0.20, and γ = 0.2; larger margin values cause a noticeable performance drop.

• Representation analysis: Measured on VALOR-32k evaluation samples, Syn-Omni produces substantially lower pairwise cosine similarity between OME-LoRA pathways than a soft-routing-only variant, indicating clearer decoupling. Under video-audio inputs, present-modality experts (video, audio) keep dominant routing weights while absent-modality experts (text, image) gradually acquire non-zero activations.

• Cost of the design: Dynamic routing in the OME-LoRA layer adds 17–18% higher latency with less than 1% additional GPU memory; the shared pathway remains mergeable after training, but the modality-experts do not.

Methodology in Plain English

The researchers start from a pretrained Qwen-2.5-Omni backbone (3B and 7B variants) and freeze it, training only LoRA adapters inserted into the LLM layers. Instead of one LoRA for everything, each layer gets a shared LoRA path that every input uses, plus modality-expert LoRAs for each modality (text, image, video, audio). An input containing, say, video and audio, activates the video and audio experts alongside the shared path.

Two mechanisms keep this organized. First, an orthogonal penalty pushes each expert's input-side projection direction away from the shared one, so experts do not simply duplicate the shared path. Second, routing controls how much each expert contributes. Early in training, routing is hard — only experts matching the input's actual modalities turn on, letting them build clean modality-specific priors. Over a scheduled warmup and blend period, a learnable soft router takes over, allowing experts from non-present modalities to contribute. Rather than forcing absent experts to zero (as binary cross-entropy would), the Synergy Margin Loss only requires present-modality experts to outscore absent ones by a margin γ.

Training mixes 1.12M samples across image, video, audio, and audiovisual data, uses an InfoNCE contrastive objective, and optimizes a combined loss of contrastive loss plus the two regularization terms. Models train for one epoch with AdamW, a cosine learning rate schedule, peak learning rate 10⁻⁴, warmup ratio 0.04, global batch size 768, GradCache, temperature τ of 0.02, and EOS-token pooling with L2 normalization. The shared path rank is 8 and each modality-expert rank is 4 (baseline LoRA rank is 16); PSR uses t_warm = 0.1 and t_blend = 0.5.

Why This Matters

Impact on research. The work challenges the default assumption in omnimodal embedding that a single shared adapter is sufficient. It shows that separating parameters along the modality axis (rather than the task axis, as in prior LoRA mixture work) produces measurable gains, and that the schedule by which experts begin cooperating matters as much as the architecture.

Real-world applications:

• Cross-modal search and retrieval: querying a video library with audio, or finding images from a spoken description.
• Content moderation and recommendation systems that must reason over composed inputs such as audiovisual content.
• Accessibility tooling that maps between text, speech, and visual media in a shared embedding space.
• Multimodal RAG and agent pipelines that need one embedding space for heterogeneous documents.

Industry relevance. The framework is built on open Qwen-2.5-Omni backbones with LoRA, so trainable parameters stay modest (48.3M at 3B, 65.1M at 7B, with router parameters of 3.4M and 4.5M respectively — less than 8% of total trainable parameters). That makes it practical for organizations that want unified retrieval without full fine-tuning, though the 17–18% latency overhead is a real constraint for latency-sensitive serving.

Future Directions

• Reducing inference overhead. The paper explicitly notes that input-dependent routing prevents merging expert weights into the frozen backbone, creating 17–18% higher latency; making experts mergeable or otherwise cheaper at inference is a clear open problem.

• Extending beyond the four evaluated modalities. The framework is defined over text, image, video, and audio; whether the shared-expert decomposition generalizes to further modalities (for example, 3D or sensor data) is untested.

• Understanding the routing schedule's generality. The warmup and blend ratios (t_warm = 0.1, t_blend = 0.5) were set as hyperparameters; whether the optimal schedule depends on dataset mixture or modality count is not resolved.

• Addressing the stated risks. The authors flag inherited data biases that could skew cross-modal retrieval and misuse for unauthorized surveillance or identity linkage — mitigation strategies are not proposed.

Target Audience

Researchers and engineers working on multimodal or omnimodal representation learning, contrastive embedding models, and parameter-efficient fine-tuning. Also relevant to practitioners building retrieval systems over mixed-media corpora who are weighing shared versus specialized adapters, and to readers interested in routing/curriculum designs for mixture-of-experts adaptation. Beginners will find the framework description accessible in outline but will need background in LoRA and contrastive learning to follow the equations.

Authors’ abstract

Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.