Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings

AI summary

Overview

Research area: Multimodal representation learning, specifically universal ("any-to-any") dense retrieval embeddings spanning text, image, video, and audio.

Technical level: Advanced. The paper assumes familiarity with contrastive retrieval training, InfoNCE objectives, hard-negative mining, and multimodal transformer architectures.

Scope: A technical report from the Ovis-Embedding Team (Alibaba Token Hub, Alibaba Group) describing an omni-modal embedding family built on native multimodal backbones, its data pipeline, a four-stage training recipe, and its reported state-of-the-art results on MMEB-v3, MMEB-v2, MVEB, MAEB, and RTEB.

What This Paper Is About

Most retrieval systems are built for a fixed set of modality pairs, and existing multimodal embedders either ignore audio or bolt an audio encoder onto a geometry that was learned without audio. The goal of this work is a single model family that maps text, images, video, and audio — including interleaved combinations — into one shared representation space where relevance scores are comparable across modalities. The authors pursue this by starting from a pretrained omni-modal understanding model rather than assembling separate modality towers, then specializing it for retrieval through a staged training pipeline.

Key Contributions

1. Native omni-modal initialization for universal embeddings. Instead of retrofitting an audio branch onto a vision–language embedder or aligning a separately trained audio encoder, Ovis-Embedding starts directly from a pretrained omni-modal understanding model. It retains the natively aligned text, vision, audio, and video front-ends together with their shared encoder, and learns any-to-any retrieval in a coherent representation space without modality-specific embedding heads.

2. Omni-modal homogeneous-source sampling. A unified sampling strategy spans text, image, video, audio, visual-document, agent, and interleaved multimodal tasks. Each micro-batch is drawn from one source, and deduplicating pooled candidates produces task-consistent hard negatives, which the authors state reduces modality shortcuts and improves fine-grained discrimination in the shared embedding space.

3. Embedding-specific training and inference optimization. During training, a difficulty-aware focal loss concentrates learning on unresolved hard examples, while similarity-based Embedding Distillation transfers the fine-grained similarity geometry of complementary experts into a single encoder. At inference, low-rank feature decomposition produces compact embeddings with flexible dimensionality, reducing retrieval storage and computation.

4. Open state-of-the-art models for the community. The team states it will release checkpoints covering both vision–language and native omni-modal retrieval, along with training and inference code and a unified evaluation toolkit. Ovis-Embedding-Omni-3B is reported to establish a new state of the art on MMEB-v3, Ovis-Embedding-VL-9B to lead MMEB-v2, and the family to further advance MVEB, MAEB, and RTEB.

Main Findings

• MMEB-v3 leadership: Ovis-Embedding-Omni-3B is reported to lead all six MMEB-v3 modality groups.
• MMEB-v2 leadership: Ovis-Embedding-VL-9B is reported to lead four of five MMEB-v2 task groups.
• Additional benchmark advances: The Ovis-Embedding family is reported to further advance the state of the art on MVEB, MAEB, and RTEB, spanning text, image, video, and audio modalities.
• Unified representation without extra heads: All three models use the final-layer hidden state at the last non-padding token as the retrieval embedding, with no additional embedding projection or modality-specific output head; conventional pairwise retrieval (text–text, image–text, video–text, audio–text, audio–video) becomes a special case of general any-to-any retrieval within one space.
• Audio without a separate branch: Ovis-Embedding-Omni-3B initializes from Qwen2.5-Omni-3B and removes the Talker speech-generation pathway while retaining the text tokenizer, vision encoder, audio encoder, and shared Thinker transformer, so audio is encoded by a backbone in which it was already aligned with the other modalities.
• Scale of training corpus: The corpus assembled for training contains approximately 50M (query, target) training pairs, a figure the paper explicitly labels as placeholder estimates based on the current data freeze, to be updated for the camera-ready version.
• Numeric benchmark scores not reported in the available content: The paper content provided describes which benchmarks and groups each model leads, but does not state the underlying percentage scores; the figure caption notes that bar lengths are normalized within each benchmark while labels show original percentage scores.

Methodology in Plain English

The team begins from backbones that were already trained to fuse modalities natively rather than from text-only or vision–language models with add-on components. The Omni model comes from Qwen2.5-Omni-3B, whose text tokenizer, vision encoder, and audio encoder feed a shared causal transformer called the Thinker, with Time-aligned Multimodal Rotary Position Embedding (TMRoPE) aligning audio and video temporal positions. The speech-generation Talker is discarded. The VL models come from Qwen3.5-2B and Qwen3.5-9B, whose vision encoders turn dynamic-resolution images and temporally sampled video frames into visual tokens inserted into the language-token stream. Qwen3.5-2B uses 24 backbone layers with hidden size 2,048 and Qwen3.5-9B uses 32 layers with hidden size 4,096, with a hybrid stack using three Gated DeltaNet linear-attention layers for every full gated-attention layer. In all cases the language-modeling output head is removed and the final-layer hidden state at the last non-padding token is used directly as the embedding, scored by cosine similarity.

For data, the authors build a corpus covering text, images and visual documents, video, audio, and interleaved inputs. Image data come from classification sources such as ImageNet, CUB-200, and SUN397, supplemented by web images labeled with Qwen3.5-Plus; question answering from TextVQA, DocVQA, and InfoVQA plus synthetic long-tail QA; retrieval from news image–text pairs, text-to-image search, and image-to-image search; and grounding from MS-COCO and its derivative datasets. They also collect real user queries and use Quark web search for open-world, product-oriented retrieval. Video data follow a collect–filter–organize pipeline in which sampled frames are validated by a vision–language model before a candidate becomes a positive, followed by conservative video-grounded query refinement that corrects only observable factual mismatches while positive and negative videos stay fixed. Audio data cover recognition, audio–language matching, and bidirectional retrieval in both directions, with negatives kept within the same task family. Text data follow five paradigms — general retrieval, instruction-following retrieval, reasoning retrieval, multi-condition retrieval, and long-context retrieval — plus RTEB-domain data in law, finance, programming, and healthcare and MTEB-style tasks recast into query–candidate form, with BGE-M3 used to mine in-domain hard negatives. Agent data cover tool, GUI, and knowledge retrieval with BM25-based hard negatives.

Everything is standardized as query–positive–negative tuples, deduplicated, and quality-filtered. Evaluation test sets are strictly deduplicated against training, using normalized text matching for text and perceptual-hash matching for images.

Training proceeds in four stages. Stage 1 is low-rank contrastive pretraining on the omni-modal corpus with in-batch mixing, where candidates from all queries are gathered across data-parallel ranks into one shared pool of size N(1+K), and every query is scored against that pool under a temperature. The objective combines a focal-weighted contrastive loss, which rescales each query's loss by its current retrieval difficulty so unresolved queries dominate updates, with an embedding distillation loss that transfers a teacher's full similarity distribution over candidates rather than a one-hot target. Stage 2 unfreezes the full model and refines it on higher-quality data using homogeneous-source batches. Stage 3 applies annealing embedding distillation, retaining teacher-correct examples, upsampling cases the student still misses, and assigning stronger ranking supervision to less confident queries. Stage 4 applies a low-rank feature transformation with lightweight residual adaptation to produce multiple compact embedding dimensions from the same encoder.

Note: the provided content is truncated mid-way through Section 4.1.1, so the detailed formulation of the distillation loss and later stages is not available here.

Why This Matters

Impact on research. The paper argues that current multimodal embedders provide an incomplete foundation for universal retrieval: vision–language specialists do not support audio, while existing omni-modal systems either add an audio pathway to a pretrained text–vision embedder or align a separate audio encoder with an established embedding space, so acoustic inputs are adapted to a geometry learned without them. By showing that a natively omni-modal backbone can be converted into a unified encoder without modality-specific heads, the work points toward a single retrieval interface for heterogeneous inputs and toward evaluation settings — audio, audio–video, and any-to-any retrieval — that the authors describe as under-served by existing open resources.

Real-world applications:

• Maintenance and field-service agents that combine an abnormal machine sound with a short text description to retrieve a video tutorial, a diagram in a PDF manual, or a previous service record.
• Search and recommendation over mixed repositories where text, images, video, and audio share one index and one relevance scale.
• Retrieval-augmented generation pipelines that need to fetch evidence in whichever modality it happens to exist.
• Agentic systems that retrieve tools, API documentation, GUI states, and knowledge passages through the same embedding interface.

Industry relevance. The work comes from Alibaba Token Hub and draws on product-scale data sources such as Quark web search and user-issued queries. The deployment angle is explicit: low-rank feature decomposition is used to trade embedding dimensionality against index storage and similarity-computation cost with minimal loss in retrieval quality, and the family spans compact vision–language models (2B), an omni-modal 3B, and a higher-capacity 9B variant to cover different accuracy and efficiency requirements.

Future Directions

• Closing the reported data estimates. The corpus size of approximately 50M pairs is described as a placeholder estimate pending a final data freeze, so an updated accounting of the data pipeline is an open item flagged by the authors themselves.
• Modality coverage beyond the current family. The VL variants support text, images, and video but not audio, while only the Omni variant carries the native audio pathway; extending the unified interface across the full family is a natural next question.
• Audio, audio–video, and any-to-any retrieval benchmarks. The authors identify these settings as under-served by existing open resources and state that their unified evaluation toolkit is intended to support development and evaluation there.
• Scaling and efficiency trade-offs. With low-rank decomposition offered as a way to generate flexible embedding dimensions from one encoder, the paper raises the question of how far compact embeddings can be pushed before retrieval quality degrades, and how the 2B, 3B, and 9B variants compare on that trade-off. These comparisons are not reported in the available content.

Target Audience

This paper is most useful to retrieval and multimodal machine learning researchers and engineers who build embedding models or search systems over heterogeneous data, particularly those working on cross-modal or any-to-any retrieval and on audio or audio–video search. It also speaks to practitioners who need a unified retrieval interface across text, image, video, and audio in production indexing systems, and to teams that want open checkpoints, training recipes, and a shared evaluation toolkit ahead of release. Readers without a background in contrastive retrieval training and multimodal transformer architectures will find the training sections dense, though the data-construction sections are comparatively accessible.

Authors’ abstract

In this report, we introduce \textbf{Ovis-Embedding}, a state-of-the-art omni-modal embedding family built on native integration of text, image, video, and audio. Instead of assembling separate modality towers, Ovis-Embedding uses a shared multimodal backbone to encode different modalities in a common representation space. Specifically, we make \textbf{three key advances}: (1) \textbf{native omni-modal initialization}: we adopt a pretrained Qwen-omni model as the embedding backbone and adapt it through contrastive training with low-rank initialization; (2) \textbf{data-centric omni-modal training}: we construct a broad, high-quality corpus spanning text, images, video, audio, and interleaved multimodal data. To improve data efficiency, we introduce homogeneous-source sampling to form task-consistent batches with informative in-batch negatives; and (3) \textbf{embedding-specific training and inference optimization}: we use focal loss to emphasize hard examples and similarity-based Embedding Distillation to transfer fine-grained similarity structure from complementary experts. At inference time, low-rank feature decomposition enables compact embeddings with flexible dimensionality and minimal performance loss. Empirical evaluations show that the \textbf{Ovis-Embedding} family achieves state-of-the-art performance on \textbf{MMEB-v3}, \textbf{MMEB-v2}, \textbf{MVEB}, \textbf{MAEB}, and \textbf{RTEB}, demonstrating its effectiveness across text, image, video, and audio modalities. These results highlight the potential of unified omni-modal training to overcome modality fragmentation and advance universal embedding models for any-to-any retrieval.