OmniRet: Efficient and High-Fidelity Omni Modality Retrieval

AI summary

Overview

• Research area: Information retrieval / multimodal machine learning (cs.IR), specifically universal cross-modal embedding models.
• Technical level: Intermediate — assumes familiarity with contrastive learning, embedding pooling, and multimodal LLM architectures, but the core ideas are explained accessibly.
• Scope: Presents OmniRet, a single embedding model that retrieves across text, images, videos, and audio, plus a new audio-centric evaluation benchmark (ACM).

What This Paper Is About

Most state-of-the-art retrieval models handle only two modalities — usually text and vision — so they cannot answer complex queries that combine, say, an image, a caption, and an audio clip. OmniRet is the first retrieval system that fuses three modalities (text, vision, audio) into one shared embedding space while staying computationally practical. The authors also build a new benchmark to test audio-heavy retrieval tasks that existing datasets ignore.

Key Contributions

1. OmniRet, a universal three-modality retriever. It uses an LLM as a cross-modal composer over text, image, video, and audio inputs, and is the first model to handle composed queries spanning all three of text, vision, and audio.
2. Two architectural innovations for efficiency and fidelity. A Shared Media Resampler compresses long media token sequences from modality encoders into short fixed-size latents, and Attention Sliced Wasserstein Pooling (ASWP) aggregates the LLM's output tokens into a single high-fidelity embedding without losing fine-grained detail.
3. A large-scale multi-task training recipe. OmniRet is trained on roughly 6.2 million query–target pairs aggregated from 30 datasets spanning text, image, video, and audio retrieval, using contrastive, triplet, and diversity losses with only ~84M trainable parameters.
4. The Audio-Centric Multimodal (ACM) benchmark. A new evaluation suite built from VGG-Sound that introduces two previously missing tasks: composed audio retrieval (audio + text → audio) and audio-visual retrieval (audio → image/video).

Main Findings

• Broad retrieval gains. On an extended M-BEIR benchmark of 13 retrieval tasks, OmniRet leads in 12 of 13, including audio-to-text, text-to-audio, and several composed visual-text tasks, despite using a 1.5B-parameter LLM.
• Audio and video strength. It outperforms dedicated audio models like CLAP and MMT, and beats video-focused baselines such as VLM2VecV2 on video retrieval, even though those models were trained on more in-domain data.
• Strong generalization on MMEBv2. Without full fine-tuning on the evaluation training sets, OmniRet achieves state-of-the-art video retrieval (48.6 CLS, 36.5 RET, 43.3 MRET) and competitive image retrieval (~65 RET, near the median of fully fine-tuned models).
• Composed audio retrieval is a real gap. On the new ACM benchmark, OmniRet reaches 23.0 recall on audio+text → audio, far above CLAP (16.1) and ImageBind (7.32), and roughly half the performance of a much more expensive multi-stage caption-based pipeline using QwenOmni + Gemma (44.6).
• Audio-visual alignment is solid. OmniRet matches ImageBind on audio-to-video retrieval (35.5) and stays competitive on the other audio-visual directions.
• Pooling choice matters enormously. Ablations show average pooling collapses performance by 29.5 points versus ASWP, and using the standard [EOS] token as the embedding loses 6.8 points.
• Diversity regularization is critical. Removing the diversity loss on resampled media tokens drops average recall by 3.1 points, while removing the triplet loss costs only 0.5.
• The resampler pays for itself. Removing it degrades performance by 3.5 points because batch sizes must shrink; sharing it across modalities with modality-specific latents (rather than using separate resamplers) is the best configuration.
• Benchmark quality verified. Human annotators reach 87% accuracy on the composed audio task while a text-only Gemini-2.5 reaches 96%, confirming the generated captions and modification texts are meaningful.

Methodology in Plain English

OmniRet reuses three off-the-shelf encoders: SigLIP for images, a QwenAudio encoder for audio, and a GTE-Qwen2-1.5B LLM as the central composer. Each media encoder produces hundreds of tokens, which is expensive to feed into an LLM. So the authors insert a Shared Media Resampler — a Perceiver-style attention module with a single shared set of query latents plus small modality-specific latents — that squeezes those tokens down to a short fixed-length representation. This keeps batches large enough for effective contrastive learning.

At the output end, instead of averaging the LLM's hidden states (which blurs detail) or keeping every token (which is slow and storage-heavy), they use ASWP. Token embeddings are treated as a probability distribution and compared against learnable reference points across thousands of random 1D projections, producing a histogram-like descriptor. A straight-through maximum selection then collapses this into the final fixed-size embedding vector. Only the projectors, resampler, pooling layer, and a small LoRA adapter are trained; the encoders and LLM stay frozen.

Training happens in two stages: a warm-up on 2M samples of simple unimodal and text-binding tasks, then a full fine-tuning pass over roughly 18M samples across all datasets, with a task-balancing sampler to keep any one retrieval type from dominating.

Why This Matters

• Impact on research: It demonstrates that universal retrieval need not be limited to two modalities, and offers a concrete recipe for efficient omni-modal representation. It also fills a real evaluation gap by providing the first benchmark for composed audio retrieval.
• Real-world applications:
  • Search engines and media libraries that let users search by a mix of photo, audio clip, and text description.
  • Podcast or music recommendation systems that match audio content to rich, multi-part user queries.
  • Video editing or archival tools that retrieve the right clip from a text-plus-reference-audio prompt.
  • Robotics and accessibility systems that need to ground sound, sight, and language in one representation.
• Industry relevance: The single-vector output format is compatible with standard approximate nearest-neighbor indexes, meaning the approach plugs into existing large-scale retrieval infrastructure without the storage costs of late-interaction models like ColBERT. The modest 84M trainable parameters also make it feasible to train and deploy without massive compute.

Future Directions

1. Scaling up. The authors deliberately kept the LLM at 1.5B parameters and training data at ~6M pairs; larger backbones and more data are expected to raise performance substantially.
2. More modalities. Extending beyond text, image, video, and audio to depth maps, 3D point clouds, and speech would push toward truly universal embeddings.
3. More retrieval tasks, not just more modalities. Combining the modality expansion here with the task expansion in MMEBv2 into a single unified training run is a natural next step.
4. Harder audio benchmarks. The ACM benchmark could be extended to interleaved, mixed-media documents where queries and candidates are themselves composites.

Target Audience

Researchers and engineers working on multimodal retrieval, embedding models, or RAG pipelines will get the most value. It is also useful for practitioners who need a single model covering audio and video alongside text and images, and for benchmark designers interested in audio-centric evaluation. Readers without a background in contrastive learning or transformer architectures will need to consult the cited baselines (CLIP, BLIP, ColBERT) first.

Authors’ abstract

Multimodal retrieval is the task of aggregating information from queries across heterogeneous modalities to retrieve desired targets. State-of-the-art multimodal retrieval models can understand complex queries, yet they are typically limited to two modalities: text and vision. This limitation impedes the development of universal retrieval systems capable of comprehending queries that combine more than two modalities. To advance toward this goal, we present OmniRet, the first retrieval model capable of handling complex, composed queries spanning three key modalities: text, vision, and audio. Our OmniRet model addresses two critical challenges for universal retrieval: computational efficiency and representation fidelity. First, feeding massive token sequences from modality-specific encoders to Large Language Models (LLMs) is computationally inefficient. We therefore introduce an attention-based resampling mechanism to generate compact, fixed-size representations from these sequences. Second, compressing rich omni-modal data into a single embedding vector inevitably causes information loss and discards fine-grained details. We propose Attention Sliced Wasserstein Pooling to preserve these fine-grained details, leading to improved omni-modal representations. OmniRet is trained on an aggregation of approximately 6 million query-target pairs spanning 30 datasets. We benchmark our model on 13 retrieval tasks and a MMEBv2 subset. Our model demonstrates significant improvements on composed query, audio and video retrieval tasks, while achieving on-par performance with state-of-the-art models on others. Furthermore, we curate a new Audio-Centric Multimodal Benchmark (ACM). This new benchmark introduces two critical, previously missing tasks-composed audio retrieval and audio-visual retrieval to more comprehensively evaluate a model's omni-modal embedding capacity.