From Fragmentation to Fusion: A Survey and Roadmap for Omnimodal Embeddings

AI summary

Overview

Research area: Representation learning / embedding models across all major data modalities (natural language processing, computer vision, audio and speech, graph learning, tabular data, and time series), written as a survey and position paper.

Technical level: Intermediate to Advanced. The paper is conceptual rather than mathematical, but it assumes familiarity with contrastive learning, Transformer encoders, and standard retrieval benchmarks.

Scope (one sentence): The paper argues that embedding research is fragmented along modality lines and proposes a unified framework — five shared model components, four convergent trends, a three-capability "3C" lens, and three progressively deeper paths to unification — to guide the field from transformation and alignment toward true omnimodal joint embedding.

What This Paper Is About

Embedding models turn unstructured data (a sentence, an image, an audio wave, a graph, a table, a time series) into vectors whose geometry reflects meaning, and they sit underneath search, recommendation, and retrieval-augmented generation. The authors observe that these models are studied in separate communities with separate surveys and separate benchmarks, so the same ideas are repeatedly reinvented and the connections between them are never named. The paper's goal is a meta-level synthesis: to build conceptual bridges between the modality communities, show that a common set of principles is already emerging, and lay out a roadmap from loosely connected single-modality embeddings to a genuinely fused omnimodal embedding space.

Key Contributions

1. A diagnosis of fragmentation on three levels. The paper argues that embedding research is siloed in research (separate communities, venues, checkpoints, benchmarks), in surveys (monolithic single-modality reviews that never cross modality boundaries), and in evaluation (islands of frozen-encoder benchmarks with no shared yardstick). It quantifies this in Appendix B: across 1,352 formally published embedding papers (2023–2025) and a decade of arXiv preprints (2015–2025), the overwhelming majority of work is single-modality, and works explicitly crossing modality boundaries account for under 5% of both samples and emerge only recently.

2. A modality-agnostic anatomy of an embedding model. The paper decomposes any embedding model into five recurring components — data unit and tokenization, encoder, pooling/aggregation, training objective, and evaluation paradigm — plus the representational object that is actually exposed to downstream consumers, and instantiates each component across text/code, image/video, audio, time series, graph, and table.

3. A 3C framework plus three paths to unification. Unification is graded by three progressively harder capabilities: coverage (what the system takes in and has tested), comparability (whether separately encoded inputs can be matched for a task), and compositionality (whether an output built from several inputs genuinely depends on all of them). Three paths satisfy these in turn: transformation broadens coverage, alignment enables comparability, and joint embedding — the terminus the authors advocate — aims at compositionality.

4. A four-part roadmap and cross-pollination blueprint. The paper organizes the field into Part I (unified foundation), Part II (convergent principles: objectives, architectures, representational granularity), Part III (paths to unification), and Part IV (road ahead: modality-specific properties, a reuse blueprint for transferring one modality's methods to another, unified evaluation, and outlook), with per-modality cheat sheets in Appendix A.

Main Findings

• Four trends are quietly unifying all modalities: contrastive learning as a universal objective (from InfoNCE in text to CLIP in vision–language and CLAP in audio–language); Transformers as universal encoders (ViT, audio Transformers, Graph Transformers, patch-based time-series Transformers, decoder-only LLMs repurposed as embedders); the pretrain–finetune paradigm (masked language modeling, MAE, masked prediction for audio, then task-specific or contrastive finetuning); and a granularity shift in which every modality moves from token- or pixel-level objectives toward sequence- or image-level semantic units.

• Mechanism is shared, pair construction is not. The contrastive loss itself is modality-agnostic, but what counts as a positive pair is where each modality's structure enters. Text uses discrete edits (EDA, AEDA, UDA, TMix) and eventually implicit augmentation via dropout (SimCSE); vision uses crops, color jitter, and flips; vision–language gets positives for free from web captions; audio manufactured positives from waveform-level augmentations. Each construction declares an intended invariance and comes with a characteristic failure mode (e.g., cropping can remove the discriminating object in vision; graph node/edge dropout can change graph semantics).

• The input unit and the output object are routinely conflated. The paper distinguishes the data unit (the atom of the input) from the representational object (the atom of the output), and identifies five distinct readouts an encoder can expose: a global dense vector in ℝ^d, a local vector sequence/set in ℝ^(N×d), a sequence of discrete codes, a probability distribution, and a magnitude-preserving vector. This choice, not the encoder, decides what an embedding can do.

• "Convergence" is four different things. The paper separates architecture reuse, objective reuse, coordinate alignment, and parameter sharing. A shared trunk does not entail aligned retrieval vectors (Meta-Transformer), nor does aligned output imply shared weights (ImageBind), while systems such as OneLLM and VLM2Vec combine both. This prevents loose claims of "unification" from being conflated.

• Representation is falling behind generation. GPT-4o has jointly consumed text, audio, image, and video since May 2024, but on the embedding side, verified joint fusion with measurable compositional gains has so far been obtained only for text–image pairs and partially for composed text, vision, and audio queries. Fused omni embedding models and benchmarks covering text, image, video, and audio appeared only in late 2025 — almost two years after the generative side crossed the same threshold.

• Early realizations exist but are partial. Systems such as e5-omni, jina-embeddings-v5-omni, MM-Embed, and UniME map text, image, video, and audio into a shared space, either by explicit cross-modal alignment or by fusing modalities inside a single encoder. The Platonic Representation Hypothesis and empirical evidence of a universal latent semantic structure across text embedding spaces are offered as reasons to expect deeper convergence.

• Modality counts are often inflated. 4M-21 claims twenty-one modalities, but almost all (depth, segmentation, edges, human pose, feature maps) are derived views of the same visual scenes; in this taxonomy they belong to the vision family rather than counting as peer modalities.

Methodology in Plain English

This is a position paper and survey, not an experimental study. The authors' method is conceptual and organizational:

• They define a consistent unit of analysis (coarse modality families: text and code, image and video, audio, graph, table, time series), explicitly excluding intra-family derived views so that "modality" means a community and benchmark ecosystem rather than a signal channel.

• They build a static anatomy — a checklist of components every embedding model has — so that models from different fields can be compared slot by slot, then describe the dynamic part: which slots are changing in the same direction everywhere.

• They apply a three-part analytical lens throughout: intersection (shared principles), difference (modality-specific structure), and reuse (directional transfer of a method from one modality to another). The lens is used to keep claims honest — for each technique, the paper asks which modality owns the problem and which is borrowing the solution.

• They support the fragmentation claim with a bibliometric analysis (Appendix B) of 1,352 formally published papers and a decade of preprints, coding each by modality and by whether it crosses modality boundaries.

• They organize the roadmap around the 3C lens, so that any headline claim of "omnimodal" capability can be split into three questions: what does the system take in, what can it match, and what can it combine.

Why This Matters

Impact on research. The paper reframes fragmentation from an organizational complaint into a scientific bottleneck. It gives cross-modal researchers a shared vocabulary (components, representational objects, the 3C lens) and argues that many "new" methods in one field are reinventions of methods already mature in another — SimCSE's implicit positives versus view augmentation, masked autoencoding in vision and masked prediction in audio a year apart. It also proposes that unified evaluation, not just unified models, must be built.

Real-world applications (from the paper):

• Retrieval-augmented generation. RAG pipelines are described as the most common case where a generative model that perceives four modalities is indexed by retrieval systems that are bimodal, pairwise-aligned, or text-centric.
• Multimodal search over a single shared index, letting users search images with text and describe audio with language through one space rather than separate indexes.
• Grounded generation, where generation is conditioned on retrieved images and text rather than text alone.
• Robot memory, where experience is currently stored as text captions rather than raw multimodal experience; a joint embedding would let agents store and retrieve the experience itself.

Industry relevance. Embedding models already power web-scale retrieval and ranking in recommendation and dense retrieval in search (where dense embeddings have largely replaced sparse bag-of-words representations like BM25). The asymmetry between omni-capable generators and narrowly aligned embedders is described as a bottleneck of the "omni stack" itself, which makes a shared geometry directly relevant to anyone building retrieval, RAG, or recommendation infrastructure on top of multimodal models.

Future Directions

• Build the hub, not just the spokes. The paper argues that current work manufactures modality-specific representations while nobody builds the modality-invariant hub; achieving verified joint fusion beyond text–image pairs (and the partial text–vision–audio case) is the central open problem.
• Respect modality-specific properties inside a joint space. Part IV (Section 9) is devoted to the properties a true joint embedding must respect — audio's superposition of factors, graph's explicit relational structure, time series' continuous axis and non-standardized units — and these remain underexplored.
• Plan cross-pollination deliberately. Section 10 proposes a blueprint for reusing one modality's hard-won techniques in another (for example, disentanglement methods provoked by audio's factor superposition as a reuse target for other modalities), rather than waiting for accidental rediscovery.
• Build unified evaluation. Section 11 addresses the unified benchmark the authors say cannot yet be built, organized around coverage, comparability, and compositionality; today each modality has its own M_XEB (MTEB for text, MIEB for image, MAEB for audio), and tables and time series are still scored mainly through task heads and prediction archives.

Note: because the supplied text is truncated partway through Section 3.1, the detailed content of Sections 4–12 is available here only at the level of the roadmap and section descriptions above; specific findings from those chapters are not reported in the available content.

Target Audience

Researchers and practitioners who work on embedding models in any single modality and want to see how their methods map onto the rest of the field; engineers building retrieval, RAG, recommendation, or search systems who need to reason about combining modalities in one index; and graduate students or newcomers looking for a single cross-cutting entry point before diving into a modality-specific survey. The paper explicitly positions itself as a complement to — not a replacement for — the deep single-modality surveys, so readers seeking implementation-level detail on one modality are pointed to those instead and to the paper's Appendix A cheat sheets.

Authors’ abstract

Embedding models underpin virtually every modern AI system, from search engines to recommendation systems. Despite shared mathematical foundations, research on embedding models remains fragmented by modality: the communities develop largely in isolation, seldom exchanging insights. We argue that this fragmentation obscures a deeper convergence. Driven by contrastive learning, Transformer architectures, and a granularity shift, embedding research across modalities is converging toward a common set of principles, even as modality-specific challenges remain underexplored. To reason across this convergence without conflating it, we propose a framework of three progressively harder capabilities: coverage, comparability, and compositionality. Three paths toward unification satisfy these in turn: transformation broadens coverage, alignment enables comparability, and joint embedding, the terminus we advocate, aims at compositionality. This position paper traces the resulting evolution, providing the first unified, cross-cutting treatment of omnimodal embedding research.