Scaling Language-Centric Omnimodal Representation Learning

AI summary

Overview

• Research area: Multimodal representation learning — building embedding models that place text, images, audio, and video into one shared similarity space, using multimodal large language models (MLLMs) as backbones.
• Technical level: Advanced. The paper mixes empirical embedding-space analysis (anisotropy, kernel alignment), a lightweight contrastive fine-tuning recipe, and a PAC-Bayesian theoretical bound.
• Scope: The paper argues that MLLMs already contain latent cross-modal alignment from generative pretraining, shows that text-only contrastive refinement unlocks it (the LCO-Emb framework), and proposes a Generation-Representation Scaling Law (GRSL) linking generative quality to representation quality.

What This Paper Is About

Embedding models that compare images, audio, video, and text have traditionally been trained with massive CLIP-style contrastive learning over paired cross-modal data, but performance plateaus on harder tasks such as multilingual image retrieval, visual text representation, and interleaved multimodal inputs. Newer systems instead fine-tune autoregressive MLLMs with contrastive learning and perform better, but nobody had explained why. This paper diagnoses the cause — implicit cross-modal alignment learned during generative pretraining — and turns that diagnosis into a training framework plus a scaling law.

Key Contributions

1. A language-centric omnimodal embedding framework (LCO-Emb). LCO-Emb fine-tunes only the language decoder of an MLLM with text-only contrastive learning via LoRA while freezing modality encoders and the projector, then optionally adds a small amount of multimodal paired data. It reaches state-of-the-art results across modalities.
2. Empirical evidence of latent cross-modal alignment in MLLMs. Through anisotropy and kernel-level similarity analyses, the authors show that text-only fine-tuning improves the geometry of text embeddings and generalizes to image, audio, and video embeddings, implying that alignment already exists inside the MLLM.
3. The Generation-Representation Scaling Law (GRSL). Across OCR-based image-text, video-text, and audio-text task families, representational quality after contrastive refinement scales positively with the backbone's generative capability measured before contrastive learning.
4. A theoretical explanation and a stress-test dataset. A PAC-Bayesian generalization bound formally links generative quality to an upper bound on representation performance, and the new SeaDoc benchmark — described as the most difficult visual document retrieval task to date in low-resource Southeast Asian languages — is used to validate that continual generative pretraining before contrastive learning improves downstream retrieval.

Main Findings

• Anisotropy drops across modalities after text-only training. Using Qwen2.5-Omni-3B, the authors measure anisotropy as the expected cosine similarity between random embedding pairs. The vanilla model shows representation degeneration in all modalities; after text-only contrastive learning on NLI anchor-entailment pairs, text, image, audio, and video embeddings all become more isotropic, even though non-text modalities received no direct supervision.
• Cross-modal kernel alignment improves and scales with model size. Measured with a mutual kNN overlap metric on Qwen2.5-VL-Instruct 3B and 7B variants (7B has 28 layers, 3B has 36 layers), cross-modal vision-language kernel alignment improves after text-only contrastive learning, and the 7B variant shows consistently stronger alignment than the 3B variant both before and after fine-tuning.
• LCO-Emb sets a new state of the art on MIEB-Lite with far less data. On the 51 tasks of MIEB-Lite, the multimodal LCO-Emb-Omni (7B) reaches a mean of 68.8, followed by LCO-Emb-VL (7B) at 67.6 and LCO-Emb-Omni (3B) at 65.3, versus GME (7B) at 64.5 (trained on ~8M pairs) and mmE5 (11B) at 61.8. The authors note this uses roughly 0.37M training pairs, about 21× less data than GME.
• Text-only variants are already competitive. Trained on only ~276k text triplets, LCO-Emb-VL (7B) scores 60.4 and LCO-Emb-VL (3B) scores 58.0 on MIEB-Lite, surpassing the proprietary Voyage Multimodal 3 at 58.1 and far exceeding E5-V (276k) at 51.9 and VLM2Vec-LoRA (662k) at 46.0.
• Adding ~94k multimodal samples calibrates downstream performance. Going from ~276k text-only triplets to ~370k total triplets lifts LCO-Emb-VL (3B) from 58.0 to 62.3 and LCO-Emb-VL (7B) from 60.4 to 67.6 on MIEB-Lite.
• Large gains on MIEB-Sub18 for text-only variants. On the 18-task subset, text-only LCO-Emb outperforms E5-V and Voyage-Multimodal-3 by 21.69 and 13.00 points on average, with margins of 21.02, 10.26, and 15.35 points on Linear Probing, Cross-lingual Visual STS, and Multilingual Image Retrieval respectively.
• Different text corpora produce different strengths, and merging them helps. On MIEB-Sub18, the all-NLI variant leads on Visual STS and Document Understanding, the Scale-1M variant leads on Linear Probing and Multilingual Image Retrieval, and the model-soup ensemble (LCO-Emb-Ens) achieves the best overall average of 72.17 versus 71.17 (all-NLI) and 71.11 (Scale-1M).
• LoRA beats full fine-tuning, projection, and CLIP-style contrastive training — at a fraction of the cost. With Qwen2.5-VL-7B, LoRA (text-only) averages 71.98 in ~9.3 GPU hours, versus full fine-tuning at 66.49 in ~17.3 hours, a linear projection at 56.22 in ~8.8 hours, and CLIP-style contrastive learning on 800K PixmoCaps image-caption pairs at 50.02 in ~550.0 hours. The CLIP-style baseline requires about 50× more training time.
• GRSL holds across three modality families. Positive correlation appears between pre-contrastive-learning generative scores and post-contrastive-learning representation scores for OCR-based image-text tasks (generation measured via TextVQA, DocVQA, OCRBench, ChartQA; representation via V-STS-English and Document Understanding), video-text tasks (generation via Video-MME w/ sub and MVBench; representation via Recall@1 on MSR-VTT and ActivityNet), and audio-text tasks (generation via MMAU and VoiceBench; representation via Recall@1 on Clotho and AudioCaps).

Methodology in Plain English

The authors start by asking whether an MLLM's internal representations for different modalities are already aligned before any multimodal embedding training. They test this by taking the language decoder out of an MLLM, fine-tuning it with LoRA on text-only contrastive pairs, merging the LoRA weights back, and then measuring the embedding geometry of the whole model. Two measurements do the work: anisotropy (how tightly random embeddings cluster together — lower means more discriminative) and kernel-level similarity (how much the nearest-neighbor structure of one modality's embeddings overlaps with another's).

Because the text-only update improves non-text modalities too, the authors conclude that alignment is a side effect of generative pretraining and that contrastive learning only needs to activate it rather than create it. LCO-Emb implements this: freeze the modality encoders and the projector, attach LoRA adapters to the language decoder, train with contrastive loss on text pairs (~276k NLI triplets, or 1M pairs curated from 20M multilingual parallel corpora), then optionally mix in ~94k synthetic image-text and interleaved pairs for a ~370k triplet multimodal variant.

For the scaling law, they run the same contrastive refinement on backbones of different sizes and generation strengths, plot generative benchmark scores against the resulting representation scores, and then formalize the pattern in a PAC-Bayesian bound where the pretrained generative prior P and the post-contrastive posterior Q appear together. The bound decomposes into a Generative Bottleneck term (log N − I_P(X;Y)), an Inefficiency Gap (ε_P), and a PAC-Bayes Complexity Penalty. They then check the prediction — that better generative pretraining should raise the ceiling — by continuing OCR-intense generative pretraining in low-resource Southeast Asian languages before contrastive learning, and observing improved retrieval on SeaDoc.

Why This Matters

Impact on research. The paper reframes contrastive learning for MLLMs from "the expensive alignment stage" into "a lightweight post-hoc refinement stage," which shifts the research question from how to scale paired cross-modal data to how to preserve and activate alignment that pretraining already produced. GRSL, if it holds, gives the multimodal representation community a reason to invest in generative capability as a route to embedding quality, and the PAC-Bayesian bound gives that claim a formal target. The finding that LoRA outperforms full fine-tuning also supports a broader argument that aggressive fine-tuning damages pretrained structure.

Real-world applications (each grounded in benchmarks used here):

• Multilingual image retrieval, where LCO-Emb shows a 15.35-point margin over prior methods.
• Visual document retrieval and document understanding for low-resource languages, the use case behind the new SeaDoc task.
• Audio-text retrieval and video-text retrieval, evaluated via Recall@1 on AudioCaps and Clotho and on MSR-VTT and ActivityNet.
• Embedding-backed search, clustering, and zero-shot classification (measured with NMI and accuracy), where MLLM-based models have historically lagged CLIP-style models and LCO-Emb reports competitive results.

Industry relevance. The efficiency numbers matter commercially: ~9.3 GPU hours for the 7B LoRA variant versus ~550.0 GPU hours for the CLIP-style multimodal baseline, plus roughly 21× less training data than GME, makes state-of-the-art multimodal embeddings far cheaper to produce and retrain. A framework that works from text-only data is also attractive where paired multimodal corpora are scarce or expensive to license.

Future Directions

• Determine how far generative scaling can be pushed for representation gains. GRSL predicts that better generative models yield better embeddings, so the natural test is whether the trend continues at larger scales and whether continual generative pretraining is a general recipe or specific to the low-resource OCR setting demonstrated with SeaDoc.
• Tighten or refute the theoretical bound. Theorem 1 rests on Hypothesis 1 (Generative Warm Start) and on the approximation L_g(P) ≈ H(Y) − I_P(X;Y); verifying these assumptions empirically and characterizing the Inefficiency Gap ε_P would strengthen or qualify the scaling claim.
• Explain why larger models align better. The 7B Qwen2.5-VL variant shows stronger cross-modal kernel alignment than the 3B variant, which the authors attribute to expanded parameter space — but the 7B has 28 layers and the 3B has 36, so architecture and scale are entangled.
• Extend language-centric refinement beyond the tested modality set and datasets. The paper uses text, image, audio, and video with datasets including Pixmo Cap, AudioCaps, MSR-VTT, and Clotho; whether the approach generalizes to other modality combinations or to interleaved data at larger scale is left open.

Target Audience

Researchers and engineers working on multimodal embeddings, retrieval, and MLLM fine-tuning — particularly those who train sentence/document embedding models or build cross-modal search systems. The anisotropy and kernel-alignment analyses suit representation-learning researchers interested in why MLLM embeddings behave differently from CLIP-style ones, while the PAC-Bayesian bound and GRSL will interest theoretically inclined readers studying the link between generative and discriminative objectives. Practitioners with limited multimodal paired data or limited GPU budgets will find the LoRA-only recipe and the GPU-hour comparisons directly actionable.

Authors’ abstract

Recent multimodal embedding approaches leveraging multimodal large language models (MLLMs) fine-tuned with contrastive learning (CL) have shown promising results, yet the underlying reasons behind their superiority remain underexplored. This work argues that a crucial advantage of MLLM-based approaches stems from implicit cross-modal alignment achieved during generative pretraining, where the language decoder learns to exploit multimodal signals within a shared representation space for generating unimodal outputs. Through analysis of anisotropy and kernel similarity structure, we empirically confirm that latent alignment emerges within MLLM representations, allowing CL to serve as a lightweight refinement stage. Leveraging this insight, we propose a Language-Centric Omnimodal Embedding framework, termed LCO-Emb. Extensive experiments across diverse backbones and benchmarks demonstrate its effectiveness, achieving state-of-the-art performance across modalities. Furthermore, we identify a Generation-Representation Scaling Law (GRSL), showing that the representational capabilities gained through contrastive refinement scales positively with the MLLM's generative capabilities. This suggests that improving generative abilities evolves as an effective paradigm for enhancing representation quality. We provide a theoretical explanation of GRSL, which formally links the MLLM's generative quality to the upper bound on its representation performance, and validate it on a challenging, low-resource visual-document retrieval task, showing that continual generative pretraining before CL can further enhance the potential of a model's embedding capabilities. Codes, models, and resources are available at https://github.com/LCO-Embedding/LCO-Embedding.