TextME: Bridging Unseen Modalities Through Text Descriptions

AI summary

Overview

Research area: Multimodal representation learning and modality expansion — specifically, aligning pretrained contrastive encoders from many different data modalities into one shared embedding space.

Technical level: Intermediate. The paper assumes familiarity with contrastive encoders (CLIP-style models), embedding spaces, retrieval metrics, and the "modality gap" phenomenon, but its core idea is geometrically simple enough to follow with introductory machine learning background.

Scope: The paper introduces TextME, a framework that expands a multimodal system to new modalities using only unpaired text descriptions, by exploiting the geometric structure of pretrained contrastive encoders and projecting them into an LLM embedding space as a unified anchor. (arXiv:2602.03098v2 [cs.LG]; code at https://github.com/SoyeonHH/TextME)

What This Paper Is About

Building multimodal systems that connect images, audio, 3D shapes, X-rays, or molecules normally requires large-scale paired datasets — for example, matched text–image or text–molecule pairs — which are expensive and often infeasible in domains needing expert annotation. TextME asks whether this paired supervision can be eliminated entirely, training cross-modal projection networks using only text descriptions so that pretrained encoders from six different modalities can be aligned into a single shared space.

Key Contributions

1. A text-only modality expansion framework. TextME is presented as the first text-only modality expansion framework (to the authors' knowledge). It exploits modality gap geometry to learn cross-modal projections using only text descriptions, eliminating the need for paired multimodal supervision during training.

2. LLM embedding space as a unified anchor. The paper investigates using LLM embedding space as a unified anchor for modality expansion and compares it against multimodal encoder representations, analyzing their varying effectiveness across tasks and modalities.

3. Empirical validation across six modalities. The framework is validated on image, video, audio, 3D, X-ray, and molecule data on both cross-modal retrieval and zero-shot classification, showing competitive performance against paired-data methods and identifying encoder characteristics that predict when text-only expansion works.

4. Emergent cross-modal retrieval. The unified anchor space enables retrieval between modality pairs never explicitly aligned during training (for example, audio-to-image and 3D-to-image), demonstrating that text-anchored alignment can create semantic bridges across arbitrary modalities.

Main Findings

• Text-only training preserves a large share of pretrained performance. TextME achieves an average of 74.5% PPR (Performance Preservation Ratio) across all tasks, with classification at 89.2% consistently outperforming retrieval at 65.3%.

• Some tasks exceed the pretrained baseline. 3D zero-shot classification surpasses pretrained Uni3D with 104.6% PPR on ModelNet40, and reaches 99.9% PPR on ScanObjectNN.

• Order-of-magnitude reduction in supervision. The results are obtained using only 100K text descriptions, compared to the 1–10M paired samples used by methods such as Ex-MCR (1M pairs) and LanguageBind (10M pairs). The conclusion states TextME reduces data requirements by over 95%.

• Centroid estimates need very little data. 5K samples suffice for stable centroid estimation across all evaluated modalities, which the paper describes as less than 5% of typical paired training requirements.

• Emergent cross-modal retrieval works. TextME outperforms Ex-MCR on 3D→Image retrieval despite Ex-MCR requiring paired supervision, and achieves comparable performance to paired-data methods on Audio→Image. Qualitatively, audio queries retrieve semantically coherent 3D models and molecular queries retrieve contextually relevant images.

• Geometric properties predict success. Variance in "gap-content orthogonality" correlates negatively with performance preservation (Pearson r = −0.67, p < 0.001, over 33 data points). Encoders with low variance — Uni3D at ±0.04 and ViCLIP at ±0.06 — preserve above 80% of performance, while higher-variance encoders such as CLAP at ±0.15 and MoleculeSTM at ±0.18 degrade more.

• Gap consistency is the gatekeeper. Gap consistency exceeds 0.96 for five of six modalities, but MoleculeSTM shows only 0.78, and this weak consistency corresponds to a distinct failure mode.

• Offset correction helps strongly where consistency holds. ModelNet40 classification rises from 4.05% to 70.86% (+94.30%), ScanObjectNN from 5.40% to 42.15% (+87.20%), AudioCaps retrieval from 8.68% to 15.35% (+43.50%), Clotho from 4.77% to 7.81% (+38.90%), and RSNA from 31.35% to 46.59% (+32.70%). Molecule on DrugBank drops slightly from 36.44% to 34.75% (−4.60%).

• Anchor space choice is task-dependent. LLM-based anchors lead on retrieval (NV-Embed-v2 at 16.20 and Qwen3-Embedding at 15.35 R@1 on AudioCaps, versus 14.54–15.91 for multimodal anchors), while multimodal anchors lead on classification (CLIP at 86.70 on Audio, LanguageBind at 81.12 on 3D). Qwen3-Embedding is adopted as the default for balance.

• Domain-specific captions matter enormously. Training on the general-purpose all-NLI corpus (MNLI + SNLI, 100K sentence pairs) instead of domain captions drops 3D Top-1 from 70.86% to 12.10%; improvements from domain captions are +141% (Audio R@1), +485% (3D Top-1), +107% (X-ray Top-1), and +116% (Molecule MRR).

• LLM embeddings separate matched from unmatched descriptions better. On 3K semantically equivalent cross-modal description pairs from FlickrNet, LLM embeddings (Qwen) show 0.56 mean cosine similarity for matched pairs versus 0.23–0.26 for unrelated pairs. On Semantic Textual Similarity benchmarks, LLM embeddings reach Spearman correlations of 85–90 compared to 67–68 for multimodal encoders.

• Detailed retrieval metrics. In the appendix, TextME's R@1/R@5 results with PPR are: COCO 28.63/54.81 (PPR 59.3/75.6), Flickr30k 51.66/77.90 (PPR 66.5/82.7), MSRVTT 26.40/50.50 (PPR 71.4/79.3), MSVD 45.82/77.01 (PPR 89.7/98.4).

Methodology in Plain English

The approach rests on a known property of contrastive encoders: text embeddings and modal embeddings (images, audio, and so on) sit in different regions of the same space, separated by a consistent offset the authors call the modality gap. The pipeline has four practical steps:

1. Estimate the offsets. For each modality encoder, compute the average (centroid) of text embeddings and the average of modal embeddings using independently sampled, unpaired data — 5K samples proved sufficient. The difference between these two centroids is the offset.

2. Build an interchangeable space. Subtract each distribution's own centroid from its embeddings. The paper reports that after centering, a text embedding and its matching modal embedding become approximately equal, because centering removes the modality-specific bias while preserving semantic content.

3. Train projections on text only. A lightweight projection network per modality maps centered text embeddings into a shared anchor space — the LLM embedding space, using Qwen3-Embedding by default. Training uses a contrastive alignment loss with hard negative mining, matching projected centered text embeddings to their LLM embeddings. No modal samples are needed.

4. Transfer at inference. For a new modal input, subtract the precomputed modal centroid and pass the result through the same projection network. Because centering places the modal embedding where text embeddings lived during training, the text-trained projection generalizes without modification, enabling zero-shot cross-modal retrieval and classification.

The authors also diagnose each encoder with four geometric measurements — intra-modal independence, gap consistency, bounded deviation, and gap-content orthogonality — and correlate these with downstream performance to explain why some modalities transfer well and others do not.

Why This Matters

Impact on research. The work reframes modality expansion from a paired-data problem into a geometry problem. It shows the modality gap, previously analyzed mainly in vision-language models and paired settings, generalizes across image, video, audio, 3D, X-ray, and molecule encoders, and that text alone can serve as a semantic bridge between encoders that share no overlapping modality and no shared backbone. It also supplies a practical diagnostic (gap consistency, orthogonality variance) for deciding in advance whether text-only expansion is likely to work for a given encoder.

Real-world applications:

• Medical imaging. The X-ray results (CXR-CLIP encoder, RSNA benchmark at 46.59% versus a 52.64% pretrained baseline) point toward clinical systems that avoid costly expert-annotated paired data and the privacy constraints the paper cites.
• Drug discovery and molecular analysis. Molecule retrieval on DrugBank (34.75% MRR, 79.19% pretrained) and the Molecule→Image emergent retrieval case show a route for linking molecular structures to other data types, though the paper flags MoleculeSTM's weak gap consistency as a limitation.
• 3D content and asset search. Strong 3D classification (104.6% PPR on ModelNet40) plus 3D→Image retrieval on Objaverse supports text-driven search over 3D model collections.
• Audio and accessibility. Text-to-audio retrieval on AudioCaps and Clotho, and audio classification on AudioSet and ESC-50, support search and captioning over sound libraries without paired audio-text data.

Industry relevance. The headline economic claim is data cost: 100K text descriptions instead of 1–10M paired samples, and no labeled target data at all (unlike the unpaired baseline COX, which required 10K labels). For teams building multimodal retrieval or assistant products in specialized verticals, that changes modality expansion from an annotation project into a lightweight adaptation step, since pretrained encoders are frozen and reused with only small projection networks trained.

Future Directions

• Per-instance corrections instead of a single offset vector. MoleculeSTM's gap consistency of 0.78 shows that one offset vector can inadequately

Authors’ abstract

Expanding multimodal representations to novel modalities is constrained by reliance on large-scale paired datasets (e.g., text-image, text-audio, text-3D, text-molecule), which are costly and often infeasible in domains requiring expert annotation such as medical imaging and molecular analysis. We introduce TextME, the first text-only modality expansion framework, to the best of our knowledge, projecting diverse modalities into LLM embedding space as a unified anchor. Our approach exploits the geometric structure of pretrained contrastive encoders to enable zero-shot cross-modal transfer using only text descriptions, without paired supervision. We empirically validate that such consistent modality gaps exist across image, video, audio, 3D, X-ray, and molecular domains, demonstrating that text-only training can preserve substantial performance of pretrained encoders. We further show that our framework enables emergent cross-modal retrieval between modality pairs not explicitly aligned during training (e.g., audio-to-image, 3D-to-image). These results establish text-only training as a practical alternative to paired supervision for modality expansion.