UEmbed: Unified Sparse and Dense Multimodal Embeddings

AI summary

Overview

• Research area: Information retrieval and multimodal representation learning, specifically learned sparse retrieval (LSR) combined with dense embedding models built on multimodal large language models (MLLMs).
• Technical level: Intermediate. The paper assumes familiarity with contrastive retrieval training (InfoNCE), SPLADE-style sparse retrieval, and decoder-only transformer architecture.
• Scope: The paper introduces UEmbed, a decoder-only multimodal embedding model that produces both sparse lexical and dense representations in a single causal forward pass, released at 2B, 4B, and 9B scales and evaluated on MMEB-v2, BEIR, and BrowseComp-Plus.

What This Paper Is About

Learned sparse retrieval (LSR) has historically depended on encoder-style bidirectional architectures such as BERT, and extending it to images and video has required auxiliary cross-modal modules. This leaves a gap: decoder-only MLLMs cannot reuse standard max-pooling over hidden states because causal attention prevents each position from seeing future tokens, and no prior work has produced sparse multimodal representations natively from a single causal backbone. UEmbed's goal is to close that gap by appending special tokens to a decoder-only MLLM so that sparse and dense embeddings emerge from the same forward pass, across text, image, video, and visually-rich document inputs.

Key Contributions

1. A unified decoder-only embedding model. UEmbed produces both sparse and dense retrieval representations from one causal forward pass of a single backbone, eliminating the need for bidirectional encoders or separate sparse and dense models.
2. A partitioned sparse head that bypasses the single-token bottleneck. Instead of projecting one token (e.g., EOS or CLS) into a vocabulary of size |V|, UEmbed appends N=16 learnable special tokens and assigns each a disjoint vocabulary subset derived by k-means clustering, then concatenates the subsets into the full sparse vector.
3. The first sparse model evaluated on MMEB-v2. To the authors' knowledge, these are the first reported sparse multimodal embedding results on that benchmark, and UEmbed sets a new state of the art in the sparse multimodal retrieval setting.
4. A three-dimensional practical evaluation. The paper examines effectiveness (hybrid dense-plus-sparse scoring), efficiency (autoregressive serving compatibility with vLLM and native inverted-index use), and agentic search, rather than reporting accuracy alone.

Main Findings

• Multimodal retrieval (MMEB-v2). UEmbed-9B scores 71.8 dense and 71.0 sparse, which the authors state leads models trained on publicly available data in dense retrieval and sets the state of the art for sparse retrieval. It outperforms RzenEmbed-V2-7B (71.1) and Ops-MM-Embed-7B (67.1). Qwen3-VL-Embedding-8B reaches 77.8, but the authors note it is trained with vast proprietary data, making direct comparison challenging.
• Scale consistency. UEmbed-4B (dense) scores 70.4, described as surpassing every other model in the 4B class, such as Embed-RL-4B (68.1). UEmbed-2B (dense) scores 66.5, which is competitive with or above some 7B-scale open baselines such as UniME-7B (64.1). UEmbed-2B sparse scores 65.5, UEmbed-4B sparse 69.7.
• Sparse trails dense by at most one point. The dense-versus-sparse gap is at most 1.0 point at every scale tested (71.8 vs. 71.0 at 9B; 70.4 vs. 69.7 at 4B; 66.5 vs. 65.5 at 2B).
• Sparse is strongest on visually-rich documents. On the VisDoc category the drop is minimal (79.2 dense vs. 79.1 sparse at 9B), while the gap is slightly more pronounced in the Video category.
• Text retrieval (BEIR, 9 datasets, nDCG@10). UEmbed-9B dense achieves the highest average among compared models at 56.3, with UEmbed-4B at 56.0, ahead of Qwen3-VL-Embedding-8B (55.5) and GME-7B (53.5). In sparse mode, UEmbed-9B averages 55.2, matching Echo-Mistral-SPLADE (55.2) and ahead of SPLADE-v3 (50.5).
• Dataset-level strengths. UEmbed-9B reaches 89.9 on Quora; UEmbed-4B reaches 40.5 on NFCorpus.
• The causal recipe beats a bidirectional SPLADE baseline. On the image subset of MMEB-v1 with a 2B backbone, UEmbed scores 64.5 dense and 63.4 sparse versus 61.3 for the SPLADE baseline sharing the same backbone, data, and FLOPS regularization—gains of 3.2 and 2.1 points. Gains are largest on IMG-QA (+8.2 dense, +6.0 sparse).
• Joint training does not hurt either mode. A jointly trained model closely matches dense-only (λ=0) and sparse-only specialists, indicating negligible negative transfer.
• Semantic clustering is the best partitioning strategy. Semantic clustering scores 63.4 average versus 63.2 for max-distance pairing and 63.0 for random assignment.
• Hyperparameters. Sparse temperature τ_s=32 is the default, with τ_s=64 performing comparably in the reported single-seed sweeps. Sweeping N over {2, 4, 8, 16, 32} shows stable performance up to N=16, with a clear drop at N=32 attributed to overly small vocabulary subsets and inflated contrastive sequence lengths.
• Hybrid scoring helps selectively. Combining dense and sparse similarities raises Text by +0.3 and VisDoc by +0.5, while natural images and video frames gain little because they carry limited surface-form lexical information. UEmbed-2B hybrid scores are 53.9 (TXT), 67.9 (IMG), 50.0 (VID), 77.5 (VDR) versus 53.6/67.8/50.0/77.0 dense and 52.3/67.0/47.7/76.7 sparse.
• Agentic search on BrowseComp-Plus. With DeepResearch-30A3B as the reasoning engine, the sparse mode consistently requires fewer tool-call search rounds than the dense counterpart. UEmbed-9B sparse uses 31.05 average search rounds versus 33.68 for dense, at the same accuracy of 49.76 and recall of 64.33 versus 64.72. Calibration error is lower for sparse at 9B (8.16) and 4B (8.79) than for dense (7.06 at 9B, 14.34 at 4B) — note the 9B case is the exception, with dense lower.
• Training dynamics. When text training data is included, the model learns effective sparse representations in under 100 steps; training exclusively on multimodal data (image, video) requires nearly 500 steps to reliably distinguish positives from negatives.
• Vocabulary compression. Stripping accents, lowercasing, and collapsing whitespace with NLTK reduces the vocabulary from 248,320 to 184,016 entries, retaining the maximum weight among merged tokens.

Methodology in Plain English

Standard sparse retrieval models like SPLADE let every token look at every other token (bidirectional attention), then max-pool across the sequence to decide which vocabulary words to activate. Decoder-only models cannot do this because each position only sees what came before it.

UEmbed sidesteps the problem by appending 16 learnable special tokens to the end of the input. Because they sit last, each special token can attend to the entire input and summarize it. The vocabulary is then split into 16 disjoint subsets using k-means clustering, and each special token owns one subset and predicts sparse weights only over the words in its cluster through a small linear projection with a ReLU and log-saturation. The 16 chunk vectors are concatenated into the final sparse vector, which is large enough to be served by a standard inverted index. Using 16 tokens with partitioned vocabularies avoids squeezing the entire vocabulary prediction through a single token's hidden state.

The dense embedding comes from the hidden state of the EOS token that precedes the special tokens, so dense-only use can skip the special tokens entirely with no extra compute. Training uses a combined loss: an InfoNCE term with cosine similarity for dense, an InfoNCE term with inner product for sparse, and two FLOPS regularizers penalizing the squared mean term weight on the query and document sides to encourage sparsity.

Training data comes from three public sources totaling 3.94M samples: Echo-Embedding training data, MLDR training data, and MMEB training sets. Because multimodal datasets lack negatives, the authors use Qwen3-VL-Embedding-8B as a teacher to mine hard negatives—retrieving top-k most similar non-relevant documents per query. Training uses LoRA on attention and MLP projections with the visual encoder frozen, bf16 mixed precision, DeepSpeed ZeRO, gradient checkpointing, a peak learning rate of 3×10⁻⁵ cosine-scheduled with 0.1 warmup, dense temperature τ=0.03, sparse temperature τ_s=32, α_q=α_d=1×10⁻⁴ with linear ramp-up over the first 200 steps, λ=1.0, one epoch, and a fixed seed of 42. The 2B and 4B models train on 16×A100 GPUs at per-device batch size 16, and the 9B model on 32×A100 GPUs at per-device batch size 8, giving a consistent batch size of 256.

Why This Matters

Impact on research. UEmbed reassigns sparse retrieval from a legacy, encoder-bound technique to a native byproduct of decoder-only MLLMs. It removes two longstanding constraints—the need for bidirectional architectures and the need for auxiliary cross-modal modules to handle non-text modalities—and it reports the first sparse results on MMEB-v2. That opens a line of work on serving sparse and dense retrieval from a single causal checkpoint without converting the backbone or adopting dedicated training curricula.

Real-world applications.

• Web and enterprise search, where sparse lexical matching complements dense semantics and can be served from a standard inverted index.
• Retrieval-augmented generation over text, images, video, and visually-rich documents using one model rather than separate dense and sparse systems.
• Document-heavy domains such as chart, infographic, and scanned-document retrieval, where sparse performance holds closest to dense.
• Cost-sensitive agentic search loops, where keyword-dense agent queries let the sparse mode cut the number of tool-call search rounds.

Industry relevance. The model is purely autoregressive, so embedding generation is compatible with high-throughput serving stacks such as vLLM, and the sparse vectors plug directly into production inverted indices. Using one checkpoint for both retrieval modes removes the operational cost of maintaining and encoding with separate dense and sparse models. Weights are released at 2B, 4B, and 9B under a CC BY 4.0 license.

Future Directions

• Multilingual coverage. Sparse activations are predominantly English and Chinese, reflecting a training corpus skewed toward those languages. Extending sparse capability to more languages requires highly language-diverse, large-scale training data.
• Vocabulary stability and artifacts. Operating over the expansive vocabulary of modern LLMs occasionally produces anomalous tokens such as "_alt". Targeted vocabulary pruning, refined semantic clustering, or post-hoc filtering are proposed to ensure representational stability in strict production environments.
• Video-domain gaps. The sparse-versus-dense gap is more pronounced for video, which the authors attribute to the high information density and temporal dynamics of video frames, suggesting that naively pooling spatiotemporal data into a flat sparse vector may hit capacity bottlenecks.
• Further efficiency analysis. The paper defers detailed analysis of inverted-index search efficiency to Appendix B.3, and the present content does not report latency or throughput measurements.

Target Audience

Researchers and engineers working on information retrieval, multimodal embedding models, and learned sparse retrieval will benefit most, particularly those building retrieval infrastructure on decoder-only LLM or MLLM backbones and those who need both sparse and dense retrieval from a single deployed model. Practitioners concerned with serving efficiency, inverted-index compatibility, or retrieval-augmented agent loops will also find the practical sections relevant. Readers without background in contrastive retrieval or SPLADE-style sparse representations will need to consult the cited prior work first.

Authors’ abstract

Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semantics. Yet LSR has so far remained tied to encoder-style bidirectional architectures, and its extension to multimodal settings still relies heavily on auxiliary cross-modal modules. To address these limitations, we introduce UEmbed (Unified Embedding), a decoder-only multimodal embedding model that produces both sparse lexical and dense representations in one causal forward pass. UEmbed appends N learnable special tokens to the input and partitions the vocabulary into N disjoint subsets. Each token's causal hidden state predicts sparse weights over its assigned subset, and the N subsets are concatenated into the full sparse vector. Trained on public data, we release UEmbed at 2B, 4B, and 9B scales. UEmbed-9B reaches 71.8 (dense) and 71.0 (sparse) on MMEB-v2, outperforming multimodal embedding models trained on publicly available data (e.g., RzenEmbed). On BEIR, UEmbed also remains competitive with strong dense and sparse baselines. Furthermore, we demonstrate the practical utility of UEmbed across three dimensions: effectiveness, efficiency, and agentic applications. Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.