EmbeddingGemma 2: Unified and Efficient Multimodal Representations

AI summary

Overview

Research area: Machine learning — multimodal representation learning and embedding models (information retrieval, semantic search, retrieval-augmented generation).

Technical level: Intermediate. The paper is readable with a general grasp of transformer embeddings and contrastive training, but it assumes familiarity with losses, quantization, and benchmark suites.

Scope (one sentence): The paper describes EmbeddingGemma 2, a 744M-parameter open embedding model that places text, images, video, audio, and interleaved combinations into one shared vector space, and reports its architecture, three-stage training recipe, ablations, and benchmark results against specialist and larger general models.

What This Paper Is About

Text embedding models need bolt-on tools such as automatic speech recognition and image captioners to handle audio and visual data, which adds cost and loses fine-grained visual detail and non-verbal sound. Multimodal embedding models built from large multimodal language models can handle interleaved input, but they typically run to several billion parameters, which makes latency-sensitive, high-throughput, or on-device use impractical. The goal is a small, open model that natively embeds all of these modalities into a single space without giving up quality.

Key Contributions

1. A modular 744M-parameter multimodal embedder. The model combines a 271M-parameter bidirectional transformer backbone (embedder included), a 167M-parameter vision encoder, and a 305M-parameter audio encoder. Because the encoders are separate, the same release can run as a 271M text-only model, a 439M vision-language model, or a 577M audio-language model, so users load only the encoders they need.
2. Simplified per-layer embeddings (PLE). Instead of the dedicated PLE token lookup table used in Gemma 3n and Gemma 4, EmbeddingGemma 2 derives per-layer inputs only from the input token embeddings produced by the main embedding table and the modality encoders. The ablation reports this is both cheaper and better.
3. A staged curriculum with staged objectives. Training runs over roughly 2.9T tokens in three stages (~300B text pre-finetuning, ~2T multimodal pre-finetuning, ~600B multimodal finetuning), moving from almost purely distillation to contrastive, spread-out, and Matryoshka losses, and finishing with checkpoint merging and a separate quantization-aware training stage.
4. Deployment knobs plus ablations. Selective modality encoder loading, configurable vision token budgets (70, 140, 280, 560, or 1120 at inference), truncatable embedding dimensions (768, 512, 256, 128 via Matryoshka Representation Learning), and int4 quantized checkpoints, supported by ablations on PLE, training stage, vision encoder freezing, and token budgets.

Main Findings

• New Pareto frontier claim: The authors report that EmbeddingGemma 2 establishes a new Pareto frontier across benchmarks covering code, image, video, visual documents, and audio, outperforming both specialist models and models two to twelve times larger, evaluated on seven benchmark suites (MTEB for text and code, MMEB-v2 and MIEB-lite for image/video/visual documents, MSEB and MAEB for sound and audio).
• Text improves over its predecessor on multilingual, regresses slightly on English: MTEB(Multilingual, v2) Mean(Task) 61.36 versus 61.15 for EmbeddingGemma 1 (driven by gains in bitext mining and classification), while MTEB(English, v2) Mean(Task) is 68.46 versus 69.67. Mean(Type) figures are 53.82 versus 54.31 (multilingual) and 63.86 versus 65.11 (English). The paper notes only two smaller models surpass it on multilingual text and one on English, all specialist text models.
• Code is a large gain: MTEB(Code) Mean(Task) 78.68 versus 68.76 for EmbeddingGemma 1 and 75.42 for Qwen3 Embed 0.6B.
• Large multimodal gains against a bigger general peer: Against jina-embeddings-v5-omni-nano (986M parameters), MMEB-v2 Mean(Task) is 59.01 versus 49.33, with image 57.28 versus 55.43, video 50.67 versus 35.87, and visual documents 67.84 versus 50.26. On text, however, the peer is stronger (MTEB Multilingual 65.52 versus 61.36; English 71.11 versus 68.46).
• Image benchmark: MIEB(lite) Mean(Task) 60.17 and Mean(Type) 65.67, versus 44.24 / 49.38 for EmbeddingGemma 1 and 52.28 / 53.50 for Qwen3 Embed 0.6B.
• Audio: MSEB(Retrieval) Mean(Task) 69.54 versus 50.95 for EmbeddingGemma 1 (the latter computed by transcribing audio with ASR). On MAEB the model scores 49.87 Mean(Task) and 48.65 Mean(Type), slightly below EmbeddingGemma 1's 50.14 / 55.31, and above LAION CLAP's 34.98 / 41.08.
• Dropping the PLE lookup table helps: On MMEB-v2, projection-only PLE scores 55.46 overall, 54.67 image, 47.82 video, 62.37 VisDoc, versus 55.07 / 54.57 / 47.13 / 61.76 with a lookup table (+0.69 on video, +0.61 on VisDoc), with similar improvements reported on text-only evaluations.
• Finetuning beats prolonging distillation: Continuing distillation for a compute-equivalent duration moves MMEB-v2 overall only from 52.27 to 52.28, whereas switching to the finetuning recipe (contrastive plus spread-out objectives) reaches 55.46, with image 54.67 versus 50.10 and video 47.82 versus 43.26; visual document results are similar (62.37 versus 62.26).
• Training the vision encoder is critical: A frozen vision encoder scores 48.18 on MMEB-v2 overall versus 52.27 when trainable, with image 44.35 versus 50.10, video 39.24 versus 43.26, and VisDoc 60.59 versus 62.26.
• Vision token budgets behave differently by modality: Scaling inference tokens from 70 to 560 raises overall MMEB-v2 from 50.07 to 57.02; visual documents keep improving well past the 280-token training budget (51.61 at 70 tokens to 67.62 at 560), image quality begins to saturate at 280 (55.15), and video degrades between 280 and 560 (48.51 to 46.52). Training with 70 video tokens helps at 70-token evaluation (49.62 versus 49.76 overall, though better on video prompts) but hurts when evaluated at 140, so the 140-token video budget was chosen.
• Quantization costs little: With Q4_C int4 backbone weights, MTEB(Code) moves from 78.68 (bfloat16) to 76.70, MIEB(lite) Mean(Task) from 60.17 to 56.71, MSEB(Retrieval) from 69.54 to 63.92, while MTEB(English, v2) is essentially unchanged (68.46 to 68.49). Peak on-device memory on a Google Pixel 11 Pro is 191 MB text-only, 447 MB text plus vision, and 567 MB for the full multimodal model.

Methodology in Plain English

Building the model. The team started from a pretrained Gemma 4 decoder, switched its causal attention mask to bidirectional attention so every token can see every other token, and dropped the extra PLE lookup table. After the 24-layer transformer (model dimension 512) they average the output token embeddings (mean pooling), project to 768 dimensions, and L2-normalize to get the final embedding.

Turning media into tokens. Images go through a ViT with patch size 16 and are spatially pooled to a token budget (280 per image and 140 per video frame during training), then wrapped in start/end-of-image delimiter tokens. Audio is fed as Mel filterbank chunks of 40ms producing 25 soft tokens per second, encoded by two downsampling convolutions plus twelve conformer layers, and wrapped in its own delimiters. Video is sampled at one frame per second, capped at 32 frames by truncation or uniform sampling, with MM:SS timestamps and optionally interleaved 1-second audio segments. All of these pieces are concatenated between start/end tokens up to a training sequence length of 8192.

Training in three stages. First, text pre-finetuning on query-target pairs from question answering, sentence similarity, code retrieval, and web search, using almost only distillation from Gemini Embedding 2; the authors treat this stage as curriculum learning and an implicit warm-up for bidirectional attention. Second, multimodal pre-finetuning on a mixture adding images, audio, video, and documents, again purely by distillation, for twice as many steps. Third, multimodal finetuning, where distillation is fully replaced by a contrastive loss with in-batch, hard, and same-tower negatives (plus a listwise Plackett–Luce variant for ranked lists and small squared-hinge penalties for score calibration), a spread-out regularizer that pushes random pairs toward orthogonality, and Matryoshka losses over overlapping sub-dimensions. The vision encoder is trained in the two multimodal stages; the audio encoder is taken bit-identical from Gemma 4 and kept frozen.

Finishing touches. The final checkpoint is an unweighted average of the last finetuning checkpoint and a significantly earlier one from the same run, which the authors say mitigates overfitting degradation. Quantized versions are produced by a separate post-merging stage that distills a QAT checkpoint from the frozen float model, rather than applying QAT during finetuning.

Evaluation protocol. They compare against top-ranked baselines under 1B parameters on MTEB, MAEB, and MIEB(lite) and under 2B on MMEB-v2, restrict to single-vector models, exclude models trained on more than 25% of the evaluation data, run EmbeddingGemma 2 in bfloat16, and count only the parameters actually used for each benchmark since most baselines support fewer modalities.

Why This Matters

The result suggests a small open model can cover text, code, image, video, documents, speech, and environmental sound in one embedding space without the multi-billion-parameter cost that has been typical of multimodal embedders adapted from large multimodal language models. That changes who can deploy multimodal retrieval: the paper reports a full multimodal footprint of 567 MB on a Pixel 11 Pro with int4 weights, which is on-device territory rather than datacenter territory.

Real-world applications implied by the tasks the model was trained and evaluated on:

• On-device and offline search over personal photos, screenshots, voice notes, and local documents without sending data to a server.
• Visual product and cropped-object search in shopping catalogues, one of the finetuning task families listed.
• Video moment retrieval and video ranking, supported by the frame-plus-timestamp and interleaved-audio input format.
• Audio classification and speech content retrieval without a separate ASR pipeline, which the authors explicitly compare against.
• Code search and tool retrieval for developer assistants, where the model shows its largest text-side gain (MTEB(Code) 78.68 versus 68.76).

Industry relevance: selective encoder loading, adjustable vision token budgets, truncatable dimensions, and shipped int4 checkpoints are all production concerns, and the ablations give teams concrete guidance — notably that freezing the vision encoder costs about 4 points on MMEB-v2 overall and that higher vision token budgets mainly pay off for visual documents.

Future Directions

• Training the audio encoder. The paper keeps it frozen and states that analogous gains to the vision-encoder ablation are likely possible from further training, which is an obvious next experiment.
• Higher train-time vision token budgets. The authors note more gains are likely possible but come with seemingly diminishing returns and significant computational cost, leaving the right budget an open trade-off.
• Closing the text gap. EmbeddingGemma 2 leads on multimodal and code benchmarks but trails specialist and general peers on MTEB text scores (61.36 multilingual versus 65.52 for a 986M peer; 68.46 English versus 69.67 for its own predecessor), so recovering English text quality while sharing one space remains open.
• Methodological details not covered here. The provided text is truncated after the start of the text evaluation section, so the paper's per-benchmark discussions for images, video, visual documents, and audio, and its appendices on task instructions, are not reported in this summary.

Target Audience

Practitioners building retrieval, search, recommendation, or RAG systems who need one embedding model across mixed media and care about memory and latency budgets; engineers working on on-device or privacy-preserving ML; and researchers studying multimodal representation learning, distillation from large teachers, Matryoshka embeddings, and quantization-aware training. Readers wanting reproducible architecture and training-re

Authors’ abstract

We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.