Multimodal Dual Encoders and Shared Embedding Spaces

Each modality is processed independently until the very last step. An image, audio clip, document or molecule becomes a vector. A paired item from another modality becomes a second vector. Both are projected and normalized into a shared comparison space. A contrastive objective moves matched pairs closer relative to selected negatives. Then one side is precomputed and indexed, so nearest-neighbor search can run at corpus scale. Only the similarity score at the end sees both modalities at once.

The best-known instance of this pipeline is CLIP, published in 2021 and trained on “a dataset of 400 million (image, text) pairs collected from the internet”. Its headline transfer result is a retrieval construction, not a fitted classifier. CLIP's models “match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on”. The classifier was never trained. It was looked up.

What the architecture buys was measured in text before it was claimed across modalities. A simple two-tower dual encoder, trained on a small number of question-passage pairs, beat a strong lexical baseline outright. Karpukhin and seven co-authors reported it in 2020: “our dense retriever outperforms a strong Lucene-BM25 system greatly by 9%-19% absolute in terms of top-20 passage retrieval accuracy”. Hold on to the shape of that claim. A two-tower encoder. A small number of recorded pairs. A margin over a lexical baseline, stated in absolute points, at a named retrieval depth. Every later section is a way of asking what that margin is made of.

A positive pair is encouraged to have higher similarity than mismatched pairs. In-batch negatives make the objective efficient, because other examples in the batch provide the competing candidates.

The resulting geometry depends on pair construction, batch composition, temperature, and duplicate semantics. A “negative” can be genuinely relevant even when it is not the recorded pair. That is not a caveat. It is a measured rate.

The BEIR benchmark counted it. On TREC-COVID it asked how often a retriever's top-10 hits had never been judged by an annotator at all. That rate — Hole@10 — was 6.4% for BM25 and 2.8% for docT5query. For DPR it was 30.6%, and for TAS-B 31.8%. A section titled “Impact of Annotation Selection Bias” puts it this way: “In contrast, dense retrieval systems like ANCE and TAS-B have a much higher Hole@10 of 14.4% and 31.8%, indicating that a large fraction of hits found by these systems have not been judged by annotators.”

They then paid to find out what was in the hole. 980 more query-document pairs were annotated by hand. DPR's nDCG@10 rose from 0.332 to 0.445, TAS-B's from 0.481 to 0.555. Nothing about either encoder changed. The documents the score had been treating as wrong answers were relevant documents the judgment pool had never seen. It is the same asymmetry the training objective builds in when it calls every unpaired item in the batch a negative.

Independent encoding enables indexing. Joint processing provides richer pair interaction. A dual encoder scores pairs through vector similarity, so corpus embeddings can be precomputed and searched approximately at speed. The cost is limited token-to-region interaction. It is excellent at candidate generation. A cross encoder processes both candidates jointly before scoring. That gives fine-grained interactions and higher pair accuracy. But the full pair score cannot be precomputed, and it is expensive over a large corpus. A two-stage system retrieves broadly and reranks jointly. It buys scalable recall and a rich final comparison. It pays in pipeline error propagation and in the need for stage-specific metrics.

The fourth option is the one usually left vague. Late interaction stores per-token embeddings for each document and scores them by MaxSim. Some precomputation is retained, and more detailed matching survives. Omar Khattab and Matei Zaharia introduced it in ColBERT in 2020. Their abstract: “Results show that ColBERT's effectiveness is competitive with existing BERT-based models (and outperforms every non-BERT baseline), while executing two orders-of-magnitude faster and requiring four orders-of-magnitude fewer FLOPs per query.”

The price of storing many vectors per document is the index, and BEIR later measured it. On the roughly 15-million-document BioASQ collection, ColBERT needed about 900GB against 18GB for BM25. On 1 million DBPedia documents, 20GB against 0.4GB. BEIR also timed it at 350ms per query on GPU, against 14–20ms for dense retrievers. "Larger index footprint" is a factor of fifty on one collection and about eighteen on the other. That number, not the effectiveness table, decides whether late interaction is affordable for your corpus.

The recorded relationship may be weaker or noisier than the product requires. That gap has been counted on the very benchmark image-text retrieval is trained and scored on. MS-COCO records 5 positive captions per image query. A re-annotation with machine and human verification found 17.9. MS-COCO records 1 positive image per caption query. The re-annotation found 8.5. It was published in 2022 as ECCV Caption, and states the gap as a multiplier: “Our dataset provides ×3.6 positive image-to-caption associations and ×8.5 caption-to-image associations compared to the original MS-COCO.”

Read that against the objective in the previous section. In the caption-to-image direction, for every one match the benchmark recorded there were on average 8.5 true ones. So the great majority of the images the loss pushed away from a caption were images that actually answered it. The pairing rule was not slightly noisy. It was systematically incomplete, in the same direction, everywhere.

Offline similarity accuracy is only the beginning of a deployable retrieval system. Define relevance so that graded or multiple valid matches are allowed where the task requires them. Audit negative construction for duplicates, near-duplicates, exposure effects, and false negatives. Separate candidate metrics from rerank metrics, tracking recall at retrieval depth and quality after reranking. Test subgroup geometry across languages, accents, image domains, and rare concepts. Benchmark the index for memory, update latency, approximate-search recall, and filtering constraints.

Steps three and four are the ones a paper's headline number quietly skips, and both have documented cases. Take the third. DPR was replicated in 2021 from the original authors' checkpoints, but with the replicators' own Pyserini and PyGaggle implementations — the work of Ma and three colleagues at the University of Waterloo. Their abstract begins: “Although our experimental results largely verify the claims of the original paper, we arrived at two important additional findings that contribute to a better understanding of DPR”. One of those findings was that the original paper under-reported the effectiveness of its own BM25 baseline. Part of the 9%-19% margin from the first section was therefore baseline weakness rather than encoder strength. No amount of re-running the dense side would have exposed that.

Now the fourth. BEIR evaluated 10 retrieval systems on 18 datasets. Dual-encoder dense retrievers generalized worst out of domain. BM25 was a robust baseline. The cross-attentional reranker beat it on 16 of 18 datasets. Of late-interaction ColBERT the BEIR authors write: “It performs a bit weaker than the cross-attentional re-ranking model, but is still able to outperform BM25 on 9/18 datasets.” DPR performed worst in generalization of all systems tested. Domain shift, not architecture, decided the ranking. That is why subgroup and out-of-domain geometry belongs in the evaluation plan, and not in the discussion section.

Cosine or dot-product similarity orders candidates inside the learned space. Its absolute value can shift with temperature, normalization, domain, and candidate pool.

If a product needs accept/reject decisions, calibrate thresholds on representative queries and corpus conditions. Candidate-set changes can alter the operating point. That holds even when the encoder is frozen. BEIR's TREC-COVID re-annotation moved TAS-B's nDCG@10 from 0.481 to 0.555 without touching a weight. All that changed was 980 more judged pairs. If a fixed encoder's measured quality can move that far on the labels alone, a fixed threshold has no claim to stability when the corpus itself changes.