When Does Mixing Help? Analyzing Query Embedding Interpolation in Multilingual Dense Retrieval

AI summary

Overview

Research area: Multilingual natural language processing, specifically multilingual dense retrieval (vector search) and code-mixed / mixed-language queries.

Technical level: Intermediate. The paper assumes familiarity with bi-encoder dense retrievers, embedding similarity, and standard IR metrics such as nDCG@10, but the central idea — interpolating two query vectors — is conceptually simple.

Scope (one sentence): A ratio-controlled study on mMARCO that varies the mixing weight between parallel monolingual query embeddings to test when mixed-language queries beat pure monolingual queries in multilingual dense retrieval.

What This Paper Is About

Multilingual dense retrievers are normally built and tested on queries written in a single language (monolingual retrieval) or fully translated into the document language (cross-lingual retrieval), so mixed-language querying — which is common in bilingual communities — is largely untested. The paper asks a concrete question: when two monolingual query translations are blended at different proportions, is the best result bounded by the better monolingual endpoint, or can an "interior" mixture beat both endpoints? To answer this without expensive and noisy LLM-generated code-mixed text, the authors mix query embeddings (a weighted interpolation of two monolingual query vectors) and measure retrieval performance across 35 language pairs and three document-language settings.

Key Contributions

1. Embedding-level mixing (embed-mix) as a controlled diagnostic. The authors define the mixed query vector as a normalized convex combination of two monolingual query embeddings, with mixture weight λ in [0, 100] interpreted as the percentage of L2. They validate this against LLM word-level code-mixing on the EN–ZH pair and three additional pairs (EN–VI, ZH–VI, HI–ID).
2. A 105-setting ratio sweep. They evaluate 35 language pairs × 3 document-language settings (L1-only, L2-only, L1+L2 union), showing that an optimal interior mixing ratio outperforms the best monolingual endpoint in 88/105 (83.8%) cases.
3. Identification of English as the dominant conditioning factor. They show a systematic asymmetry: mixing is uniformly beneficial when documents do not include English, but indices containing English are best served by pure English queries, and English is the strongest mixing partner for every non-English document language tested (all 13).
4. Robustness checks across models and scales. The patterns are replicated on multilingual-e5-large-instruct, gte-multilingual-base, jina-embeddings-v3, and Qwen3-Embedding-0.6B/4B/8B, in addition to the main BGE-M3 model.

Main Findings

• Mixing usually helps, and the worst harm is small. Across the 105 groups, 88/105 (83.8%) have Δ > 0 and 17/105 (16.2%) have Δ < 0. Mean Δ = +0.70, median Δ = +0.65, with an observed range of −0.34 to +2.92. Under 95% bootstrap intervals, 66/105 settings are reliably positive, 38/105 are indistinguishable from zero, and 1/105 is reliably negative.
• Largest and smallest effects. The largest gain is EN–AR on AR documents (Δ = +2.9203 at λ* = 50); the most negative case is EN–ZH on EN+ZH documents (Δ = −0.3359 at λ* = 10).
• English in the index flips the effect. When documents exclude English (non-EN monolingual and non-EN+non-EN bilingual settings), all groups have Δ > 0 with an average of +0.95. When English is present in the index (EN-only or EN+X bilingual documents), Δ clusters near zero with a mean of −0.04. All negative/zero cases occur when the documents include English.
• Asymmetric English effect. Adding English to ZH queries improves ZH-document retrieval (Δ = +1.72, λ* = 50), but adding ZH to EN queries consistently worsens English-document retrieval. The English effect overrides raw document–query similarity.
• English is the strongest mixing partner. For each non-English document language L, English yields the largest Δ among tested partners on L-only documents, consistently across all 13 languages studied, and surpassing the second-best partner by a large margin. The authors report this holds even for monolingual queries with no English loanwords, and they attribute it to pre-training data imbalance rather than lexical overlap.
• Document–query language match matters, but 100% is not always best. Across all 70 monolingual document settings, the endpoint matching the document language beats the opposite-language endpoint; performance rises as p_doc increases and typically peaks near p_doc ≈ 70, so a 50% mixture nearly matches a fully monolingual query.
• Where the optimum sits. For non-English pairs on monolingual document settings, the peak is typically near p_doc(λ*) ≈ 70 (35/44 settings). For EN pairs optima are bimodal: λ* = 0 in 8/13 of EN-only document settings, while λ* = 50 in 11/13 of non-English document settings. When English is present in the pair and documents are EN+L2, λ* = 0 is the peak.
• Recall benefits more than ranking. Δ computed on different metrics: nDCG@10 mean +0.7037 (88 positive, 17 negative), MRR@10 mean +0.5844 (88 positive, 17 negative), Recall@10 mean +1.2021 (98 positive, 7 negative). In 11 settings ΔnDCG@10 < 0 while ΔR@10 > 0, all within EN pairs on EN-only or EN+L2 document settings.
• Typological distance correlates negatively with mixing gains. On the controlled non-English subset of 44 settings (22 pairs × 2 document languages), typological distance shows a moderate negative correlation (Spearman ρ = −0.405, 95% CI [−0.622, −0.124]).
• Other language metadata have weak effects. Script match: matching pairs gain Δ = 0.814 versus Δ = 0.622 for mismatched pairs, a mean difference of Δ = 0.192 with 95% CI [−0.018, 0.395]. Family distance: Spearman ρ = −0.306, 95% CI [−0.559, −0.019]. Resource level: no monotonic trend, but lower-resource L–L pairs show larger mean gains (Δ = 1.006) than H–H pairs (Δ = 0.560).
• Strong endpoints leave less room for improvement. Δ is negatively correlated with the better endpoint score (Spearman ρ = −0.607). This is driven by settings with English documents (EN pairs: ρ = −0.684); when documents exclude English the correlation is weak (ρ = −0.117).
• Bilingual documents help mainly without English. Comparing bilingual L1+L2 documents against the better monolingual document setting: non-EN pairs mean gain +0.4475 (21/22 positive, 19/22 above 0.1); EN pairs mean gain +0.0101 (8/13 positive, 3/13 above 0.1).
• Word-mix and embed-mix show the same trends. On EN–ZH, both methods show decreasing effectiveness as the ZH proportion grows on EN-only documents, increasing effectiveness on ZH-only documents, and smooth interpolation between endpoints on EN+ZH documents. Embed-mix achieves higher nDCG than word-level mixing, which the authors attribute to generation variability and tokenization/LID errors in word-mix. Diagnostic analysis shows word-mixed query embeddings move approximately linearly along the EN→ZH axis (median r increases with the intended ZH proportion) with small off-axis deviation δ.
• Ablations preserve the patterns. Across E5, GTE, Jina, and Qwen3 at 0.6B/4B/8B on a 100k-passage subset, Δ is near zero on English documents and clearly positive on non-English documents (for example, EN–AR on AR documents ranges from +1.190 to +7.384 across the six ablation models, and EN–ZH on ZH documents from +0.843 to +2.619). English remains the strongest mixing partner on ZH documents across all four families. Scaling generally improves monolingual endpoints and thereby reduces the magnitude of Δ without changing most patterns.

Methodology in Plain English

The researchers take mMARCO, a multilingual passage collection where the same query exists in translated versions aligned by a shared query ID, along with relevance judgments. They pick 14 languages (ar, de, en, es, fr, hi, id, it, ja, nl, pt, ru, vi, zh) and 35 representative pairs, using a filtered subset of 1,484 sufficiently long queries. Passages are encoded once into vectors and stored in a FAISS flat index; all query and passage embeddings are L2-normalized and scored by inner product (equivalent to cosine similarity). At retrieval time they fetch the top K = 100 passages and compute nDCG@10 (primary), MRR@10, and Recall@10 on the development split.

For each pair they evaluate three document-language settings: L1-only, L2-only, and a union of L1+L2 documents. Only the query representation changes across conditions; documents and the retrieval procedure stay fixed. The mixed query vector is simply a weighted average of the two monolingual query embeddings, normalized, with the weight λ swept over {0, 10, 30, 50, 70, 90, 100} — interior ratios {10, 30, 50, 70, 90} versus endpoints {0, 100}. Because no encoder call is needed after the two monolingual encodings, this is cheap and gives exact control over the mixing ratio.

For validation they also generate word-level code-mixed queries using an LLM (gpt-5-mini), prompting it to insert approximately the desired number of L2 content tokens, estimating the realized L2 share with a Stanza-based language-identification count, and bucketing outputs into five bands: (0,20], (20,40], (40,60], (60,80], (80,100]. After filtering for at least 6 tokens per language, 2,883 EN–ZH query pairs remain for this comparison.

Gains are summarized per (pair, document-language) setting as Δ = best_mid − endpoint_best, where Δ > 0 means an interior mixture beats the best monolingual query under the same setting. Language pairs are annotated with script match, family distance and typological distance from DistaL (glot_tree and lang2vec_knn), and resource level from the Microsoft linguistic diversity taxonomy (bands 0–5; band 5 treated as high-resource, bands 3–4 as lower-resource). Main experiments use BGE-M3 on the full 8.8-million-passage mMARCO corpus; ablations and word-mixing experiments use a 100k-passage subset per document-language setting.

Why This Matters

Impact on research: The paper reframes mixed-language retrieval as a predictable, structured property of current encoders rather than noise, and offers embedding interpolation as a cheap test-time probe for language-mix sensitivity. It also documents an asymmetry — English dominates the geometry of multilingual vector spaces — that has implications for how multilingual models are trained and evaluated, and it links mixing gains to typological distance once English is controlled for.

Real-world applications:

• Multilingual retrieval-augmented generation (RAG) pipelines: default to pure English queries when searching English-inclusive indices, but use embedding mixing for non-English indices to boost recall.
• Search engines serving bilingual or code-switching users who phrase a single information need in more than one language.
• Enterprise document search across translated or parallel corpora, where relevant material may exist in only one language or in several translated versions.
• Test-time augmentation or re-ranking, since Recall@10 benefits more from mixing than top-rank ordering does (98/105 settings positive on Recall@10).

Industry relevance: The recommended behavior is directly deployable — it requires only encoding a query twice (in two languages) and mixing vectors, with no retraining and no extra LLM inference at serving time. The ablation across E5, GTE, Jina, and Qwen3 at three scales indicates the patterns are not an artifact of one vendor or one model size.

Future Directions

• Extending the framework to naturally typed, organically mixed queries, which the authors note have spelling variation, transliteration, irregular switch points, and language-identification errors that translation-based data cannot capture.
• Interpolating more than two languages simultaneously, which the authors frame as a promising direction for test-time scaling across diverse and low-resource linguistic settings.
• Expanding beyond the 14 mMARCO languages and the 35 selected pairs by creating new translations and parallel data, which the authors deferred because translation quality control and evaluation alignment would be a separate project.
• Determining whether the English-dominance effect can be mitigated by training-time interventions (for example, code-switched training data), since the paper documents the bias but does not attempt to remove it.

Target Audience

Researchers and practitioners working on multilingual information retrieval, dense bi-encoder retrievers, cross-lingual search, and multilingual RAG systems. It is also relevant to engineers deploying vector search for bilingual user populations, and to NLP researchers studying code-mixing, embedding-space geometry, or the impact of English pre-training imbalance on multilingual representations. Readers should be comfortable with IR metrics and embedding similarity but do not need deep mathematical background, since the core method is a weighted average of two vectors.

Authors’ abstract

While mixed-language querying is ubiquitous in multilingual communities, the sensitivity of dense retrievers to such queries remains poorly understood. We present a ratio-controlled study on mMARCO that systematically evaluates retrieval performance by varying the mixing proportion of parallel query translations via embedding-level mixing -- constructing mixed queries as an interpolation of monolingual embeddings. Experiments with BGE-M3 demonstrate that an optimal mixing ratio outperforms the best monolingual endpoint in 88/105 cases. We uncover a distinct asymmetry driven by English dominance: mixing is uniformly beneficial when retrieving from non-English document indices, whereas indices containing English are best served by pure English queries. Furthermore, English acts as the strongest mixing partner for every non-English document language. Finally, when controlling for English dominance, mixing gains correlate negatively with typological distance. We conclude that language-mix sensitivity is structured and predictable, and we validate the robustness of these patterns across model families and scales.