Quality-Aware Translation Tagging in Multilingual RAG system

AI summary

Overview

Research area: Multilingual Retrieval-Augmented Generation (mRAG), cross-lingual information retrieval, and machine translation quality estimation.

Technical level: Intermediate. The paper builds on well-known components (dense retrieval, NMT, LLM prompting) and its novelty is architectural rather than mathematically heavy, but readers should be comfortable with RAG pipelines, reranking, and QA evaluation metrics.

Scope in one sentence: The paper proposes QTT-RAG, a pipeline that translates only non-query-language retrieved documents and attaches LLM-judged translation quality scores (semantic equivalence, grammatical accuracy, naturalness & fluency) as metadata instead of rewriting text, and evaluates it against CrossRAG and DKM-RAG on XOR-TyDi QA and MKQA in Korean, Finnish, and Chinese across six instruction-tuned LLMs ranging from 2.4B to 14B parameters.

What This Paper Is About

Multilingual RAG systems often retrieve English documents and translate them into the user's query language, but poor translations drag down answer quality, and previous fixes either assume translations are good enough (CrossRAG) or rewrite passages with an LLM (DKM-RAG), which can distort facts. The authors' goal is to let the generator decide how much to trust each translated passage by explicitly scoring translation quality along three dimensions and attaching those scores as tags, without modifying the translated content itself.

Key Contributions

1. LLM-based translation quality assessment: An evaluation module that scores each translated document on semantic equivalence, grammatical correctness, and linguistic naturalness using an LLM agent.
2. The QTT-RAG architecture: A five-stage multilingual RAG pipeline (retrieval, reranking, language detection & translation, quality tagging, generation) that attaches translation quality scores as metadata so the generator can weigh sources by reliability, avoiding factual distortion from rewriting.
3. Empirical validation: Experiments on two multilingual QA benchmarks across Korean, Finnish, and Chinese with six instruction-tuned LLMs, showing consistent improvements in character 3-gram recall and robustness to translation errors over CrossRAG and DKM-RAG.
4. A quantified failure analysis of rewriting-based mRAG: A manual analysis showing entity hallucination rates in DKM-RAG's refinement step.

Main Findings

• Overall win on Korean: QTT-RAG consistently outperforms all baselines on XOR-TyDi–ko and MKQA–ko across six LLMs, with performance gains ranging from 0.4% to 6.8% over the baselines. Exaone-3.5-7.8B-Instruct reaches the highest score, 43.8 on XOR-TyDi–ko and 40.0 on MKQA–ko (character 3-gram recall, %).
• Finnish results are strong but not universal: On XOR-TyDi–fi, QTT-RAG achieves 66.8 with Phi-4 (14B), 60.0 with Llama-3.1-8B-Instruct, 59.3 with Exaone-3.5-7.8B-Instruct, 58.6 with Qwen2.5-7B-Instruct, and 50.4 with Exaone-3.5-2.4B-Instruct, but drops to 55.4 with Aya-Expanse-8B — below CrossRAG's 60.1 for that model.
• Chinese shows smaller and mixed gains: On MKQA–zh, CrossRAG achieves better performance with Exaone-3.5-2.4B-Instruct (25.2 vs QTT's 24.4), Exaone-3.5-7.8B-Instruct (26.2 vs 25.8), and Llama-3.1-8B-Instruct (30.1 vs 29.3), while QTT-RAG wins with Phi-4 (14B) at 34.5, Aya-Expanse-8B at 33.9, and Qwen2.5-7B-Instruct at 31.9.
• Rewriting causes measurable entity hallucination: Manual analysis of 1,855 retrieved documents for 371 questions from XOR-TyDi–ko found that in 214 cases (11.5%), entities in the query (names, places, dates) that were absent from the original documents were added during DKM-RAG's rewriting.
• Quality tagging beats hard filtering as a default: In an ablation against Hard filtering (documents excluded if they fall below a threshold of 3.5 on all three criteria), QTT-RAG consistently outperformed Hard filtering on all Korean models, with average relative gains of 3.8% on XOR-TyDi–ko and 12.6% on MKQA–ko.
• Hard filtering wins in specific regimes: On XOR-TyDi–fi, Hard filtering achieves the best score with Aya-Expanse-8B (61.0), beating QTT-RAG (55.4) and all other baselines. On MKQA–zh, Hard filtering surpasses both QTT-RAG and CrossRAG with Exaone-3.5-2.4B-Instruct (25.4 vs 24.4) and Exaone-3.5-7.8B-Instruct (26.8 vs 25.8).
• Cross-lingual share explains the Chinese gap: The MKQA–zh split has a cross-lingual share of r_lang = 5.0%, versus r_lang = 22.7% for MKQA–ko, meaning far fewer translated passages are involved in Chinese and therefore less headroom for translation-quality tagging to help.
• No single strategy dominates universally: The authors state that effectiveness varies with resource level, proportion of cross-lingual passages, retriever and MT quality, filtering thresholds, retained ratio, and the generator backbone.

Methodology in Plain English

The pipeline has five stages.

1. Retrieval: For a query in Korean, Finnish, or Chinese, retrieve top-k candidate documents from a Wikipedia index using BGE-M3 as a dense multilingual retriever.
2. Reranking: Re-score the candidates with BGE-M3 as a reranker to improve precision at early ranks. Figure 2 describes passing the top-5 most relevant documents forward.
3. Language detection and translation: Detect each document's language. Documents already in the query language are preserved as-is and skip translation. Foreign-language documents are translated into the query language with NLLB-200-distilled-600M, a model supporting over 200 languages.
4. Quality tagging: An LLM agent (Llama-3.1-8B-Instruct) scores each translated document from 0.0 to 5.0 on semantic equivalence, grammatical accuracy, and naturalness & fluency, using language-specific prompts. The scores are attached to the document as tags; documents originally in the query language receive no score. The translated content itself is never rewritten.
5. Generation: The generator receives the query plus the quality-tagged documents through a prompt template that instructs it to prioritize passages with higher quality scores and to be cautious with uncertain information. No fine-tuning is used — only in-context learning from the visible scores.

Experimental setup: The framework is implemented with Bergen. The document index is built from Wikipedia with 25M English, 1.6M Korean, 1.5M Finnish, and 11M Chinese articles. Evaluation uses character 3-gram recall (the proportion of three-character sequences from the gold answer that appear anywhere in the prediction), chosen because it tolerates orthographic variation while penalizing hallucinations and is language-agnostic.

Data: MKQA consists of 10,000 examples from Natural Questions translated into 26 languages; because MKQA lacks document-level annotations, the authors use a subset of 2,827 MKQA samples overlapping with KILT-NQ. XOR-TyDi QA is evaluated on the Korean (371 questions) and Finnish (615 questions) splits.

Models compared: Base (retrieval only, no translation), CrossRAG (translates all retrieved passages), DKM-RAG (LLM-based rewriting of translated passages), and QTT-RAG. Generators are Exaone-3.5-2.4B-Instruct, Exaone-3.5-7.8B-Instruct, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Aya-Expanse-8B, and Phi-4 (14B).

Why This Matters

Impact on research: The paper reframes translation quality in mRAG from an implicit assumption or a rewriting step into an explicit, non-destructive signal. It provides documented failure cases for both prior paradigms — DKM-RAG fabricating a death date for "Gwisil Boksin" from a passage about "Rumer Godden," and CrossRAG dropping the answer-bearing clause about Nordic versus Scandinavian terminology — and quantifies the rewriting problem (11.5% entity hallucination).

Real-world applications:

• Multilingual question answering and customer support over English-dominated knowledge bases for Korean- and Finnish-speaking users.
• Enterprise or government document search where queries are in a low-resource language but the corpus is mostly English, and where altering source text is unacceptable.
• Medical and health information retrieval, relevant to co-author affiliations at the University of Ulsan College of Medicine.
• Any deployment needing a confidence signal on machine-translated evidence, such as retrieval systems that surface sources to human reviewers.

Industry relevance: The work comes from LG Electronics' Toronto AI Lab with support from CARTE and LG Toronto AI Lab, and its design — tagging rather than rewriting — is aimed at preserving factual integrity, which matters for production systems where hallucinated rewrites are riskier than slightly awkward translations.

Future Directions

• Expanding evaluation to a wider set of languages to test scalability, since the current study covers only Korean, Finnish, and Chinese.
• Exploring hybrid retrieval strategies, such as deliberately inducing cross-lingual usage via English-only retrieval for non-English queries.
• Determining when Hard filtering should be preferred over quality tagging, since Hard filtering wins with Aya-Expanse-8B on XOR-TyDi–fi and with the Exaone models on MKQA–zh, while being sensitive to threshold choice and to abundant in-language evidence.
• Addressing the dependency on strong instruction-following generators, as QTT-RAG requires the model to reliably interpret structured metadata — a limitation the authors note for models with weaker instruction following or shorter effective context windows.

Target Audience

Researchers and engineers working on multilingual retrieval-augmented generation, cross-lingual information retrieval, and machine translation quality estimation; practitioners building QA or search systems for low-resource languages over English-heavy corpora; and evaluation-focused researchers interested in character 3-gram recall for cross-lingual QA. The paper is also useful for teams deciding between translation, rewriting, and filtering strategies in production RAG pipelines.

Authors’ abstract

Multilingual Retrieval-Augmented Generation (mRAG) often retrieves English documents and translates them into the query language for low-resource settings. However, poor translation quality degrades response generation performance. Existing approaches either assume sufficient translation quality or utilize the rewriting method, which introduces factual distortion and hallucinations. To mitigate these problems, we propose Quality-Aware Translation Tagging in mRAG (QTT-RAG), which explicitly evaluates translation quality along three dimensions-semantic equivalence, grammatical accuracy, and naturalness&fluency-and attach these scores as metadata without altering the original content. We evaluate QTT-RAG against CrossRAG and DKM-RAG as baselines in two open-domain QA benchmarks (XORQA, MKQA) using six instruction-tuned LLMs ranging from 2.4B to 14B parameters, covering two low-resource languages (Korean and Finnish) and one high-resource language (Chinese). QTT-RAG outperforms the baselines by preserving factual integrity while enabling generator models to make informed decisions based on translation reliability. This approach allows for effective usage of cross-lingual documents in low-resource settings with limited native language documents, offering a practical and robust solution across multilingual domains.