Can QE-informed (Re)Translation lead to Error Correction?

AI summary

Overview

• Research area: Machine translation (MT) quality estimation (QE) and error correction, specifically the WMT 2025 Automated Translation Quality Evaluation Systems Task 3 on QE-informed Segment-level Error Correction.
• Technical level: Intermediate. The paper assumes familiarity with MT evaluation metrics (COMET, COMETKiwi, BLEU, chrF++), automatic post-editing (APE), and prompt-based use of open-source LLMs.
• Scope: The paper describes and compares two training-free, LLM-based systems — one that uses QE as a selector among multiple retranslations and one that masks QE-identified error spans for an LLM to fill in — evaluated on 6,000 English-to-six-language translation segments.

What This Paper Is About

Machine translation still produces errors, and Automatic Post-Editing (APE) systems designed to fix them are known to overcorrect, which degrades output. This paper asks whether QE-informed retranslation can sidestep that problem, and tests two training-free approaches on the WMT 2025 shared task: one that generates fresh translations from several LLMs and picks the highest-QE one, and one that uses the dataset's provided error spans to replace only the flagged substrings.

Key Contributions

1. Two training-free systems for the WMT 2025 QE-informed Segment-level Error Correction subtask, both built on pre-trained open-source LLMs rather than task-specific fine-tuning.
2. "Best MT Wins": QE used as a selector over candidate translations from six LLMs (Aya-Expanse-8B, GPT-SW3-6.7B, Tri-7B, GLM4-9B, Phi4-mini-instruct, TowerPlus-9B) plus the original system outputs, scored with the wmt22-cometkiwi-da model.
3. "Fill in the Blanks": a masked-refill correction method using TowerPlus-9B, in which error-span substrings are replaced with a __BLANK__ token and the model must fill them in, governed by a conditional masking heuristic based on error severity and the overall QE score.
4. A conditional heuristic (Algorithm 1) aimed at maximizing the Gain-to-Edit ratio: skip masking if QE score ≥ 0.90; if > 0.50, mask minor spans only when all spans are minor, otherwise mask all non-minor spans; below 0.50, mask all error spans.

Main Findings

• The selection approach won the subtask. The primary approach achieved an overall Δ COMET of 0.0201 and took the winning position on the subtask leaderboard, with an average Gain-to-Edit (G2E) Ratio of 6.22e-4, BLEU of 62.78, and chrF++ of 74.08.
• Icelandic benefited most. Language-wise Δ COMET for the primary approach was: Icelandic 3.65e-2, Russian 2.01e-2, Czech 1.89e-2, Chinese 1.84e-2, Ukrainian 1.63e-2, and Japanese 1.03e-2. The paper describes the Icelandic gain as roughly 3% improvement.
• Contributions were highly uneven. TowerPlus-9B contributed the most selections (2836), followed by the original translations (2829), GLM4 9B (149), Phi4 Mini Instruct (106), Aya Expanse 8B (76), and GPT-SW3 6.7B v2 Instruct (9). Tri 7B contributed 0.
• The masked-refill approach degraded quality. The secondary approach scored an overall Δ COMET of -0.0108, with an average G2E Ratio of -1.59e-5, BLEU of 67.71, and chrF++ of 88.87. Language-level Δ COMET was negative for all six pairs, ranging from -7.24e-3 (Czech) to -1.35e-2 (Ukrainian), which the paper characterizes as roughly -0.7% to -1.4% quality degradation.
• High surface overlap does not imply better metric scores. The secondary approach has notably higher BLEU and chrF++ than the primary approach, yet consistently negative Δ COMET.
• A positive case exists for the masked-refill approach. In Appendix C (Czech, original system DeepSeek-v3), COMET rose from 0.8215 to 0.8317 after correction. The paper attributes the improvement partly to TowerPlus-9B having been fine-tuned on instruction data from models including DeepSeek-v3, and notes the original had 3 major and 1 minor error.
• Output artifacts hurt the masked-refill approach. In Appendix B (Japanese, original system MMMT), COMET dropped from 0.8183 to 0.7314, with an untranslated artifact (__HEARTBREAK__) and prompt-format leakage ("Corrected words: [...") in the output.
• A case where retranslation beat correction. In Appendix A (Chinese, original system Wenyiil), the new MT scored 0.7797 versus the original's 0.7751, while the edited output scored 0.7723.
• Listed failure factors for the masked-refill approach: use of bigger and closed-source models in the original MT, use of varied systems and models, inclusion of low-resource languages, correcting only critical + major error spans while ignoring minor spans in some instances, and insufficient prompt tuning and post-processing.

Methodology in Plain English

The team used the complete test data supplied with the shared task: 6,000 machine translations from English into Chinese, Czech, Japanese, Icelandic, Russian, and Ukrainian, with 1,000 instances per language pair. The texts span news, social, speech, literary, and dialogue domains, unevenly distributed — news is the largest domain with 1,808 entries, and dialogue is the smallest with 28 segments, all in the English-to-Czech group. The original translations came from 38 different translation systems, including LLMs such as GPT4.1, Claude-4, and DeepSeek-v3.

For the primary approach, each English source was translated from scratch by several LLMs using tailored prompts (some resembling the models' own Hugging Face translation prompts, others modified because reusing the same prompt caused hallucinations, chain-of-thought output, or worse translation quality in limited internal testing). Because most models do not support all six target languages, each translated only the languages it supports. All candidates, including the original systems' translations, were then scored with wmt22-cometkiwi-da using the English source and the candidate output, and the highest-scoring translation was taken as final. This reframes APE as re-ranking rather than editing.

For the secondary approach, the provided error spans were used to cut the flagged substrings out of the translation and replace them with a __BLANK__ token, turning correction into a fill-in-the-blank task. TowerPlus-9B (9 billion parameters, based on Gemma2, 8192-token context window, supporting all six task languages) performed the filling, with a one-shot example in the prompt, domain included as context, a different example per language, and an explicit instruction not to reuse the removed "wrong words." To keep edits minimal, the conditional masking heuristic was applied: if the segment's QE score was 0.90 or above, no masking was done at all; between 0.50 and 0.90, only minor spans were masked if all errors were minor, otherwise all non-minor spans were masked; below 0.50, all error spans were masked.

Evaluation used the shared task's two metrics: Δ COMET (the COMET difference between the proposed approach and the baseline translation) and the Gain-to-Edit Ratio (Δ COMET divided by total edit rate). BLEU and chrF++ are reported as supplementary information but are not the focus of the analysis.

Why This Matters

The paper offers a concrete, negative-result-rich comparison of two ways of using QE signal: as a selector over retranslations versus as guidance for targeted edits. The clear asymmetry — the selection method gains, the masking method loses — is useful evidence for anyone deciding how to wire QE into a translation pipeline, and it reinforces the paper's framing that QE is best treated as a decision mechanism rather than a correction oracle. It also adds a data point to the ongoing overcorrection debate in APE.

Real-world applications:

• Localization pipelines for low-resource languages, where the paper reports the strongest selection gains (Icelandic showed the largest Δ COMET at 3.65e-2).
• Quality triage in translation workflows, where cheap QE scores could decide whether to accept a machine output, retranslate it, or escalate it to a human.
• Domain-specific translation of news, literary, speech, social, and dialogue content, the domains covered in this test set.
• Post-editing tooling: the reported failure modes (untranslated artifacts like __HEARTBREAK__, prompt leakage like "Corrected words: [", and high-BLEU-but-worse-COMET outputs) are practical failure signatures worth detecting in deployed LLM editing tools.

Industry relevance: the paper's honest accounting of cost is notable — the authors state that the winning "Best MT Wins" approach is impractical because it requires generating full hypotheses from large models, and that several of the six LLMs contributed little. The contribution counts (2836 from TowerPlus-9B plus 2829 from the original translations, out of 6,000 segments) quantify how concentrated the benefit actually is.

Future Directions

• Better prompt engineering and output post-processing for the masked-refill approach, which the authors identify as the direct cause of the artifacts and leakage that degraded output.
• A two-model system: a smaller language model to propose words or phrases for the masked tokens via masked language modeling, paired with a larger LLM to select the most suitable candidate.
• Further exploration of system combinations for the selection approach, since the paper notes its performance "ultimately depends on the model and system selection."
• Human evaluation, which the paper explicitly says is absent, to validate whether metric-based gains on English-centric automatic metrics reflect true quality improvements, especially for non-English language pairs.
• Efficiency improvements, since the selection approach currently consumes substantial compute and time by generating full translations from multiple large models.

Target Audience

Researchers and practitioners working on machine translation quality estimation, automatic post-editing, and LLM-based translation systems — particularly those participating in or following WMT shared tasks. It is also relevant to localization engineers and product teams evaluating whether to use QE as a selector versus an editing guide, and to readers interested in the practical limitations of prompt-based LLM correction under compute and time constraints. Readers should bring at least intermediate familiarity with MT evaluation terminology, since metric names, language codes, and statistical notation are used without extended explanation.

Authors’ abstract

The paper presents two approaches submitted to the WMT 2025 Automated Translation Quality Evaluation Systems Task 3 - Quality Estimation (QE)-informed Segment-level Error Correction. While jointly training QE systems with Automatic Post-Editing (APE) has shown improved performance for both tasks, APE systems are still known to overcorrect the output of Machine Translation (MT), leading to a degradation in performance. We investigate a simple training-free approach - QE-informed Retranslation, and compare it with another within the same training-free paradigm. Our winning approach selects the highest-quality translation from multiple candidates generated by different LLMs. The second approach, more akin to APE, instructs an LLM to replace error substrings as specified in the provided QE explanation(s). A conditional heuristic was employed to minimise the number of edits, with the aim of maximising the Gain-to-Edit ratio. The two proposed approaches achieved a Delta COMET score of 0.0201 and -0.0108, respectively, leading the first approach to achieve the winning position on the subtask leaderboard.