LangMark: A Multilingual Dataset for Automatic Post-Editing

AI summary

Overview

• Research area: Natural Language Processing — machine translation, automatic post-editing (APE), and multilingual benchmark datasets
• Technical level: Intermediate
• Scope: The paper introduces LangMark, a large human-annotated multilingual APE dataset, and uses it to benchmark how well modern LLMs can correct machine-translated text across seven target languages.

What This Paper Is About

Automatic post-editing (APE) systems are supposed to fix errors in machine translation output, but progress has been slowed by a lack of large, multilingual datasets built from real neural machine translation (NMT) outputs. The authors address this by releasing LangMark, a human-annotated corpus of 206,983 source–NMT–post-edit triplets spanning English to seven languages, and by using it to evaluate whether large language models can outperform the commercial and proprietary translation systems that produced the pre-translations.

Key Contributions

1. LangMark dataset release. A human-annotated multilingual APE corpus containing 206,983 triplets from English into Brazilian Portuguese, French, German, Italian, Japanese, Russian, and Spanish — described as the largest multilingual, human-annotated APE dataset on NMT outputs to date.
2. LLM benchmarking on APE. Empirical evidence that LLMs with 20-shot prompting can perform APE at a level that improves on outputs from leading commercial and proprietary MT systems, including in hard cases like Japanese and Russian.
3. Analysis of the "edit or not" problem. A formal precision/recall framing of the classification decision that underlies APE (deciding whether a segment even needs editing), showing that "conservative" high-precision models tend to score better on MT metrics than "aggressive" high-recall ones.
4. Argument for better evaluation. A critique of current MT metrics (CHRF, TER, BLEU) for APE, arguing that they miss the classification aspect of the task and calling for metrics that account for human-like editing behavior.

Main Findings

• Dataset scale and diversity. LangMark covers 7 language pairs with roughly 8.6K–33.3K triplets each and is sourced from real marketing content, making it both broader and larger than prior human-annotated APE resources like WMT APE tasks (15K–18K triplets) or SubEdits (161K triplets, EN-DE only).
• NMT baseline is strong. The proprietary in-domain MT engine outperforms Google Translate, DeepL, Microsoft Translator, and Amazon Translate on most language pairs (e.g., CHRF 89.73 vs. 82.57 for EN-IT against Microsoft), setting a demanding baseline for APE.
• GPT-4o is the strongest APE model. It consistently surpasses the NMT baseline on CHRF across all test languages except Portuguese, with the largest gains in Japanese (73.94 vs. 70.22) and Russian (69.68 vs. 68.90).
• Most models fail to beat the baseline. Seven of the eight evaluated models do not improve on the proprietary NMT output, which the authors present as evidence that LangMark is a genuinely challenging benchmark rather than a saturated one.
• Open-source model is competitive. Qwen2.5-72B rivals top closed-source models for its size and is the best-performing model on Russian in the CHRF evaluation.
• Precision beats recall for APE. Models classified as "conservative" (high precision, low recall) such as GPT-4o achieve better MT metric scores than "aggressive" models like Claude 3.5 Sonnet, which over-detect edits.
• Short segments are hardest. Performance drops sharply on short segments that require edits, likely because limited context makes it difficult for models to apply the right correction.
• All models under-edit relative to humans. Even the most edit-happy models make far fewer changes than human linguists did, indicating substantial headroom for improvement.

Methodology in Plain English

The authors collected real marketing documents from Smartsheet, stripped out personally identifiable information, deduplicated segments, and had a translation management system split the text into sentence-level chunks. Each chunk was translated by a proprietary MT engine trained on in-domain data, then post-edited by professional linguists with formal translation credentials and domain specialization. The resulting triplets (source, NMT output, human post-edit) form the dataset.

For evaluation, the data was split 90/10 per language pair. At test time, each source segment is embedded using OpenAI's text-embedding-3-small model, and the 20 most similar source/edited-translation pairs from the training portion are retrieved by cosine similarity within the same language pair. These 20 examples are inserted into a structured prompt (built with DSPy and routed via LiteLLM) that instructs the model to act as an expert linguist, make only necessary changes, preserve casing, leave technical terms and variables untouched, and return JSON output. Eight models — six closed-source and two open-source — were scored with CHRF, TER, and BLEU, and the authors additionally analyzed edit counts, precision, and recall to characterize each model's editing style.

Why This Matters

Research impact. LangMark fills a concrete resource gap: prior human-annotated APE datasets for NMT outputs were either small, single-language, or both. By providing over 200K triplets across seven languages with a deliberately strong NMT baseline, the dataset gives APE researchers a benchmark where improvement is meaningful rather than trivial, and its precision/recall framing reorients the field toward the "when to edit" question that most prior work neglected.

Real-world applications.

• Localization workflows: Companies can use APE models to reduce the human post-editing load in translation pipelines, cutting cost and turnaround time while maintaining quality standards.
• Low-resource language coverage: Extending APE quality to languages like Russian and Japanese, where edits are more frequent and model gains are largest, helps organizations serve markets they might otherwise under-serve.
• Quality assurance tooling: The precision/recall analysis directly informs how conservative an APE system should be before it is deployed in production, where spurious edits can be more damaging than missed ones.
• Model selection and procurement: Businesses choosing between commercial MT and LLM-based post-editing now have multilingual evidence about which systems actually improve output.

Industry relevance. The dataset comes from an industry setting (Welocalize, with Smartsheet data), a proprietary engine, and professional linguists working within a TMS. That grounding makes the benchmark unusually close to real production conditions — including the practical detail that a proprietary in-domain engine is often harder to beat than off-the-shelf commercial MT.

Future Directions

• Document-level APE. The current release is segment-level only; the authors plan a future version with full contiguous documents to enable context-aware, document-level post-editing experiments.
• Beyond marketing. Because the data comes from a single domain, extending LangMark to legal, medical, or literary content would test whether APE findings generalize.
• New evaluation metrics. The paper explicitly leaves open the design of a metric that jointly captures output quality and editing restraint — a metric that would reward conservatism appropriately rather than relying on CHRF or BLEU alone.
• Open-source MT baselines. Including outputs from open MT systems would enable glass-box analysis of APE behavior, which the proprietary engine currently precludes.
• Reverse and non-English directions. Adding translations into English, or between non-English pairs, would broaden the benchmark's applicability.

Target Audience

This paper is most valuable to NLP researchers working on machine translation, automatic post-editing, and evaluation methodology, particularly those who need a large multilingual benchmark to test LLM-based correction systems. It is also directly useful to localization engineers and translation technology teams evaluating whether LLM post-editing can replace or augment commercial MT in production. Practitioners interested in the precision/recall trade-off of automated editing, and researchers studying in-context learning with retrieved examples, will find the experimental setup and analysis relevant. A working familiarity with MT evaluation metrics and prompt-based LLM evaluation is helpful but not required to follow the main arguments.

Authors’ abstract

Automatic post-editing (APE) aims to correct errors in machine-translated text, enhancing translation quality, while reducing the need for human intervention. Despite advances in neural machine translation (NMT), the development of effective APE systems has been hindered by the lack of large-scale multilingual datasets specifically tailored to NMT outputs. To address this gap, we present and release LangMark, a new human-annotated multilingual APE dataset for English translation to seven languages: Brazilian Portuguese, French, German, Italian, Japanese, Russian, and Spanish. The dataset has 206,983 triplets, with each triplet consisting of a source segment, its NMT output, and a human post-edited translation. Annotated by expert human linguists, our dataset offers both linguistic diversity and scale. Leveraging this dataset, we empirically show that Large Language Models (LLMs) with few-shot prompting can effectively perform APE, improving upon leading commercial and even proprietary machine translation systems. We believe that this new resource will facilitate the future development and evaluation of APE systems.