HERIT: Enabling Global Access to Hanja-to-English Historical Translation by Mitigating Temporal Bias

AI summary

Overview

Research area: Natural Language Processing — low-resource machine translation, domain adaptation, retrieval-augmented generation (RAG), and digital humanities.

Technical level: Intermediate. The paper assumes familiarity with LLM fine-tuning, in-context learning, retrieval-augmented generation, and standard MT metrics (BLEU, METEOR, chrF++), but the pipeline itself is explained clearly enough for a motivated non-specialist.

Scope (one sentence): The paper presents HERIT, a domain-adaptation pipeline that uses RAG-based pseudo-labeling of abundant Hanja–Korean corpora to fine-tune a Hanja-to-English translation model for Korean historical archives, reducing reliance on scarce, temporally skewed human translations.

What This Paper Is About

Korean historical archives such as the Annals of the Joseon Dynasty (AJD) and the Journal of the Royal Secretariat (JRS) are written in Hanja and are valuable to researchers worldwide, but very little of this material exists in English. Human translation is extremely slow and expensive — the National Institute of Korean History's (NIKH) AJD-to-English project is projected to take 22 years at roughly 20 million USD — and only about 2.5% of the AJD (the King Sejong era) is currently translated, so whatever expert data exists is concentrated in one historical period.

The paper's goal is to build a Hanja-to-English translation model that generalizes across the whole 500-plus-year archive rather than overfitting to the one reign where human translations exist, and then to use that model to translate the remaining untranslated documents.

Key Contributions

1. A systematic domain-adaptation pipeline for historical Hanja-to-English translation that generates high-quality pseudo-labeled data from abundant Hanja–Korean pairs using retrieval-augmented in-context learning combined with FusioN (multi-draft aggregation), producing 315K pseudo-labeled samples at a cost of about $3,000.

2. HERIT, a Hanja-English RAG-based Intelligent Translation model — a Qwen3 32B model trained in two stages (first on pseudo-labeled data, then on human-translated data) that performs direct end-to-end Hanja-to-English translation with no retrieval at inference time.

3. Evidence that pseudo-labeling mitigates temporal bias: the authors built a held-out test set covering every reign (80 records per king, 2,080 entries total), strictly excluded from pseudo-labeling and training, and showed gains hold across periods rather than only on King Sejong-era text.

4. A released model and translated corpus, used to translate 2.09 million previously untranslated AJD and JRS documents (completed in 94.6 hours on eight NVIDIA H200 GPUs).

Main Findings

• Glossary and Korean input both help substantially. Adding a 25K-entry glossary raised BLEU from 15.720 to 23.747 and chrF++ from 43.971 to 51.478; additionally supplying the pre-translated Korean text raised BLEU to 25.457 and chrF++ to 52.637.

• Retrieved (dynamic) few-shot examples beat static ones by a wide margin. With RAG at K = 12 and Korean queries, BLEU reached 36.430 versus 30.387 for static shots; the final configuration (K = 64 for the Proposer, K = 12 for the Fusor, 5 drafts) reached BLEU 38.462, METEOR 0.625, and chrF++ 62.501 at a cost of $7.059.

• FusioN outperforms Best-of-N and no aggregation. FusioN scored BLEU 38.462 / chrF++ 62.501, compared with BoN at 38.337 / 62.117 and no aggregation at 37.905 / 61.534.

• A single proposer model beat a diverse multi-model proposer. The single-model setup achieved BLEU 38.462 / chrF++ 62.501 at $58.517, while combining Qwen3 235B-A22B, DeepSeek-V3.1, GPT-5.1, Gemini-2.5-Flash, and Sonnet 4.5 reached BLEU 36.250 / chrF++ 61.507 at $234.304.

• Model selection favored cost-effectiveness. Gemini-2.5-Pro (R) had the best raw score on the test set (BLEU 19.640, chrF++ 48.926) but cost $18.17, whereas Gemini-2.5-Flash achieved BLEU 15.720 / chrF++ 43.971 at $0.42; Hunyuan-MT-7B, a leading Chinese translation model, scored only BLEU 8.318 / chrF++ 35.450.

• Two-stage training beats mixed training. HERIT (Qwen3 32B) reached BLEU 38.432 / METEOR 0.626 / chrF++ 62.525, versus 36.849 / 0.613 / 61.246 for jointly training on the union of pseudo-labeled and human data, and versus 36.930 / 0.613 / 61.583 for training on human data alone. It also beat a pivot-translation baseline (Pivot-MT: 35.539 / 0.596 / 59.804) and the prior H2KE model (BLEU 24.62).

• Retrieval at inference is not worth it. Fine-tuning with glossary and RAG applied at both training and inference reached BLEU 38.117, below HERIT, while expanding context length beyond 8K tokens (13 to 14 times longer prompts) and multiplying training compute by 9.

• LLM-based evaluation on unseen reigns favors HERIT. Using Sonnet-4.6 as evaluator on a 1–10 scale over the 2,080-entry all-reign test set, the two-stage model scored 8.009 (Qwen3 32B) and 7.887 (Qwen3 8B), versus 7.003 and 6.539 for the glossary-and-RAG baseline and 6.883 and 6.312 for human-data-only training.

• Human experts preferred HERIT. On 200 examples rated 1–5 by specialists in Sinographic literature and Korean history, HERIT scored 3.93 against 3.22 for the baseline, and won 66.84% of pairwise comparisons — more than twice the baseline's rate.

• Temporal overfitting is visible and reduced. Per-reign analysis shows the baseline is competitive only during King Sejong's reign, the period where human translations are concentrated, while HERIT is consistently better elsewhere.

Methodology in Plain English

The researchers started from the observation that while English translations barely exist, Hanja-to-Korean translations are plentiful (350K Hanja–Korean pairs). Their strategy was to use those Korean translations as a bridge to manufacture English training data.

First, they picked a cheap but capable LLM (Gemini-2.5-Flash) to act as a "Proposer" that drafts English translations, and a second pass called a "Fusor" that reads several drafts (five, generated at temperature 0.3) and synthesizes a single better translation. Both were fed extra help: the original Hanja, the existing Korean translation, entries from a 25K-term glossary of names and terms, and a handful of similar example translations retrieved from the human-labeled training pool (64 examples for the Proposer, 12 for the Fusor, retrieved using Gemini-Embedding).

Second, they ran this pipeline over 317.4K untranslated Hanja–Korean pairs from reigns other than King Sejong's, yielding 315K pseudo-labeled (Hanja, Korean, English) triples for about $3,000. Excluding King Sejong's era was deliberate: it forces the model to learn from periods where no human English exists.

Third, they fine-tuned Qwen3 32B in two stages — first on the large pseudo-labeled set to learn Hanja semantics and the Korean pronunciations of proper nouns, then on the small 15.7K human-translated set to match human style. They used stage-specific learning rates, one epoch per stage, FSDP and the PagedAdamW 8-bit optimizer on eight H200 GPUs (15.1 hours for stage one, 1.2 hours for stage two).

Finally, they evaluated against LLM baselines, a pivot-translation baseline, their own ablated variants, an LLM judge, and human domain experts, and then deployed the model on all remaining untranslated documents.

Why This Matters

Impact on research: The work shows a practical route out of the classic low-resource bottleneck: instead of needing direct parallel data, use an intermediate language where data already exists and let a strong LLM plus retrieval manufacture the missing pairs. It also directly attacks temporal bias — the less-discussed problem that scarce expert data clusters in one era, so models silently fail on the rest of a long archive.

Real-world applications:

• Digital humanities and historical linguistics: making 500-plus years of Korean court records searchable and readable by non-Korean-speaking historians.
• Historical natural science: the paper notes these archives record solar eclipses, supernovae, seismic activity, famines, and epidemics, so translated text supports astronomy, seismology, climate, and epidemiology research.
• Cultural heritage institutions and archives: national libraries and museums can expose holdings to global audiences without 22-year manual translation programs.
• Low-resource classical language translation generally: the same pseudo-labeling-from-an-intermediate-language pattern could apply to other classical corpora with limited direct translation pairs.

Industry relevance: The cost analysis is notable for practitioners — $3,000 for 315K pseudo-labeled samples, $0.42 versus $18.17 per test-set evaluation for competing models, and 2.09 million documents translated in 94.6 hours. That kind of throughput and pricing shapes decisions about whether to build translation pipelines in-house rather than license human translation.

Future Directions

• Broadening the proposer pool: the authors note they used only Gemini-2.5-Flash for both Proposer and Fusor due to budget constraints, and expect more diverse proposer LLMs to yield higher-quality pseudo-labels — though their own single-model-versus-multi-model experiment (Table 9) suggests diversity is not automatically better.
• Extending pseudo-labeling to the JRS: only AJD documents were pseudo-labeled, even though the JRS is five times more voluminous than the AJD; adding it is an explicit next step.
• Lifting the length limit: hardware constraints capped sequences at 2,048 tokens (800 Hanja characters) and forced about 6.7% of training data to be filtered out, with chunking used at inference — the authors expect longer context to improve quality.
• Testing generalization beyond Korean Hanja: the paper argues the approach could transfer to other low-resource classical languages where direct translation pairs are scarce, but does not test this.

Target Audience

Most useful to NLP researchers and graduate students working on low-resource or historical machine translation, domain adaptation, and RAG-based data augmentation; to digital humanities scholars and archivists who want to understand what current LLM pipelines can and cannot deliver for classical texts; and to engineers building production translation pipelines where per-token API cost and inference latency matter as much as benchmark scores.

Authors’ abstract

Historical archives are invaluable for multidisciplinary research but remain inaccessible globally due to severe language barriers. While manual translation is prohibitively expensive and time-consuming, simply applying existing machine translation models is inadequate due to unique linguistic nuances and the temporal bias inherent in scarce human-labeled corpora. To address these challenges, we propose HERIT, a systematic domain-adaptation pipeline that leverages retrieval-augmented generation (RAG) to generate high-quality pseudo-labeled data from abundant Hanja-Korean corpora. This approach substantially expands the training dataset, mitigating both data scarcity and temporal overfitting across different historical periods. Extensive automated and human evaluations demonstrate that HERIT consistently outperforms baselines. Finally, we translate previously untranslated portions of the archives using our model, thereby enabling global access to these resources for researchers.