Leveraging External Knowledge for Historical Document Restoration via Retrieval-Augmented Large Language Models

AI summary

Overview

Research area: Natural Language Processing — historical document restoration, retrieval-augmented generation (RAG), and domain-specific large language model fine-tuning.

Technical level: Intermediate. The paper assumes familiarity with masked language modeling, causal LLMs, retrieval methods (BM25 vs. dense embeddings), and standard fine-tuning practice, but its core argument is conceptual rather than mathematically dense.

Scope: The paper proposes and evaluates ARI (Archive Restoration Intelligence), an LLM-based framework that restores damaged Hanja characters in Korean historical archives by combining internal model knowledge, chronological metadata, few-shot examples, and retrieval of relevant passages from a large historical corpus.

What This Paper Is About

Centuries-old Korean historical records such as the Annals of the Joseon Dynasty and the Journal of the Royal Secretariat have degraded physically, leaving roughly 42,000 unreadable characters across 11,000 documents in one corpus alone. Prior restoration models treat the damaged document as the only input, which works for ordinary characters but fails on named entities — personal names, place names, book titles — because those require knowledge that simply is not present in the surrounding text. The paper's goal is to show that supplying an LLM with explicitly retrieved external context, on top of what it absorbed during pre-training, substantially improves restoration of exactly the characters that matter most to historians.

Key Contributions

1. A RAG-based restoration framework for historical documents. The authors build a pipeline that feeds an LLM the damaged text, chronological metadata, five format exemplars, and twenty retrieved reference documents, then fine-tune an open-source model on this format. The resulting ARI-32B and ARI-8B models are released publicly alongside their code.

2. Systematic isolation of what "external knowledge" actually contributes. Through ablations, they separate the effects of metadata, few-shot examples, retrieval method (random / embedding / BM25), retrieval count, and deduplication, showing that each component adds measurable accuracy and that lexical retrieval beats semantic retrieval for Hanja restoration.

3. A domain-appropriate from-scratch baseline. They train BERT-Res, a ModernBERT-large model whose tokenizer was extended with Hanja characters, using the same named-entity-prioritized masking strategy — providing a clean control for "restoration without any external context."

4. Dual evaluation: synthetic benchmarks plus blinded expert review. Beyond standard top-K accuracy on two constructed test sets, three experts in Sinographic literature and Korean history evaluated the models on 100 genuinely damaged documents through a custom web interface, with accuracy, nDCG@10, and win-rate metrics.

Main Findings

• External knowledge is the dominant factor for named entities. For Qwen3 32B, adding chronological metadata, few-shot examples, and BM25 retrieval with deduplication raised named-entity restoration accuracy from 5.63 to 27.85 and random-character accuracy from 14.83 to 61.45.

• Lexical retrieval outperforms dense retrieval for Hanja. BM25 exceeded embedding-based retrieval and random selection. A hybrid sweep found the pure BM25 configuration (16:0) best, and adding a modern Qwen3 reranker actually lowered named-entity accuracy from 38.58% to 37.70%, suggesting re-rankers trained on modern text do not transfer well.

• Deduplicating retrieved passages helps, but over-filtering hurts. Performance peaks when documents with more than 80% string similarity are removed; exact-match-only deduplication performed worst, indicating that near-duplicates crowd out diverse evidence.

• Reasoning mode is a trade-off. Enabling thinking mode in Kimi K2 and Gemini-2.5-Flash improved named-entity restoration but slightly reduced random-character restoration.

• "Restoration knowledge" and "general language knowledge" are separable. BERT-Res, trained purely on in-document context, beat Gemini-2.5-Pro on random characters but lagged LLMs badly on named entities — direct evidence of the two different capability requirements.

• ARI-32B leads overall, including against larger proprietary models. It outperformed Sonnet 4.5, Gemini-2.5-Pro, and GPT-5.1 on both test sets despite its smaller parameter count, and dominated the fine-grained PER and LOC categories.

• One category breaks the pattern. Gemini-2.5-Pro surpassed ARI-32B on POH (publications of history), a rare category dominated by Confucian classics of Chinese origin that fall outside the Joseon-specific domain the retrieval corpus covers.

• Human experts preferred ARI-32B. On 100 real damaged documents, ARI-32B achieved Accuracy@1 of 0.383 versus 0.273 for Gemini-2.5-Pro and 0.207 for BERT-Res, with a 46.0% win rate as a collaborative tool.

• Temporal distance degrades performance. Tested on the Goryeosa, a chronicle predating the training corpora, ARI stayed robust under moderate time shifts but declined as the gap widened, reflecting both language change and mismatched period knowledge.

• A 25% named-entity-prioritized masking ratio was the balance point. Higher proportions improved entity restoration at the cost of general character restoration; 100% random masking depressed entity accuracy sharply.

Methodology in Plain English

The authors start with two large digitized corpora of Hanja documents, AJD and JRS, provided by the National Institute of Korean History with metadata such as date, reigning king, and annotated entity types. They measure how often damage occurs in real records, then use Gemini-2.5-Pro as a classifier to estimate what kinds of entities the unreadable characters represent — finding that about 44.8% are named entities, most commonly persons and locations.

Because real damage lacks ground truth, they build synthetic test sets by masking characters at random positions using corruption rates and span lengths matched to the observed real distribution (centered on 2.96%). Two versions are created: one where corruption falls anywhere, and one where corruption specifically targets named entities.

For the model, each damaged character is replaced with a placeholder like [D1], and the LLM must return a JSON object mapping each placeholder to the restored Hanja character. Around this basic format they layer three sources of external knowledge: the document's date and reign era, five worked input-output examples that teach the output schema, and twenty reference documents retrieved from the training corpus. They test random selection, Gemini Embedding dense retrieval, and BM25 sparse retrieval, then apply string-similarity deduplication.

Finally, they fine-tune Qwen3 32B and Qwen3 8B on this prompt format using dynamically varying mask positions across epochs, with 25% of training examples masking named entities preferentially. The comparison baseline is ModernBERT-large trained from scratch on the same data with no retrieval. Evaluation uses top-K accuracy on the synthetic sets, and for real damaged documents a blinded web interface where three domain experts select valid candidates and indicate model preference.

Why This Matters

The paper makes a clean empirical case that in low-resource historical NLP, retrieving the right external evidence can beat scaling up a general model. That reframes a common assumption — that bigger pre-trained models will eventually absorb enough historical knowledge on their own — and gives a concrete recipe for domain adaptation where training data is scarce. It also publishes models, code, and a documented corruption-distribution methodology that other historical NLP projects can reuse.

Real-world applications:

• National archives and digital libraries such as the National Institute of Korean History can integrate ARI into their transcription platforms to automatically flag and propose readings for damaged passages.
• Historical and linguistic research is accelerated when scholars receive a ranked list of candidate characters with confidence scores, reducing manual lookup time in classical reference works.
• OCR post-correction pipelines for scanned manuscripts can use the same retrieval-plus-LLM pattern to clean up low-confidence character recognitions in degraded scans.
• Translation and cross-lingual access to classical East Asian texts benefits, since restored Hanja enables downstream machine translation of previously unreadable records.
• Cultural heritage preservation programs, including UNESCO Memory of the World initiatives, gain a scalable triage tool for prioritizing which damaged documents to restore first.

Industry relevance: The approach generalizes to any domain with large unlabeled corpora and localized gaps — legal archives, medical records, epigraphy, and manuscript digitization services — and the finding that BM25 beats dense retrieval for logographic scripts has direct implications for search infrastructure in East Asian language products.

Future Directions

• Extending the context window. The 4,096-token limit forced the authors to discard about 1.7% of documents; longer-context models could handle full-length records without truncation.
• Multimodal restoration. The National Institute of Korean History supplies original document images, so combining visual stroke-level evidence with text retrieval could resolve damage that text alone cannot.
• Closing the synthetic-to-real gap. The paper acknowledges that synthetically masked data may not perfectly reflect the statistics of genuine physical deterioration; better damage simulation or semi-supervised learning on real damage is an open problem.
• Temporal domain adaptation. Performance degrades as the temporal gap widens, raising the question of how to build retrieval corpora or adapters that transfer across dynasties and centuries.
• Handling knowledge that lies outside the target domain. The POH category showed that broad multilingual classical Chinese knowledge can beat domain-specific retrieval, suggesting a hybrid of specialized and general corpora.
• Interactive human-in-the-loop workflows. The byte-level constrained decoding that produces Top-K candidates is a promising foundation for interfaces where experts correct one character and see downstream predictions update.

Target Audience

NLP researchers working on low-resource languages, domain adaptation, or retrieval-augmented generation will find the ablations and the negative results on reranking directly useful. Digital humanities scholars and historians of East Asia will benefit from the practical tooling and the expert-evaluation protocol. Archivists, librarians, and engineers building document digitization or OCR post-correction systems can adopt the prompt-and-retrieval architecture without deep ML expertise. Finally, practitioners in cultural heritage technology evaluating whether general-purpose LLMs suffice for specialized restoration tasks will find the comparison against Gemini-2.5-Pro, Sonnet 4.5, and GPT-5.1 particularly informative.

Authors’ abstract

Historical documents act as invaluable knowledge archives but often suffer from illegibility due to physical deterioration and damage. While existing restoration methods based on masked language modeling effectively utilize local context, they struggle to restore named entities that require external historical knowledge. To address this limitation, we introduce a novel framework for historical document restoration that leverages large language models with retrieval-augmented generation (RAG). By combining the implicit knowledge of pre-trained LLMs with explicitly retrieved external context, our model ARI effectively mitigates the challenge of inferring context-dependent proper nouns. Extensive experiments on Korean historical documents demonstrate that our approach significantly outperforms baselines, achieving substantial gains in restoring both general characters and named entities. Furthermore, comprehensive evaluations including expert assessments confirm that ARI serves as a practical tool for domain experts, promising to accelerate the analysis of historical records.