EpiAgent: An Agent-Centric System for Ancient Inscription Restoration

AI summary

Overview

Research area: Computer vision and AI for cultural heritage digitization, specifically the intersection of agentic LLM orchestration, image restoration, and Chinese epigraphy (ancient inscription and rubbing restoration).

Technical level: Advanced. The paper assumes familiarity with diffusion models, multimodal LLMs, retrieval-augmented generation, image-quality metrics, and agentic planning architectures.

Scope: The paper introduces an LLM-planned, tool-composing agent system that restores degraded ancient Chinese inscriptions by iterating through Observe–Conceive–Execute–Reevaluate stages, and it evaluates that system against unified image restoration, text-image enhancement, and inscription-specific baselines on the CIRI dataset.

What This Paper Is About

Ancient inscriptions carved in stone or preserved as paper rubbings carry both textual records and calligraphic art, but centuries of environmental decay, material damage, and human intervention corrupt them in spatially coupled, heterogeneous ways. Existing AI approaches either restore a single character at a time, or apply a fixed, predefined restoration pipeline that cannot adapt to different degradation patterns—often distorting the original glyph in an image-to-image mapping. The goal of this work is to build a system that mirrors how trained human epigraphers work: analyzing damage, drawing on accumulated expertise, selecting and sequencing specialized tools, and re-evaluating results until both textual accuracy and calligraphic fidelity are achieved.

Key Contributions

1. An agent-centric restoration system with an explicit deliberation loop. EpiAgent formalizes epigrapher workflow as an Observe–Conceive–Execute–Reevaluate paradigm, driven by an LLM-based central planner (implemented as Kimi-K2) that performs hierarchical closed-loop decision making rather than a single-pass restoration.

2. Decomposition of restoration into atomic multimodal operations and a composable toolkit. The system decomposes inscription restoration into atomic operations, allowing the planner to dynamically assemble and schedule a specialized toolkit—Background Denoising, Stroke Completion, Font Imitation, and Character Retrieval—based on degradation pattern analysis and historical metadata.

3. A multi-perspective evaluation and self-refinement mechanism. Restoration quality is judged by Text Authenticity, Style Consistency, and optional expert Human Feedback, which together define a failure set of characters that the planner replans around, progressively distilling experience into empirical priors.

4. Extensive experiments and ablations demonstrating state-of-the-art performance over unified restoration, text-image enhancement, and inscription-restoration baselines, plus insights for expert-level AI in cultural heritage preservation.

Main Findings

• EpiAgent leads all baselines on all metrics across all three CIRI test splits. On Testing Set S, EpiAgent reaches PSNR 22.14, SSIM 0.9684, LPIPS 0.0254, CLIP-IQA 0.9004, MUSIQ 53.98, MANIQA 0.4553, NIMA 0.5576, Top-1 Accuracy 0.9889, Top-5 Accuracy 0.9942, Macro Accuracy 0.6877, and 1-NED 0.9069. The strongest inscription-specific baseline, IR3, reaches PSNR 21.15, SSIM 0.9540, LPIPS 0.0388, 1-NED 0.8855, and the other methods fall below both.

• Restoration approaches the ceiling set by intact inscriptions. On Set S, ground-truth intact characters score Top-1 Accuracy 0.9971, Top-5 Accuracy 0.9996, Macro Accuracy 0.7064, and 1-NED 0.9120, against EpiAgent's 0.9889, 0.9942, 0.6877, and 0.9069.

• Real-world generalization is the largest differentiator. On split R-I, EpiAgent scores CLIP-IQA 0.9393, MUSIQ 50.29, MANIQA 0.4179, NIMA 0.5414, and 1-NED 0.5766. On split R-II, with entirely unseen degradation, it scores CLIP-IQA 0.9388, MUSIQ 49.94, MANIQA 0.4157, NIMA 0.5381, and 1-NED 0.5546. Methods such as CharFormer and IR3 remain competitive on synthetic images but drop markedly on the real splits.

• Human experts overwhelmingly prefer EpiAgent. In a user study involving over twenty epigraphers and scholars, EpiAgent received 59.66 percent Top-1 Ranking, 84.18 percent Top-3 Ranking, and a Mean Ranking of 82.11, versus 15.60, 51.14, and 67.41 for the next-best method, IR3.

• Multimodal analysis compounds in value. Ablating the Observe-stage modules, 1-NED on Set S rises from 0.6481 with the MLLM alone, to 0.9211 after adding the Corrective Language Model, to 0.9742 after adding RAG over the Chinese corpus. The same pattern holds on R-I (0.5509 to 0.8759 to 0.9694) and R-II (0.4336 to 0.8486 to 0.9606).

• Experience-guided adaptive planning beats both random and fixed schemes. On Set S, random tool invocation yields PSNR 18.53, SSIM 0.9078, LPIPS 0.0869, and 1-NED 0.7702; fixed Scheme A (Denoising–Completion) yields 21.19, 0.9605, 0.0371, 0.8814; fixed Scheme B (Denoising–Completion–Imitation) yields 20.78, 0.9526, 0.0401, 0.8935; experience-guided planning yields 22.14, 0.9684, 0.0254, 0.9069.

• Each evaluation perspective adds measurable value. Disabling text authenticity, style consistency, and human feedback entirely gives PSNR 21.48 and 1-NED 0.8969; adding text authenticity gives PSNR 21.59 and 1-NED 0.9026; adding style consistency instead gives PSNR 21.73 and 1-NED 0.8994; combining both gives PSNR 22.02 and 1-NED 0.9041; adding expert review as well gives PSNR 22.14 and 1-NED 0.9069.

• Reflective experience improves both quality and cost. Disabling the refinement mechanism (evaluating without recording and reusing feedback) increases average restoration time and decreases quality, per the comparison of restoration time and CLIP-IQA with and without reflective experience.

• Tool choice is degradation-dependent. Background denoising and stroke completion handle slight-to-medium degradation well, but under severe spalling or complete absence, denoising and completion alone become insufficient because local evidence makes inpainting ill-posed; font imitation and character retrieval then serve as alternatives, though they too can fail, motivating dynamic tool combination.

Methodology in Plain English

EpiAgent runs a loop with four stages, controlled throughout by an LLM acting as a central planner.

Observe. The system first builds a structured record of the damaged inscription. An MLLM produces an initial layout and text hypotheses. A Corrective Language Model, fine-tuned on a 7B LLM and equipped with retrieval-augmented generation, consults a large-scale Chinese corpus to correct the reading; human experts are permitted to verify corrections for authenticity. Separately, a Layout Rectification Module predicts a rectified layout that accounts for fully missing or occluded regions, and a Degradation Assessment Module produces pixel-level degradation masks plus a discrete severity level (none, slight, middle, or severe, encoded as 0 through 3).

Conceive. The planner converts this assessment into a per-character plan. It draws on distilled experience priors—statistical mappings mined from historical execution logs that link degradation patterns to how effective each tool tends to be—and produces, for each character in the corrected reading, an ordered sequence of tools to apply. Planning is done at the level of individual characters, which lets the system isolate character regions and remove cross-character background noise globally before applying localized tools.

Execute. The plan is instantiated through a composable toolkit of three diffusion-based tools—Background Denoising (masked diffusion to remove surface noise while preserving strokes), Stroke Completion (targeted inpainting of missing or severely degraded regions), and Font Imitation (style-consistent synthesis for heavily corrupted characters, learning style priors from high-quality exemplars of the same stele)—plus a Character Retrieval module that acts as a fallback, searching for identical characters elsewhere in the same inscription to avoid style drift. Tools are applied in the planned order and can be composed into task-specific routines.

Reevaluate. After each pass, every character is scored on Text Authenticity (comparing OCR output against the corrected reading), Style Consistency (cosine similarity of a style embedding against a reference distribution), and optional binary Human Feedback. Characters failing thresholds or rejected by experts form a failure set. If that set is non-empty and the iteration budget is not exhausted, the planner generates a revised plan focused on the failed characters; otherwise the process terminates and the execution logs are updated to refine experience priors for the future.

Evaluation uses the Chinese Inscription Rubbing Images (CIRI) dataset: 24k synthetic inscription images (20K training, 4K testing) plus 2k real rubbings split into R-I (degradation patterns partially reused in synthesis) and R-II (entirely unseen degradation). Seven image-quality metrics are used—PSNR, SSIM, and LPIPS as full-reference measures, and CLIP-IQA, MUSIQ, MANIQA, and NIMA as no-reference measures—alongside Top-1, Top-5, and Macro Accuracy and 1-NED for textual fidelity.

Why This Matters

Impact on research. The paper argues that image-to-image translation, which underlies most current restoration and text-image enhancement work, compromises structural fidelity and introduces glyph distortion—a flaw that is unacceptable for ideographic scripts where semantics and morphology are inseparable. By reframing restoration as hierarchical planning with closed-loop self-refinement, the work offers a template for domain-specific agentic systems that general-purpose degradation-aware pipelines cannot match. It also provides a benchmark of evidence that agentic orchestration, not just a stronger backbone, can drive large gains on real, heterogeneous degradations.

Real-world applications

• Museum conservation and digital archives, restoring damaged rubbings and stone inscriptions for preservation and display, an application directly reflected by the involvement of Nanjing University Museum.
• Digital humanities and historical linguistics, recovering textual content from fragmented inscriptions so that lost records can be read and studied.
• Calligraphic and stylistic record-keeping, preserving the aesthetic character of historical scripts rather than producing visually generic reconstructions.
• Expert-assist workflows in epigraphy, where the system proposes restorations and scholars accept, reject, or correct them via the built-in human feedback channel.

Industry relevance. Organizations digitizing cultural heritage—museums, libraries, national archives, and cultural-tech companies—face volumes of degraded material that far exceed expert capacity. A system that composes specialized tools adaptively, logs its own experience, and improves with expert review offers a scalable alternative to either manual restoration or one-size-fits-all pipelines, and it demonstrates a general pattern for deploying agentic AI in expert domains where both objective accuracy and subjective fidelity matter.

Future Directions

• Scaling and automating expert involvement. Human feedback produces the highest gains in the ablation study, but the paper describes it as optional and, in high-stakes cases, a hard criterion requiring third-party experts. How to reduce dependence on scarce epigrapher time while retaining the benefit is unresolved.

• Extending beyond the current script and dataset. Evaluation is confined to Chinese inscription rubbings in CIRI. Whether the Observe–Conceive–Execute–Reevaluate structure and the specific toolkit transfer to other scripts, materials, or languages is not established.

• Better baselines and reproducibility. The paper notes that existing agentic frameworks for natural images cannot be directly compared because their perception modules and restoration tools cannot readily transfer to inscription restoration without substantial redesign, and that some recent tailored methods are difficult to reproduce faithfully due to unavailable components. Closing this gap would sharpen the comparison.

• Tuning the stopping and rollback machinery. The framework depends on thresholds for text authenticity and style consistency and a maximum iteration budget, whose concrete values are not reported in the paper content. How sensitive performance is to these settings, and how to set them automatically, remains open.

Target Audience

Researchers and practitioners working on agentic AI systems and multimodal LLM orchestration; computer vision researchers interested in image restoration, document enhancement, and text-image models; digital humanities and epigraphy scholars seeking computational tools for inscription analysis; and cultural heritage technologists and museum digitization teams evaluating AI-assisted restoration workflows. Readers without a background in diffusion models, RAG, or image-quality metrics will find the system-level framing accessible but the experimental sections demanding.

Authors’ abstract

Ancient inscriptions, as repositories of cultural memory, have suffered from centuries of environmental and human-induced degradation. Restoring their intertwined visual and textual integrity poses one of the most demanding challenges in digital heritage preservation. However, existing AI-based approaches often rely on rigid pipelines, struggling to generalize across such complex and heterogeneous real-world degradations. Inspired by the skill-coordinated workflow of human epigraphers, we propose EpiAgent, an agent-centric system that formulates inscription restoration as a hierarchical planning problem. Following an Observe-Conceive-Execute-Reevaluate paradigm, an LLM-based central planner orchestrates collaboration among multimodal analysis, historical experience, specialized restoration tools, and iterative self-refinement. This agent-centric coordination enables a flexible and adaptive restoration process beyond conventional single-pass methods. Across real-world degraded inscriptions, EpiAgent achieves superior restoration quality and stronger generalization compared to existing methods. Our work marks an important step toward expert-level agent-driven restoration of cultural heritage. The code is available at https://github.com/blackprotoss/EpiAgent.