Ancient-Bench: A Comprehensive Multi-millennial, Multi-medium, and Multi-script Benchmark for Ancient Chinese Artifact Text Recognition

AI summary

Overview

• Research area: Computer vision and document analysis — specifically optical character recognition (OCR) for cultural heritage materials, evaluated through a new benchmark for ancient Chinese artifact text.
• Technical level: Intermediate. The paper is a dataset/benchmark paper; understanding the results requires some familiarity with OCR evaluation metrics (NED, F1) and vision-language models, but the core idea is accessible without deep technical background.
• Scope: Ancient-Bench is a 2,700-image benchmark spanning 3,000 years of Chinese character evolution, nine artifact media, and seven historical script forms, with three unified annotation standards and a large-scale evaluation of general and OCR-specialist vision-language models.

What This Paper Is About

Ancient Chinese text appears on many physical materials (oracle bones, bronzes, slips, silk, seals, steles, cliffs, printed editions, calligraphy) across thousands of years and several script forms, but existing recognition benchmarks each cover only one medium, one era, or a partial set of scripts. The authors argue this "fragmentation" makes it impossible to test whether a model generalizes across eras, media, and scripts. They build Ancient-Bench to unify all three dimensions and then run a broad evaluation that shows how far current models still are from solving the task.

Key Contributions

1. A comprehensive benchmark, Ancient-Bench, with 2,700 annotated images covering 9 media types, 7 script forms, and spanning over 3,000 years (with the comparison table listing a timeline of 1200 BCE–1911 CE). Images come from 14 national-level cultural heritage institutions, with 1–4 institutions per medium type.
2. Three standardization protocols for annotation, tailored to ancient artifact media and knowledge systems: symbol standardization, character standardization, and parsing standardization.
3. A large-scale evaluation of general VLMs (closed-source and open-source) and OCR-specialist models (end-to-end and pipeline) on the benchmark, using character-level NED and F1.
4. Failure-mode analysis, including a dedicated hallucination study on a calligraphy subset, identifying variant/rare character confusion, missed specialized symbols, layout-induced reading-order errors, and systematic hallucination patterns.

Main Findings

• The task is unsolved at the benchmark level. The paper states that even the best-performing model, Doubao-seed-2-0-lite-260428, reaches only 52.08% NED and 57.86% F1 overall. The paper separately states that kimi-k2.6 "achieves state-of-the-art performance" (its table entry is 50.62% NED / 56.62% F1), so the text and the results table are not fully consistent on which model is best.
• General VLMs outperform OCR-specialist VLMs. This is attributed to the limitations of domain-specific OCR systems when faced with heterogeneous historical scripts and media.
• Pipeline OCR models beat end-to-end OCR models. Modular recognition pipelines were consistently stronger than end-to-end OCR-specific VLMs.
• Best specialist model. Within OCR-specific VLMs, HunyuanOCR was the strongest, at 40.42% NED and 46.11% F1.
• Weakest media. Oracle bone and bronze inscriptions were the hardest; the best Oracle result reported is Gemini-3.1-pro-preview with an F1 of only 15.75%, and the best Bronze result cited is Doubao-seed-2-0-lite-260428 at 38.48% F1. OCR-specialist VLMs completely failed on both.
• Strongest media. Steles were the easiest category, where Doubao-seed-2-0-lite reached 88.88%.
• Slip and Silk. The highest scores were 39.14% and 52.21% respectively; errors came from visually similar characters, complex archaic and rare characters, and special symbols. Damaged regions in silk triggered hallucinated output.
• Seals. OCR-specific VLMs struggled (F1 18.59%) while general VLMs reached 44.56%; because seal inscriptions often contain classical poetry, general VLMs produced hallucinations.
• Cliff and Edition. Both pose reading-order challenges — Cliff from natural scenes, Edition from layout design. OCR-specific VLMs showed a gap of roughly 6–10 points between NED and F1.
• Calligraphy. Most models performed poorly on cursive and running scripts, and frequently missed or misrecognized distinctive calligraphic symbols.
• Two hallucination patterns identified. (1) Prior-driven semantic completion, where language priors produce fluent but non-grounded tokens under complex styles such as stroke adhesion and deformation; (2) cropping/detection-induced over-recognition, where inaccurate crops or boxes include neighboring strokes or noise, producing spurious extra characters and concatenated readings.
• Scale and diversity statistics. Resolutions span 41×80 to 5,795×16,745 pixels. Text length is skewed toward short texts: short-text media (seals, oracle bones, bronzes) span 1–77 characters, medium-text media (slips, cliff inscriptions, steles) span 2–816 characters, and long-text media (silk manuscripts, editions, calligraphy) reach 361–1,666 characters.

Methodology in Plain English

The authors first collected images from 14 authoritative institutions such as museums and libraries, then let the taxonomy of media, scripts, and periods emerge from the actual artifacts rather than imposing categories in advance. Because institutional data varied in format and annotation quality, they built a four-step pipeline: image preprocessing (converting PDFs to high-resolution images, using MLLMs to filter images with watermarks or irreversible damage, and cropping text regions), symbol standardization, character standardization, and parsing standardization.

The three annotation standards are the paper's main methodological innovation. Symbol standardization keeps repetition marks "=" and "-" as they appear, uses a square placeholder (□) for damage — a single square for a whole region when the number of missing characters cannot be determined — and uses <unrecognizable> for ancient forms not encoded in Unicode. Character standardization follows a "what you see is what you get" principle: ancient oracle bone and bronze characters are converted to clerical or regular forms (transcription) using traditional characters as the reference, as when the pictographic character for "horse" (马) is annotated as 馬; simplified and traditional forms are recorded as they appear, since merged characters like 後/后 carry distinct meanings in Classical Chinese; unencoded archaic characters are either replaced by an attested variant (a character with the structure 虎口虫虫 is annotated as 虐) or marked with <unrecognizable>. Parsing standardization records line breaks, inter-line spacing (<space>), double-line interlinear notes in parentheses, reading order (vertical text right-to-left, top-to-bottom; horizontal right-to-left; cliff inscriptions follow the body text), and printed-edition layout elements (<ignore></ignore> for non-transcription zones, <note></note> for annotation zones). Collector and connoisseur seals from later periods are not annotated except in the Seal category.

Annotation took six months and involved 20 trained annotators, character-by-character verification against paleographic reference tools (Yinqi Wenyuan, Guyin Xiaojing, Zitong Wang, and Shuowen Jiezi), feedback from domain experts, and independent double-checking. Evaluation used character-level F1 and Normalized Edit Distance (NED), defined as 1 minus the Levenshtein distance between prediction and ground truth divided by the longer of the two string lengths.

Why This Matters

For research, Ancient-Bench gives cultural heritage OCR a single comparable test bed across media, eras, and scripts, replacing a landscape where each benchmark measured a different slice of the problem. Its failure-mode and hallucination analyses are as important as its leaderboard: they show that strong general-purpose VLMs still produce confident, fluent, and wrong output when confronted with degraded or complex ancient glyphs, which is a serious concern for any downstream scholarly use of automated transcription.

Real-world applications include:

• Digitization and cataloguing programs at museums and libraries holding oracle bones, bronzes, slips, silk manuscripts, seals, steles, cliff inscriptions, printed editions, and calligraphy.
• Digital humanities and philology research that depends on faithful transcription of variant, rare, and unencoded characters.
• Textual criticism and scholarly editing, where the paper's distinction between transcription, simplified/traditional forms, and placeholders preserves evidentiary value.
• Quality assurance for AI systems deployed in heritage workflows, where hallucinated or reading-order-scrambled output could silently corrupt archival records.

Industry relevance centers on the model builders themselves. The paper reports that domain-specific OCR systems underperform general VLMs on this task, and that pipeline architectures outperform end-to-end ones — concrete signals for teams building document AI and multimodal OCR products, and for anyone evaluating whether an off-the-shelf VLM can be trusted on historical material.

Future Directions

• Character-level bounding boxes. The current dataset provides only line-level or region-level transcriptions; the authors state that character-level spatial annotations would better support fine-grained text detection evaluation and plan to enrich them in future updates.
• Data contamination. Because the data come from publicly available repositories, the authors cannot guarantee that open-source or closed-source models were not trained on portions of these materials, which is an open question for the validity of future leaderboard comparisons.
• Closing the oracle bone and bronze gap. The extremely low scores on these earliest media (best F1 reported at 15.75% for Oracle) point to a need for methods that specifically handle transcription from archaic pictographic forms.
• Robustness to hallucination and reading order. The identified failure modes — prior-driven semantic completion, crop-induced over-recognition, variant/rare character confusion, and layout-induced reading-order errors — remain unsolved targets that the benchmark can now measure.

Target Audience

Researchers and engineers working on OCR, document analysis, and multimodal vision-language models, particularly those interested in historical and cultural heritage text. It is also relevant to digital humanities scholars and to museum or library staff planning digitization programs, who need to understand what current automated transcription can and cannot reliably do. Benchmark designers will find the three annotation standards useful as a template for handling heterogeneous, symbol-rich, and historically layered source material.

Authors’ abstract

Ancient Chinese artifact text recognition is fundamental to heritage digitization, and benchmarks for ancient texts are essential for evaluating current model capabilities. However, existing benchmarks suffer from ''fragmentation'', manifested in limited temporal coverage, limited medium diversity, and incomplete script types. Therefore, we present Ancient-Bench, a comprehensive benchmark of 2,700 images for ancient Chinese artifact text recognition, featuring three dimensions: Multi-millennial (spanning 3,000 years of character evolution), Multi-medium (covering nine artifact categories), and Multi-script (encompassing seven historical script forms). To enable consistent and fair evaluation across heterogeneous media, we further define three annotation standards tailored to the medium-specific characteristics of ancient texts: symbol standardization, character standardization, and parsing standardization. Extensive experiments on Ancient-Bench covering general Vision-Language Models (VLMs) and OCR-specialist models reveal that ancient Chinese artifact text recognition remains fundamentally unsolved, with persistent challenges in variant characters, specialized symbols, and hallucination. The dataset is available at https://github.com/SCUT-DLVCLab/Ancient_Bench.