CHisAgent: A Multi-Agent Framework for Event Taxonomy Construction in Ancient Chinese Cultural Systems

AI summary

Overview

• Research area: Natural Language Processing and computational linguistics, specifically multi-agent large language model (LLM) systems applied to historical knowledge organization and digital humanities.
• Technical level: Advanced. The paper assumes familiarity with LLM prompting and role assignment, taxonomy induction (hierarchy building), embedding-based semantic similarity, and standard taxonomy evaluation metrics.
• One-sentence scope: The paper introduces CHisAgent, a three-agent LLM framework that automatically builds a large-scale event taxonomy for ancient Chinese history from the Twenty-Four Histories, and evaluates it with both automatic and human judgment.

What This Paper Is About

Historical Chinese documents such as the Twenty-Four Histories span a very long time period and use highly varied, condensed expressions, which makes it hard for LLMs to reason about Chinese history and culture. Taxonomies (hierarchies of "is-a" relations) help organize such knowledge, but hand-building them is labor-intensive, so existing historical taxonomies stay small and coarse. The paper's goal is to automate this process with a multi-agent LLM framework that induces, expands, and enriches an event-centered taxonomy of ancient Chinese history.

Key Contributions

1. CHisAgent, a role-specialized multi-agent framework. The framework decomposes historical taxonomy construction into three stages, each handled by dedicated LLM agents: an Inducer (bottom-up hierarchy induction), an Expander (top-down insertion of missing intermediate concepts), and an Enricher (evidence-guided integration of external structured resources).

2. A large-scale event taxonomy for ancient Chinese history and culture. Built from the Twenty-Four Histories, the taxonomy spans over 3,000 years and covers major political, military, diplomatic, and social events, organized across eight domains.

3. Extensive dual-track evaluation. The authors report both reference-free metrics (Path Granularity, CSC, Coverage Rate) and reference-based metrics against the human-curated CHED taxonomy (Node Recall, Novelty, Significance), plus a human evaluation with three domain experts.

4. Evidence of cross-cultural generality. The induced taxonomy is applied to Japanese, Korean, and Vietnamese classical Chinese corpora, and a case study on tribute events shows it can align semantically related practices across cultures while preserving differences in narrative perspective.

Main Findings

• Best coverage and node recall among compared methods. On the reference-free/reference-based table, CHisAgent (Corpus + Domain) reaches a Coverage Rate of 75.13 and Node Recall of 68.89, the best in both columns, with Path Granularity 91.54, CSC 0.2784, Novelty 77.70, and Significance 0.1887. For comparison, TaxoAdapt (Corpus + Domain) scores 70.98 Path Granularity, 0.2075 CSC, 74.17 Coverage Rate, 60.00 Node Recall, 69.21 Novelty, and 0.3927 Significance.

• Single-agent methods win on structure, multi-agent methods win on coverage. Single-agent generation achieves higher Path Granularity and CSC (for example GPT-4o Corpus at 100.00 Path Granularity and 0.3931 CSC) because it builds the whole taxonomy in one unified process, but multi-agent decomposition yields better structural coverage and alignment with the reference taxonomy. Traditional induction methods (HiExpan, HiSetExpan) show no clear advantage in this large event-type space.

• TaxoAdapt produces the most fine-grained taxonomy. TaxoAdapt obtains the highest Significance score (0.3927), and Chain-of-Layer (Domain) achieves the best Novelty (80.92).

• All three agent modules are complementary. In the ablation, Inducer alone gives the lowest Node Recall (61.11) and Coverage Rate (71.61). Adding the Expander raises Path Granularity to 94.35 and Node Recall to 65.56, with a slight CSC drop to 0.2464. Adding the Enricher improves Coverage Rate to 75.13, Node Recall to 68.89, and Significance to 0.1887.

• Human evaluation shows balanced quality. Over three independent evaluators with Chinese history expertise, agreement was a weighted Cohen's kappa of 0.6982, Spearman correlation 0.5722, and ICC(2,1) of 0.6992. TaxoAdapt scored highest on Clarity and Completeness but lower on Cross-domain Orthogonality; GPT-4o (Corpus + Domain) scored highest on Cross-domain Orthogonality but lower on completeness. CHisAgent was reported as the most balanced across all five dimensions.

• Event extraction is reliable but imperfect. On 10 Classical Chinese passages, Qwen3 extracted 242 events with 195 correct, 101 missed, precision 0.8058 and recall 0.6588. DeepSeek-V3 extracted 237 events with 197 correct, 120 missed, precision 0.8312 and recall 0.6215. Annotator agreement on event correctness was 0.83, and the paper states the two extractors obtain comparable F1 scores. A separate appendix evaluation of 400 sampled event instances found 85% accuracy, with factual extraction errors under 5% of samples.

• Evaluation is robust to the choice of LLM judge. Re-running the Path Granularity evaluation with GLM-5.2 at temperature 0 reproduced the relative ordering: CHisAgent went from 91.54 (GPT-4o judge) to 82.67, HiExpan from 62.80 to 57.81, HiSetExpan stayed at 83.33, CoL from 80.99 to 71.53, and TaxoAdapt from 70.98 to 70.62.

• The taxonomy transfers across East Asian corpora. On Nihon Shoki, Joseon Sillok, and Dai Viet Su Ky Toan Thu (roughly 300–400 event types each), CHisAgent reaches Coverage Rates of 80.50 (Japan), 75.51 (Korea), and 60.55 (Vietnam). Every method except CHED scores lowest on the Vietnamese corpus. In the tribute case study, Chinese and Japanese records predominantly match the incoming tribute (來貢) perspective, Vietnamese records more often match outgoing tribute (進貢), and Korean records show a mixed pattern.

• Hierarchical over-segmentation is the dominant error. Among sampled nodes, 32% received low scores (lower than 4) for hierarchical coherence. The paper attributes this to near-synonymous intermediate nodes, citing examples such as "任用 (Appointment) → 任命 (Nomination)" and "辯解 (Justification) → 解釋 (Explanation)".

Methodology in Plain English

The authors build an event-centered taxonomy from the Twenty-Four Histories, selecting chapters of the types 傳 (Biographies), 誌 (Treatises), 書 (Records), and 紀 (Annals), which yields 192 chapters and 584,096 characters. The pipeline then runs in three stages staffed by different LLMs.

In the Inducer stage, multiple extractor agents (DeepSeek-V3 and Qwen3) pull event instances and event types out of each chapter. A classifier agent (GPT-4o) assigns each event type to one of eight domains: Politics, Military, Diplomacy, Society, Ritual, Economy–Livelihood, Nature, and Individual. Within each domain, three Generator LLMs merge semantically related event types into higher-level nodes, and a Merger LLM (GPT-5) deduplicates and unifies concepts. The loop repeats until a domain has fewer than ten high-level categories, typically after two or three rounds.

In the Expander stage, GPT-5 acts as a judger that resolves semantic duplicates (using cosine similarity over text-embedding-3-small embeddings, with 0.6 as the redundancy threshold) and also as an expander. Working layer by layer from the top down, the model looks for missing sibling categories, missing intermediate abstractions, and inconsistent parent–child assignments, then inserts nodes and reassigns children. A final global deduplication keeps the structure compact.

In the Enricher stage, the framework collects candidate events from three sources: high-frequency events from the initial extraction (frequency above 5), topic-based events derived with BERTopic, and ontology-derived relations from external resources such as the Chinese Biographical Database (CBDB) and the relation ontology of Tang et al. (2025). An LLM-based Event Conceptualizer (GPT-4o) converts topics and relations into explicit event-type candidates, after which GPT-5 assigns each surviving candidate to a domain and places it in the taxonomy if it is not a duplicate.

Evaluation combines reference-free metrics (Path Granularity, CSC, Coverage Rate) with reference-based metrics against the human-curated CHED taxonomy (Node Recall, Novelty, Significance), plus human scoring on Node Clarity, Hierarchical Coherence, In-domain Orthogonality, Cross-domain Orthogonality, and Completeness on a 0–5 scale.

Why This Matters

• Research impact: The work shows that decomposing historical taxonomy construction into distinct, role-specialized LLM stages outperforms both single-agent prompting and traditional taxonomy induction methods on coverage-oriented metrics, while remaining competitive on structure. It also provides a reusable, publicly released resource—code and experimental data are at https://github.com/tangxuemei1995/CHisAgent—and offers a template for automating knowledge organization in other long-span historical corpora.

• Historical information retrieval: The paper explicitly positions taxonomies as support for downstream historical information retrieval, and states that future work will integrate the taxonomy into a historical event retrieval system.

• Cultural knowledge alignment: The cross-cultural experiments show the taxonomy can align semantically related practices (such as tribute relations) across Chinese, Japanese, Korean, and Vietnamese records while preserving their differing narrative perspectives.

• Knowledge graph and ontology construction: Taxonomies are described as a semantic foundation for building knowledge graphs and ontologies, which matters for cultural heritage projects that model artifacts and historical entities.

• Historically grounded reasoning in LLMs: Because LLMs struggle with historical and cultural reasoning outside English contexts, a structured taxonomy offers a way to supply hierarchy-level knowledge rather than raw text alone.

• Industry relevance: Organizations working on digital libraries, cultural heritage archives, search and knowledge platforms, and educational technology for historical content can use such taxonomies as scaffolding for indexing, browsing, and knowledge access. The paper cautions that generated taxonomies are research aids and should not be treated as authoritative historical interpretations or replacements for expert judgment.

Future Directions

1. Ontology refinement under stronger logical constraints, to address the mixed ontological criteria and weak conceptual distinctions that cause hierarchical over-segmentation.

2. Temporal and causal event modeling, extending the current event-centered hierarchy to capture when and why events occur and how they relate.

3. Validation across multilingual historical corpora, broadening the cross-cultural study beyond the Japanese, Korean, and Vietnamese sources tested here.

4. Stability analysis across repeated runs. Because LLM agents introduce stochasticity, the authors note that running the framework multiple times may produce slightly different taxonomies, and a systematic study of structural variance is left for future work.

Additional open questions raised in the paper include mitigating bias inherited from LLM pretraining and from the Twenty-Four Histories' official and elite perspective, which underrepresents folk practices and regional traditions; and handling cultural phenomena that are not naturally event-driven, such as philosophical traditions or long-term ideological shifts.

Target Audience

This paper is most valuable to NLP and computational linguistics researchers working on taxonomy induction, multi-agent LLM systems, or knowledge organization; digital humanities and Chinese history scholars who need large-scale structured representations of historical sources; and engineers building retrieval, knowledge graph, or cultural heritage platforms who want a concrete pipeline design and an honest accounting of evaluation trade-offs. Readers should be comfortable with LLM prompting and taxonomy evaluation terminology, since the paper does not explain these from first principles.

Authors’ abstract

Despite strong performance on many tasks, large language models (LLMs) show limited ability in historical and cultural reasoning, particularly in non-English contexts such as Chinese history. Taxonomic structures offer an effective mechanism to organize historical knowledge and improve understanding. However, manual taxonomy construction is costly and difficult to scale. Therefore, we propose \textbf{CHisAgent}, a multi-agent LLM framework for historical taxonomy construction in ancient Chinese contexts. CHisAgent decomposes taxonomy construction into three role-specialized stages: a bottom-up \textit{Inducer} that derives an initial hierarchy from raw historical corpora, a top-down \textit{Expander} that introduces missing intermediate concepts using LLM world knowledge, and an evidence-guided \textit{Enricher} that integrates external structured historical resources to ensure faithfulness. Using the \textit{Twenty-Four Histories}, we construct a large-scale, domain-aware event taxonomy covering politics, military, diplomacy, and social life in ancient China. Extensive reference-free and reference-based evaluations demonstrate improved structural coherence and coverage, while further analysis shows that the resulting taxonomy supports cross-cultural alignment.