CAPC-CG: A Large-Scale, Expert-Directed LLM-Annotated Corpus of Adaptive Policy Communication in China

AI summary

Overview

• Research area: Natural Language Processing, specifically Chinese-language corpus construction, annotation methodology, and document classification; the work sits at the intersection of computational linguistics and comparative political science.
• Technical level: Intermediate. The NLP methods (fine-tuning LLMs, kappa-based agreement metrics, LDA topic modeling) require some familiarity, but the paper's core argument is conceptual and readable by non-specialists.
• Scope (one sentence): The paper introduces CAPC-CG, an open corpus of 337,038 Chinese central-government policy documents (1949–2023) segmented into 3,275,477 paragraphs and annotated with a five-color typology of policy signals, together with an expert-directed LLM annotation method and baseline classification results.

Note: the paper content provided ends partway through Appendix C, so several appended materials referenced in the main text (full open-weight benchmarks in Appendix F, failure cases in Appendix H, the field-level schema in Appendix I, topic-modeling details, and descriptive statistics) are cited but not reproduced here.

What This Paper Is About

Central authorities in China govern a vast, decentralized bureaucracy by issuing written policy directives that local officials must interpret and act on. Some directives are clear ("you can," "you cannot"), while others are deliberately ambiguous, allowing local flexibility. Before this paper, no dataset systematically captured this variation in clarity and ambiguity at scale. The authors build one, and in doing so propose a way to combine expert human judgment with LLM scaling to label millions of paragraphs.

Key Contributions

1. A five-color typology of policy signals, operationalized. Building on Ang's original three categories (Black = Authorizing, Red = Prohibiting, Grey = Ambiguous), the authors add Yellow (Pressuring) and Charcoal (Flexible) to produce Black, Red, Grey, Yellow, Charcoal, plus a Neutral label for non-directive paragraphs. They release a gold-standard labeled set of about 6,000 paragraphs with high inter-annotator agreement.
2. A large-scale, computable corpus of Chinese policy directives. CAPC-CG contains 337,038 documents spanning 1 October 1949 to 13 March 2023, segmented into 3,275,477 paragraph units (2,647,695 after filtering), with normalized metadata, de-duplication, and paragraph-level segmentation.
3. An expert-directed LLM annotation method. The approach combines an expert-designed codebook, a three-month annotator training program, a two-stage labeling workflow, and LLM fine-tuning to scale annotation — treating human expertise as the foundation rather than as something LLMs replace.
4. Released code, guidelines, evaluation splits, and baseline models for reproducible research, including a cost-optimized segmentation pipeline that reduced processing cost by 72.20% ($294.99 versus an estimated $1,061.30).

Main Findings

• High human agreement: Inter-annotator agreement on directive labels reached a Fleiss' kappa of κ = 0.86 (reported as K = 0.864 at the end of the training phase). During training, agreement rose from 0.670 to 0.864 across three independent coding rounds, while expected agreement shifted only from 0.460 to 0.508 — the authors attribute the gain to strengthened coding rules rather than forced consensus.
• Segmentation quality validated: Human annotators checked 4,600 randomly sampled segmented paragraphs and found 97.74% were correctly segmented.
• Fine-tuning beats prompting: For Level-1 classification (R/W/N) with GPT-4o-mini, fine-tuning achieved Kappa = 0.841 and Macro-F1 = 0.844, versus 0.618/0.692 zero-shot, 0.686/0.724 for few-shot (k=3), and 0.659/0.719 for few-shot (k=6). The authors describe this as a 22.6% improvement in Kappa over the best few-shot baseline.
• Zero-shot model comparison: GPT-4o-mini had the best zero-shot API metrics (Kappa = 0.618; Acc. = 0.793; M-F1 = 0.692) and was chosen for its balance of performance, cost, and latency. Others: Qwen3-235B (0.570/0.780/0.700), DeepSeek-V3 (0.481/0.733/0.622), SVM (0.175/0.508/0.442), XGBoost (0.153/0.487/0.420).
• Strong Level-2 color classification: The fine-tuned GPT-4o-mini model reached 87.5% overall accuracy and a 4-class Cohen's Kappa of 0.833, with per-class accuracy of 92.0% (B, Authorizing), 85.0% (C, Flexible), 84.0% (G, Ambiguous), and 89.0% (Y, Pressuring).
• Open-weight models trail but are competitive: Fine-tuned BERT-base-Chinese reached Kappa = 0.817 / M-F1 = 0.846 on Level-1, and Kappa = 0.800 / Acc. = 0.850 on Level-2, still behind fine-tuned GPT-4o-mini, especially on Grey (Ambiguous) cases.
• Errors cluster in interpretable ways: Level-1 mistakes mainly involve confusing affirmative directives with prohibitions in passages authorizing the state to sanction others. Level-2 errors concentrate on the Charcoal (Flexible) versus Grey (Ambiguous) distinction — separating explicit encouragement of flexibility from strategic hedging.
• Descriptive pattern: Red (Prohibiting) directives form only a small share compared to Black (Authorizing), especially after the Reform and Opening-Up period began in 1978, and the pattern appears tied more to major events such as economic shifts or public health crises than to leadership transitions.
• Domain-specific signaling: Across 17 policy domains, the intent to "urge flexible means" (C) dominates technology-driven domains such as Digital Transformation (0.91) and Science & Technology Innovation (0.91). Water Resources is offered as a counterexample, where flexible-means signaling is comparatively weak because governance relies on specified targets, procedures, and technical standards.

Methodology in Plain English

The team first defined what counts as a policy directive: a formally issued document in which a higher-level authority addresses state or public-sector actors with actionable implications. They collected three types — National Laws (3,137 documents), Administrative Regulations (11,621), and Ministerial Rules (322,280) — and built standardized metadata for each (title, validity status, hierarchical level of legal force, issuing body, reference number, promulgation date, entry-into-force date).

Segmentation used a hybrid pipeline rather than a single LLM. A local script numbered each line of a document; the model GPT-4.1-mini-2025-04-14 then returned only a compact JSON object listing structural labels and line numbers, rather than echoing the entire document. A deterministic local script merged that JSON back with the original text. This design kept output tokens to roughly 5–10% of input tokens and cut costs sharply.

Annotation proceeded in two rounds. Three domain experts underwent three months of training — reading sample directives, attending a codebook workshop, and coding roughly 500 paragraphs per week with discussion. After training, they coded 1,000 randomly sampled paragraphs weekly without replacement until reaching 6,000 annotated samples. At Level-1, each paragraph was labeled W (Affirmative, covering B/C/G/Y), R (Prohibitive), or N (Neutral). At Level-2, affirmative paragraphs were refined into B, C, G, or Y, with U (Unsure) reserved as a last resort. Mixed-signal paragraphs were resolved by a "proportion method": splitting into sentences, assigning a color per sentence, and labeling the paragraph by the largest proportion.

For modeling, the 6,000 gold-standard samples were split 80/20 into training and validation, with a separate 1,901-sample test set built collaboratively by the three experts. A prompt optimized through iterative testing on OpenAI models was fixed for all experiments. The authors benchmarked traditional machine learning (SVM, XGBoost), zero-shot LLMs, few-shot prompting (k=3 and k=6), and fine-tuning, then applied the winning strategy to the more granular Level-2 task. Fine-tuning used the OpenAI fine-tuning API with n_epochs=3 and automatic batch size and learning-rate multiplier. Error analysis and a topic model (LDA) were used to probe what the corpus reveals.

Why This Matters

Impact on research. The paper argues that existing Chinese-language datasets are mainly built from internet and social media content with limited quality control, while political and legal documents have been scarce and lack rigorous annotation. CAPC-CG addresses that gap and, importantly, reframes how LLM annotation should be used: not as a substitute for domain experts, but as a scaling layer on top of a codebook and training process that experts designed and validated. The high agreement figures and the large zero-shot-to-fine-tuned gap make a concrete case that expert supervision remains essential for subtle tasks like detecting ambiguity.

Real-world applications:

• Diachronic and cross-regime analysis of how central authorities mix clear and ambiguous signals over time.
• Information extraction from long, heterogeneous bureaucratic documents, including automated structuring and metadata assignment.
• Compliance monitoring, where distinguishing authorizing, prohibiting, and pressuring language matters for determining what downstream actors are actually required to do.
• Policy-impact assessment, using the corpus to connect shifts in directive style to outcomes in specific domains.

Industry relevance. The cost-optimization result is broadly applicable: the JSON-only output pattern for LLM segmentation delivered a verified 72.20% cost reduction, a template for anyone processing large document archives. The benchmark results also give practitioners a realistic picture of what zero-shot, few-shot, and fine-tuned models each deliver on a genuinely hard classification task — and where fine-tuned small models such as BERT-base-Chinese land relative to larger fine-tuned API models.

Future Directions

• Extend to sub-national levels. CAPC-CG currently covers central-level documents only. The authors propose collecting provincial and municipal directives to enable analysis of multi-level policy communication and diffusion, and integrating supplementary materials such as leaders' speeches and state media commentary.
• Improve ambiguity detection. The concentration of Level-2 errors on Charcoal (Flexible) versus Grey (Ambiguous) shows the task is not saturated. The authors state that ambiguity detection leaves substantial room for future work.
• Broaden benchmarking. The paper explicitly notes that its model experiments are not exhaustive; evaluating a wider array of open-weight models, adaptation strategies, and hyperparameter settings is left to the community, with the released dataset intended as a standard testbed.
• Anchor adaptive political economy (APE). The authors position the corpus as a high-resolution empirical resource for APE, a paradigm for studying political economies as complex adaptive systems, and contrast it with principal-agent models that assume central authorities hold clear goals.

Target Audience

NLP researchers working on Chinese-language corpora, document classification, and annotation methodology — particularly those interested in human-LLM hybrid labeling workflows. Computational social scientists and political scientists studying governance, lawmaking, and policy communication in China and comparatively will find the descriptive analyses and the typology most directly useful. Practitioners building large-scale annotation pipelines will benefit from the segmentation architecture and the reported cost savings. A working knowledge of Chinese policy terminology is not required to follow the paper, but is needed to use the corpus itself.

Authors’ abstract

We introduce CAPC-CG, the Chinese Adaptive Policy Communication (Central Government) Corpus, the first open dataset of Chinese policy directives annotated with a five-color taxonomy of clear and ambiguous language categories, building on Ang's theory of adaptive policy communication. Spanning 1949-2023, this corpus includes national laws, administrative regulations, and ministerial rules issued by China's top authorities. Each document is segmented into paragraphs, producing a total of 3.3 million units. Alongside the corpus, we release comprehensive metadata, a two-round labeling framework, and a gold-standard annotation set developed by expert and trained coders. Inter-annotator agreement achieves a Fleiss's kappa of K = 0.86 on directive labels, indicating high reliability for supervised modeling. We provide baseline classification results with several large language models (LLMs), together with our annotation codebook, and describe patterns from the dataset. This release aims to support downstream tasks and multilingual NLP research in policy communication.