LexRel: Benchmarking Legal Relation Extraction for Chinese Civil Cases

AI summary

Overview

Research area: Natural Language Processing, specifically legal AI and information extraction for Chinese civil law.

Technical level: Intermediate. The paper is accessible to readers familiar with relation extraction and LLM evaluation, but some grounding in Chinese civil law concepts (rights, obligations, causes of action) helps.

Scope: This paper introduces a structured schema of Chinese civil legal relations, the LexRel benchmark for extracting those relations from case facts, and an evaluation of current LLMs on that task.

What This Paper Is About

Legal relations — the rights-and-obligations connections between parties, such as a creditor-debtor relationship — are a core analytical unit in Chinese civil cases, but they are rarely explicitly stated in judgments and are not well covered by existing legal AI resources, whose schemas tend to be coarse-grained. The authors build the first comprehensive schema of Chinese civil legal relations (a hierarchical taxonomy with argument definitions), turn it into an expert-annotated extraction benchmark called LexRel, and use it to test whether today's large language models can recover these relations from factual case text.

Key Contributions

1. A comprehensive legal relation schema. The authors propose a hierarchical taxonomy covering 9 major domains of Chinese civil law and 265 fine-grained relation types, together with precise definitions of the subject, object, and content arguments.
2. A new task and benchmark. They define legal relation extraction as two subtasks (type extraction and argument extraction) and construct LexRel, an expert-annotated benchmark of 1,140 annotated samples (described in the long-tail analysis as 1,140 civil cases and 1,863 fully annotated legal relations).
3. An evaluation of state-of-the-art LLMs. They benchmark 12 models across a zero-shot baseline and a relation-enhanced (supervised fine-tuning) baseline, showing substantial limitations on the task.
4. Evidence that legal relations help downstream tasks. They show that injecting extracted legal relations into LawBench tasks improves performance for most models and tasks.

Main Findings

• LLMs struggle on zero-shot type extraction. The best zero-shot micro-F1 on type extraction is o3-mini at 0.762, followed by DeepSeek-R1 at 0.693. Among open-source models, Qwen3-8B, Qwen3-14B, and Qwen3-32B score 0.464, 0.583, and 0.583 micro-F1 respectively.
• Argument extraction is much harder. The best zero-shot argument extraction micro-F1 is again o3-mini at 0.382, followed by DeepSeek-R1 at 0.268 and Claude-Sonnet-4 at 0.258. Most open-source models fall below 0.2 micro-F1.
• Supervised fine-tuning from distilled relations helps substantially. Distilling from GPT-4o and DeepSeek-R1 into 8B and 14B models improves both tasks; the best relation-enhanced type extraction result is Qwen3-14B at 0.733 micro-F1, close to o3-mini's zero-shot score. InternLM3-8B's argument extraction micro-F1 rises from 0.048 to 0.323.
• Micro-F1 consistently exceeds macro-F1. Across every setting, macro-F1 is lower than micro-F1, indicating that model capability varies widely across relation types.
• The benchmark mirrors real-world long-tail distributions. Legal relations and causes of action in LexRel show a downward frequency trend. Using cause of action as a proxy (validated with a chi-square test, p < 0.05, Cramér's V = 0.528), the authors find that 26.6 million real-world civil court judgments follow a long-tail distribution, with the top 25 case types accounting for 80% of all civil judgments.
• Legal relations improve downstream legal AI tasks. Adding relation-aware inputs ("w/ LR") improves nearly all combinations on LawBench tasks: for example, MiniCPM4-8B rises from 32.0 to 45.0 on Case Analysis and from 65.0 to 76.8 on Criminal Damages Calculation, and GPT-4o rises from 84.4 to 85.8 on Calculation.
• Two dominant error sources. Type extraction errors come from confusing legal relations with general social relations (occurring in 81.1% of Llama3.1-8B-Instruct's predictions and 28.6% of o3-mini's) and from long-tail sparsity. Argument extraction errors come from object misidentification and content omission.
• Evaluation reliability was checked. The LLM-as-a-Judge approach using DeepSeek-V3 was validated on 60 items yielding 193 pairs, with subject accuracy 0.954, object accuracy 0.969, and content accuracy 0.810. Inter-annotator agreement on a 300-instance sample yielded a Cohen's Kappa of 0.706.

Methodology in Plain English

The authors first built the taxonomy in two stages. They used keyword matching over Chinese civil court judgments to pull out candidate legal relation terms, kept only those with legally recognized prefixes, and arrived at a draft of 123 candidate relation types across six domains. Two senior legal scholars then refined the draft using textbooks and legal doctrine, adding three new domains (Bill Relations, Letter of Credit Relations, and Independent Guarantee Relations) and expanding the taxonomy to 265 relation types across 9 domains. Because the distribution of types across domains is uneven, Legal Relations of Obligations holds the largest share at 200 types.

Next they defined the arguments of each relation — subject, object, and content — for 8 of the 9 domains plus 4 subdomains under Legal Relations of Obligations (Contractual Legal Relations, Legal Relations of Tort Liability, Legal Relations of Negotiorum Gestio, and Legal Relations of Unjust Enrichment), producing 12 distinct definitions.

To build LexRel, they had DeepSeek-V3 draft relation types and arguments from full-text judgments and separately extract the factual text recognized by the court. Six legal experts, each assigned 200 items, then verified and refined the facts, types, and arguments, overseen by a senior legal AI expert. Sixty items with no identifiable legal relations were removed, leaving 1,140 samples.

For evaluation, the zero-shot baseline tests models directly on LexRel. The relation-enhanced baseline gives models the full judgment text instead of only the facts, uses the generated relations as supervision, and fine-tunes smaller models via SFT on 5,500 synthetic training samples. Only 8B and 14B open-source variants were fine-tuned due to training cost constraints, and all experiments ran on 4× A800 GPUs with 40GB memory each. Scoring uses precision, recall, micro-F1, and macro-F1; type extraction is scored by exact match, while argument extraction uses DeepSeek-V3 as a judge comparing predicted and gold arguments.

Why This Matters

Impact on research. The paper provides the first operational, fine-grained schema of Chinese civil legal relations, moving legal AI beyond general social relations and entity co-occurrence toward legally meaningful structures. It also establishes a reusable evaluation benchmark with published data and code (https://github.com/thunlp/LexRel) and quantifies how far current LLMs are from solving the task.

Real-world applications:

• Legal information retrieval that matches cases by underlying legal relationship rather than surface wording.
• Statute prediction and case outcome analysis, where identifying the correct relation points to the governing rules.
• Structured case analysis and legal consultation, where relation identification is a prerequisite step.
• Criminal damages calculation, where relations such as the Legal Relation of Ownership help identify which property was infringed.

Industry relevance. Legal tech vendors, court digitization projects, and developers of legal LLMs could use the schema and benchmark to build relation-aware tools, and the finding that distilled relation knowledge improves smaller models on LawBench tasks suggests a practical path for resource-constrained deployments.

Future Directions

• Cross-jurisdiction transfer. The authors explicitly note the schema and dataset are limited to Chinese civil law; adapting the conceptual structure to other civil law systems would require localized doctrines and terminology.
• Reducing model-family coupling. Both synthetic data generation and the LLM-as-a-Judge evaluation rely on DeepSeek-family models. The authors suggest diversifying model sources and refining evaluation protocols to reduce perceived stylistic dependencies.
• Closing the zero-shot gap. Since fine-tuned models still trail the best zero-shot reasoning model on type extraction and all models remain weak on argument extraction, better methods for structured legal extraction are needed.
• Improving long-tail performance. With rare relation types systematically misclassified into high-frequency categories, techniques for handling imbalanced relation distributions remain an open problem.

Target Audience

This paper is most useful for legal AI and legal NLP researchers, builders of legal knowledge graphs and legal LLMs, and computational law scholars working on Chinese civil law. It also benefits evaluation-focused researchers interested in domain-specific benchmarks, and practitioners in legal tech who need structured representations of case facts for retrieval, consultation, or outcome prediction systems.

Authors’ abstract

Legal relations serve as an important analytical framework for dispute resolution in civil cases. However, legal relations in Chinese civil cases remain underexplored in the field of legal AI, largely due to the absence of comprehensive schemas. In this work, we first introduce a comprehensive schema for legal relations in civil cases, which contains a hierarchical taxonomy and definitions of arguments. Based on this schema, we formulate a legal relation extraction task and present LexRel, an expert-annotated benchmark for legal relation extraction in the Chinese civil law domain. We use LexRel to evaluate state-of-the-art large language models (LLMs) on legal relation extraction, showing that current LLMs exhibit significant limitations in accurately identifying civil legal relations. Furthermore, we demonstrate that explicitly incorporating information about legal relations leads to promising performance gains on other downstream legal AI tasks.