AppellateGen: A Benchmark for Appellate Legal Judgment Generation

AI summary

Overview

Research area: Legal NLP / AI for law (arXiv category listed as AI Safety & Ethics, cs.CY). The work sits at the intersection of benchmark construction, long-context generation, and multi-agent LLM systems applied to Chinese civil-law adjudication.

Technical level: Advanced. The paper assumes familiarity with LLM prompting, retrieval-augmented generation, agent orchestration, and legal-domain evaluation metrics.

Scope (one sentence): The paper introduces a 7,351-pair benchmark for generating second-instance (appellate) legal judgments and evaluates an SOP-based four-agent system against general, reasoning, and legal-domain LLMs.

What This Paper Is About

Most legal AI research on judgment generation assumes a first-instance setting: a model reads a static set of facts and produces a verdict or document. That framing misses appellate review, which is inherently comparative — a judge must weigh the original judgment against an appellant's grievances and against new evidence introduced on appeal. The paper builds a benchmark for this second-instance task and proposes a structured multi-agent workflow to handle it.

Key Contributions

1. A new task definition — Second-Instance Legal Judgment Generation. The paper reframes legal NLP from static fact-to-verdict mapping toward dynamic dialectical reasoning, where a model must identify judicial errors and reconcile conflicting narratives between an original verdict and new evidence.

2. The AppellateGen benchmark. A dataset of 7,351 paired cases linking first-instance judgments to their appellate outcomes, drawn from China Judgments Online, with a fine-grained annotation schema covering reversal rationales, newly ascertained facts, and statutory citations.

3. The SLMAS framework. A judicial Standard Operating Procedure (SOP)-based Legal Multi-Agent System that decomposes generation into four stages — Analysis, Search, Predict, and Write — using specialized agents for issue identification, legal retrieval, reversal prediction, and document drafting.

4. An empirical finding about model type. General-purpose reasoning LLMs outperform legal-domain-specific models on this task, suggesting appellate reasoning depends more on comparative reasoning ability than on static legal knowledge.

Main Findings

• Full-system gains over single-agent: With Qwen3-235B-A22B-Instruct-2507, SLMAS raised the average LLM-as-a-Judge score from 2.257 to 2.536 and reversal prediction accuracy from 61.42% to 67.24%.

• Open-source plus agentic strategy beats a proprietary model: The 67.24% figure from SLMAS with Qwen3-235B-A22B-Instruct-2507 exceeds Gemini-2.5-Flash at 66.53% (average judge score 2.432).

• Domain-specific models lag badly: DISC-LawLLM reached 35.22% and Wisdom Interrogatory 37.34% reversal prediction accuracy, while general models exceeded 60%. The paper attributes this to a lack of cognitive flexibility for dynamic evidentiary conflicts.

• Generation quality remains the bottleneck: Even the best configuration scored 2.536 out of 5 on average; the paper identifies the Logical Reasoning metric (2.615) as the primary bottleneck, pointing to failures in constructing closed-loop arguments from new evidence to statutory conclusions.

• Issue identification is the critical ablation: Removing the Disputed Issues Identification agent dropped the average score from 1.971 to 1.909 and reversal accuracy from 61.27% to 60.41% (Qwen3-8B), and also degraded legal application, because retrieval loses directional focus without defined issues.

• Surface metrics mislead: The "No Agents" baseline recorded the highest ROUGE-L (0.331) but a mediocre judge score (1.871) by repeating the first-instance judgment verbatim, whereas SLMAS lowered n-gram overlap while improving Verdict Consistency (1.712 vs. 1.621).

• Results are stable across runs: Three independent runs (N=3) with Qwen3-30B showed SLMAS consistently outperforming the single-agent baseline.

• Data quality validation: A random 5% sample of paired cases was manually expert-reviewed, yielding a consistency rate of approximately 96%. Dataset label distribution is 10.7% reversal and 89.3% affirmation, matching the reported ~10–11% national reversal rate for second-instance cases in China from 2021 to 2023 (exceeding 637,000 cases).

• Human validation of the judge model: Four legal experts scored a random 50-case subset, and the automated DeepSeek-V3.2 scores correlated strongly and significantly with human judgments across all dimensions.

Methodology in Plain English

Data construction. The team collected 4,706,987 raw Chinese judgment documents from China Judgments Online, covering 2021 to 2024 (a window chosen because the Civil Code and the 11th Amendment to the Criminal Law took effect in 2021). Because the repository stores first- and second-instance documents separately with no explicit links, the authors built a multi-stage matching pipeline: regular expressions extract the first-instance case identifier from each second-instance document, then an LLM acts as a semantic validator, comparing litigants, cause of action, and court hierarchy with a binary verification. Documents were de-identified.

Annotation. A three-stage pipeline — structural decomposition into factual and reasoning sections, LLM-driven annotation using DeepSeek-V3.2 with chain-of-thought prompting, and human-in-the-loop verification — produced four labels per pair: the outcome (Affirm or Reverse), the rationale for reversal (erroneous fact-finding, misapplication of law, or procedural violations), newly ascertained facts from the second instance only, and the operative statutory citations.

The model. Rather than mapping input directly to judgment, SLMAS routes the case through four agents in a directed acyclic graph. First, an analysis agent compares the first-instance judgment against new evidence to distill disputed issues. Second, a retrieval agent queries an external legal knowledge base using those issues. Third, a prediction agent acts as a "logical checkpoint," deciding whether to affirm or reverse. Fourth, a writing agent drafts the final judgment conditioned on all three intermediate states. All agents were instantiated with Qwen3 at temperature 0.1.

Evaluation. The test set is 1,000 cases stratified to a 1:1 reversal-to-affirmation ratio (500 each). Evaluation combines reference-based metrics (ROUGE-L, BERTScore) with a reference-free LLM-as-a-Judge using DeepSeek-V3.2 on a 0–5 Likert scale across four rubric dimensions: Verdict Consistency (scored 0 on a mismatch), Fact Consistency, Legal Application, and Logical Reasoning. Baselines spanned general open-source models (gpt-oss-120b, Qwen3 family), a commercial model (Gemini 2.5), legal-specific models (DISC-LawLLM, Wisdom Interrogatory), and inference-matched self-consistency and self-reflection baselines run at N=4 iterations to match SLMAS's number of LLM calls.

Why This Matters

Impact on research. The paper argues that existing legal benchmarks evaluate single-stage, often discriminative tasks and treat legal documents as isolated data points, ignoring procedural dependency between adjudicatory stages. AppellateGen reframes the problem as causal reasoning across trial stages and provides a semantic ground truth that goes beyond lexical overlap — a point the ablation study makes concrete, since the highest-ROUGE baseline was also among the weakest by expert scoring.

Real-world applications (as framed by the paper):

• Assistive drafting tools for appellate judges, who must synthesize an original judgment and new evidence into a binding ruling.
• Case-triage support for courts handling high volumes of appeals, given the reported ~10–11% reversal rate across more than 637,000 Chinese second-instance cases from 2021 to 2023.
• Structured retrieval of operative statutes tied to specific disputed issues, as instantiated by the SLMAS retrieval agent.
• Training and evaluation infrastructure for legal-domain models, replacing surface-overlap benchmarks with rubric-based, dimension-level scoring.

Industry relevance. The finding that general reasoning models beat legal-specialized models has direct implications for legal-tech vendors: investment in general reasoning capability and agentic orchestration may pay off more than domain-specific fine-tuning on static corpora. The paper is explicit that the dataset and method are intended for academic research and as assistive tools only, and must never replace human judges or automate final judicial decisions.

Future Directions

• Extending beyond one jurisdiction. The authors acknowledge AppellateGen is confined to Mainland China and Chinese; cross-jurisdictional and multilingual benchmarks remain open.
• Closing the quality gap. With a best average judge score of 2.536 out of 5 and 67.24% reversal accuracy, the paper calls for paradigms specifically designed for long-chain causal reasoning rather than instruction tuning alone.
• Improving causal dependency modeling. The conclusion frames appellate reasoning over conflicting narratives as an unsolved problem requiring new architectures for robust comparative reasoning.
• Testing alternative agent decompositions. The ablation shows the ordering and presence of the four agents materially affects outcomes; the design space of SOP-style decompositions is left largely unexplored.
• Fairness auditing. The ethics statement notes the data reflects historical judicial decisions and may contain biases by gender, region, or crime type, and recommends fairness evaluations before deployment.

Target Audience

Legal NLP and AI-for-law researchers building benchmarks or generative models for judicial documents; multi-agent and long-context reasoning researchers interested in SOP-style task decomposition; legal-tech practitioners evaluating whether general reasoning models or legal-specific models fit their pipelines; and legal scholars or court technologists assessing the feasibility and limits of AI-assisted appellate drafting.

Authors’ abstract

Legal judgment generation is a critical task in legal intelligence. However, existing research in legal judgment generation has predominantly focused on first-instance trials, relying on static fact-to-verdict mappings while neglecting the dialectical nature of appellate (second-instance) review. To address this, we introduce AppellateGen, a benchmark for second-instance legal judgment generation comprising 7,351 case pairs. The task requires models to draft legally binding judgments by reasoning over the initial verdict and evidentiary updates, thereby modeling the causal dependency between trial stages. We further propose a judicial Standard Operating Procedure (SOP)-based Legal Multi-Agent System (SLMAS) to simulate judicial workflows, which decomposes the generation process into discrete stages of issue identification, retrieval, and drafting. Experimental results indicate that while SLMAS improves logical consistency, the complexity of appellate reasoning remains a substantial challenge for current LLMs. The dataset and code are publicly available at: https://anonymous.4open.science/r/AppellateGen-5763.