Segmentation and Processing of German Court Decisions from Open Legal Data

AI summary

Overview

Research area: Legal Natural Language Processing, specifically corpus construction and text segmentation for German court decisions.

Technical level: Intermediate. The work combines rule-based HTML parsing and regular-expression header detection with a statistically grounded manual validation protocol; readers should be comfortable with concepts such as confidence intervals, JSONL data formats, and retrieval pipelines.

Scope: The paper describes the construction, extraction pipeline, verification, and public release of a 251,038-document German court decision corpus segmented into Tenor, Tatbestand, and Entscheidungsgründe.

What This Paper Is About

The Open Legal Data collection offers a large set of German court decisions with clean metadata, but the decision texts themselves are stored as inconsistently formatted HTML with few or no clearly marked sections. This matters because German decisions follow a conventional structure (operative part, facts, and reasoning), and downstream tasks like retrieval, citation analysis, and rhetorical role classification depend on knowing where one section ends and the next begins. The authors build a cleaned, section-segmented version of that corpus and verify the segmentation quality through manual review of a statistically representative sample.

Key Contributions

1. A segmented corpus of 251,038 German court decisions, derived from the official Open Legal Data dump (as of 2022-10-18), with the texts separated into Tenor, Tatbestand, and Entscheidungsgründe.
2. A rule-based extraction pipeline that parses heterogeneous HTML, normalizes court metadata via Open Legal Data APIs (/api/states/ and /api/cities/), detects section headers with exact line-level regular-expression patterns, and handles the differing drafting conventions of Urteile and Beschlüsse.
3. A statistically justified verification protocol using Cochran's formula with finite population correction (95% confidence level, 5% margin of error), yielding a 384-case random sample that was manually reviewed.
4. A publicly released, ready-to-use resource in JSONL format containing normalized metadata, sectioned text, and extracted statutory and case references, available via Hugging Face datasets.

Main Findings

• Extraction accuracy: Manual review of the 384 sampled decisions confirmed correct segmentation in 97.40% of cases (±1.59%). The resulting 95% confidence interval with finite population correction is (0.9581, 0.9899) for the full dataset, meaning the true proportion of correctly segmented decisions lies between 95.8% and 98.9%.
• Sample size calculation: Cochran's formula gave an initial sample size of approximately 384.16, which after finite population correction for N = 251,038 came to approximately 383.58, rounded conservatively to 384.
• Section coverage: Tenor appears in 220,273 (87.7%) of decisions, Tatbestand in 164,222 (65.4%), and Entscheidungsgründe in 238,666 (95.1%).
• Structural composition: 144,383 (57.5%) decisions contain all three sections, 63,720 (25.4%) contain only Tenor and Entscheidungsgründe, and 11,388 (4.5%) contain only the Tenor.
• Missing content: 176 decisions (0.07%) contain no sections at all, corresponding to cases where the original content field is blank.
• Appeal notices: The Rechtsmittelbelehrung was extracted as a separate field and appears in 8,335 decisions (3.32% of the corpus).
• Source of errors: Segmentation failures were mainly due to rare formatting irregularities in the HTML.
• Drafting-practice explanation: The lower coverage of Tatbestand relative to Entscheidungsgründe reflects the difference between Urteile (which usually state all three sections explicitly) and Beschlüsse (where factual background is sometimes merged into the reasoning when there is no subdivision into Gründe I and Gründe II).

Methodology in Plain English

The authors started from the official Open Legal Data dump, in which each decision's text sits inside a content HTML attribute with varying structure. They parsed that HTML and iterated over visible elements, limiting themselves to p, h1–h4, td, and a custom rd tag, then collapsed whitespace and dropped empty or duplicate lines.

Court metadata was normalized by resolving city and state identifiers through the Open Legal Data public APIs, with missing entries set to "Unspecified". Section boundaries were found by looking for exact, full-line headers from a fixed vocabulary — tenor, tatbestand, entscheidungsgründe, and gründe — each matched in two forms: a compact form and a spaced-letter form (for example, tenor versus t e n o r), applied case-insensitively. Until the first header appears, text is assigned to the Tenor, consistent with German drafting practice where decisions begin with the operative part. A gründe heading is split by Roman numeral subdivisions so that Gründe I maps to Tatbestand and Gründe II to Entscheidungsgründe; when no subdivision exists, the whole Gründe block is treated as Entscheidungsgründe. The Rechtsmittelbelehrung is stored in a separate field because it is a procedural instruction rather than part of the decision's reasoning. Finally, the widely used Legal Reference Extraction tool identifies and categorizes citations by type (law or case), and each decision is written out as a single JSON object with metadata, sectioned text, and references.

To check reliability, the authors determined how many decisions needed manual inspection rather than guessing. Using Cochran's formula with p set to 0.5 (the most conservative choice), Z = 1.96, and e = 0.05, they drew 384 cases uniformly at random. Reviewers checked that Tenor, Tatbestand, and Entscheidungsgründe were correctly identified, defining correctness strictly as the absence of overlap between sections.

Why This Matters

Structured legal text is a prerequisite for computational work on the German legal system, and the widely used source corpora have not offered consistent section boundaries. This dataset supplies those boundaries at scale and releases them in a format usable without further preprocessing, which lowers the barrier for both retrieval research and model training on German legal text.

Real-world applications include:

• Legal case retrieval, where section-aware indexing prevents factual history from being confused with judicial reasoning and producing false matches.
• Citation analysis, where the significance of a cited statute depends on whether it appears in the operative part or in the reasoning.
• Retrieval-Augmented Generation systems, where section-aware chunking improves interpretability and stops models from blending argumentative and operative content.
• Summarization and rhetorical role classification, both of which require knowing where sections begin and end before a model can be trained or evaluated.

Industry relevance: Legal tech platforms, courts, and compliance teams that need machine-readable decisions can build search, citation-linking, and question-answering tools directly on this corpus. The authors also report that they are currently using the dataset to build a RAG system for German legal texts, indexing case summaries, statutory references, and reasoning paragraphs separately.

Future Directions

• Improving retrieval quality through ranking and reranking strategies in the RAG pipeline the authors are building on top of this corpus.
• Evaluating the RAG system across specific subtasks, named as statute retrieval (aligning reasoning passages with cited provisions), reasoning coverage (distinguishing factual context from arguments), and interpretability (surfacing only the relevant section of a decision to the user).
• Extending the corpus with additional court decisions as they become available.
• Refining the handling of legal drafting variations, since the errors observed in manual review stemmed mainly from rare HTML formatting irregularities and the differing conventions of Urteile versus Beschlüsse.

Target Audience

Researchers and practitioners in legal NLP, particularly those working with German-language corpora, information retrieval, citation analysis, or retrieval-augmented generation. The paper also suits dataset builders who want a worked example of combining rule-based extraction with formal statistical sampling for quality assurance, and legal tech developers who need a structured, ready-to-use source of German court decisions.

Authors’ abstract

The availability of structured legal data is important for advancing Natural Language Processing (NLP) techniques for the German legal system. One of the most widely used datasets, Open Legal Data, provides a large-scale collection of German court decisions. While the metadata in this raw dataset is consistently structured, the decision texts themselves are inconsistently formatted and often lack clearly marked sections. Reliable separation of these sections is important not only for rhetorical role classification but also for downstream tasks such as retrieval and citation analysis. In this work, we introduce a cleaned and sectioned dataset of 251,038 German court decisions derived from the official Open Legal Data dataset. We systematically separated three important sections in German court decisions, namely Tenor (operative part of the decision), Tatbestand (facts of the case), and Entscheidungsgründe (judicial reasoning), which are often inconsistently represented in the original dataset. To ensure the reliability of our extraction process, we used Cochran's formula with a 95% confidence level and a 5% margin of error to draw a statistically representative random sample of 384 cases, and manually verified that all three sections were correctly identified. We also extracted the Rechtsmittelbelehrung (appeal notice) as a separate field, since it is a procedural instruction and not part of the decision itself. The resulting corpus is publicly available in the JSONL format, making it an accessible resource for further research on the German legal system.