How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awards

AI summary

Overview

Research area: Legal natural language processing, specifically regression on continuous monetary outcomes in judicial decisions.

Technical level: Intermediate. The modeling methods (gradient-boosted trees, encoder fine-tuning, prompted decoder LMs, ReAct agents) assume some familiarity with modern NLP, but the paper's framing is accessible to legal scholars and benchmark-oriented researchers.

Scope: The paper introduces ECtHR-NPD, a benchmark of 14,575 European Court of Human Rights judgments for predicting case-level non-pecuniary damage awards in euros when no statutory formula determines the amount.

What This Paper Is About

Most legal NLP benchmarks predict categorical outcomes like whether a violation occurred or which statute applies. Few test whether models can predict how much money a court actually awards, which is a harder problem because awards are discretionary, zero-inflated, and heavy-tailed. This paper builds a benchmark for predicting non-pecuniary damage (NPD) awards under Article 41 of the European Convention on Human Rights, where the Court awards "just satisfaction" on an equitable rather than formulaic basis, and evaluates a broad range of models against it.

Key Contributions

1. A new task formulation. The authors define continuous non-pecuniary damage award prediction as a regression problem, extending legal NLP beyond classification, retrieval, and rule-bounded numerical tasks such as sentencing or tax calculation.

2. The ECtHR-NPD benchmark. They release 14,575 English-language ECtHR judgments with validated case-level euro targets, two input representations (raw text and structured features), chronological train/validation/test splits, and three diagnostic test views (ID, OOD, Challenging).

3. A leakage-controlled protocol. The benchmark explicitly separates target construction from model input: applicants' Article 41 claims, government compensation submissions, the Court's just-satisfaction reasoning, operative provisions, and award tables are withheld from models and used only to build and validate labels.

4. A broad empirical comparison. Six method families are evaluated—constant predictors, gradient-boosted trees, retrieval methods, fine-tuned encoder LMs, prompted decoder LMs, and knowledge-augmented ReAct agents—with diagnostic breakdowns by award range, test view, respondent state, and violated Convention article.

Main Findings

• Simple feature-based models win. CatBoost trained on structured features achieves the lowest test MAE (€9,881), which is 10.2% below the training-set median predictor (€11,006). The best prompted decoder LM (Qwen3.5-9B with chain-of-thought) reaches €10,782 MAE, and larger or agentic configurations do not reliably improve on this.

• Larger models do not help. GPT-5.4, Qwen3.5-Plus, and ReAct agents with knowledge-base access are all beaten by or comparable to gradient-boosted trees, consistent with prior evidence that LMs treat numbers as tokens and struggle with magnitude-sensitive prediction.

• Zero-award recognition is a universal failure. All zero-shot and agentic systems fail to identify cases where the Court awarded nothing; Zero-F1 is near zero for most prompted LMs. Retrieval methods do somewhat better on zeros but this does not translate into good positive-award prediction.

• High awards dominate aggregate error. Awards above €50,000 are only 4.1% of the test pool (118 of 2,897 cases) but drive RMSE and R²; most systems have low or negative R² because of them.

• Challenging cases roughly double the error. On the Challenging view (Grand Chamber judgments plus cases with multiple applicants and multiple concurrent violations), MAE for representative systems is roughly twice the full-test MAE.

• Structured inputs help but do not solve the problem. Serialising structured features compresses inputs from a 95th-percentile of 5,337 words to 237 words and gives modest, inconsistent gains across prompted LMs. Adding more permitted case information shows diminishing returns.

• Award-related information is trivially exploitable. When applicants' claims and Article 41 reasoning are added diagnostically, ReAct agents' MAE roughly halves (e.g., Qwen3.5-Plus from €18,220 to €8,810) and Zero-F1 rises to about 0.99, confirming that the benchmark's difficulty lies in inferring awards from case facts alone.

• Models over-scale multi-applicant cases. Two examples show 90 and 195 applicants with ground-truth awards of €12,500 and €7,500 being predicted at €450,000 and €1,500,000, suggesting models assume a roughly linear per-applicant sum that the Court does not apply.

• Test-period distribution shift is substantial. Zero-award prevalence rises from 25.1% in validation to 32.9% in test, and committee-judgment, multi-applicant, and multi-violation rates all increase by 7.7–9.9 percentage points.

Methodology in Plain English

The researchers collected English ECtHR judgments from the Court's public HUDOC database and segmented each judgment into its structural parts. They then built a strict separation: models see only metadata (respondent state, year, court formation), the "Facts" section, which Convention articles were violated, and external macroeconomic covariates like inflation indicators. Everything that reveals the award amount—applicants' claims, government responses, the Court's reasoning about compensation, and the operative provisions—is hidden and used only to construct the labels.

Targets were extracted from the Article 41 sections, cross-checked against operative provisions and award tables, and validated with deterministic rules covering award-head separation, per-applicant sums, currency normalisation, and recoverability. Zero awards were kept only when supported by an accepted legal rationale. The final dataset was split chronologically (1968–2019 train, 2019–2021 validation, 2021–2026 test) to prevent temporal leakage.

The test pool was further divided into three diagnostic views: an in-distribution subset matching training/validation structure, an out-of-distribution residual, and a "Challenging" overlapping subset of Grand Chamber and multi-applicant, multi-violation cases.

Models were evaluated across six families on MAE as the primary metric, with RMSE, R², median and 95th-percentile absolute error, Pearson and Spearman correlations, and Zero-F1 as secondary diagnostics. Prompted LMs were tested zero-shot, with fixed chain-of-thought developed by a legal expert, and with retrieved few-shot prior cases. ReAct agents had access to a whitelisted legal knowledge base, train-only empirical priors, and eligible reference cases.

Why This Matters

Impact on research. The paper exposes failure modes that existing legal NLP benchmarks do not measure: zero-versus-positive recognition, upper-tail magnitude calibration, and degradation under distribution shift and violation/applicant aggregation. It argues that aggregate accuracy is insufficient for evaluating monetary remedy prediction and that calibration-aware protocols—award-range errors, zero-award recognition, and diagnostic views—should become standard.

Real-world applications:

• Empirical legal analysis. The benchmark provides a structured way to study patterns in how the ECtHR quantifies suffering, humiliation, and other intangible harm across articles, states, and case structures.
• Transparency tools for human rights litigation. Researchers could use the benchmark's diagnostics to surface which case features correlate with award variation, without producing automated advice.
• Comparative court research. The same target-construction methodology could be ported to other discretionary-damages jurisdictions (Inter-American Court, Brazilian consumer aviation cases, US jury verdicts).
• Calibration research for numeric prediction. The benchmark offers a realistic testbed for developing hurdle models, quantile regression, distributional regression, or number-aware loss functions.

Industry relevance. Legal-tech companies building outcome-estimation or settlement-support tools now have a public benchmark that quantifies how far current LMs are from reliable monetary prediction. The authors explicitly warn against using model predictions for legal advice, settlement anchoring, or judicial decision-making, and note that historical awards may encode institutional biases about whose suffering is recognized. Insurance and litigation-finance analysts interested in judicial risk modeling can read the paper as a cautionary assessment of LLM readiness for these tasks.

Future Directions

1. Two-stage and calibration-aware models. Hurdle models that first predict zero versus positive, followed by magnitude prediction for positives, are a natural next step; the paper notes these are difficult to fit under the current input setting because zero-award rationales are withheld.

2. Number-aware training objectives. Standard cross-entropy and token-level objectives do not penalise a €5,000 error differently from a €500,000 error. Distributional regression, quantile heads, and numerically-aware losses are candidates the authors flag.

3. Range-based and band-level evaluation. Because Article 41 is equitable, several awards within a range may be legally defensible; point-error metrics treat all deviations as wrong. Future protocols could score predictions against defensible bands rather than single realized values.

4. Beyond English and beyond case-level targets. The authors flag non-English judgments, pecuniary damages, costs and expenses, per-applicant award prediction, and other courts as unexplored extensions.

Target Audience

Legal NLP researchers and benchmark designers will find the core contribution; empirical legal scholars studying ECtHR remedies and Article 41 practice will benefit from the methodology and error analyses; machine learning researchers working on numeric regression with skewed, zero-inflated targets will find a realistic domain; and legal-tech practitioners should read the ethics and limitations sections before considering any deployment. AI and NLP readers without a legal background can use the paper's Appendix A as a bridge to the Convention system and the structure of ECtHR judgments.

Authors’ abstract

Existing legal benchmarks cover diverse tasks, while continuous monetary remedies remain comparatively underexplored. We introduce ECtHR-NPD, to the best of our knowledge, the first benchmark for predicting non-pecuniary damage (NPD) awards at the European Court of Human Rights (ECtHR) from case information when no statutory formula or explicit calculation rule determines the amount. ECtHR-NPD contains 14,575 cases with case-level awards in nominal euros, chronological splits, and a protocol separating target construction from model input. We evaluate a battery of methods, including constant predictors, gradient-boosted trees, retrieval methods, fine-tuned encoder language models (LMs), prompted decoder LMs, and knowledge-augmented agents. Our results show that more sophisticated LM and agentic approaches do not consistently outperform the strongest feature-based baseline. All model families struggle to identify zero awards and to calibrate high-award predictions, with further degradation on the Challenging test view, making ECtHR-NPD a challenging testbed for current state-of-the-art open-weight and proprietary LMs.