CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval

AI summary

Overview

Research area: Natural Language Processing, specifically automated fact-checking (AFC) applied to the legal domain, legal precedent retrieval, and benchmark construction for U.S. Supreme Court case law.

Technical level: Advanced. The paper assumes familiarity with retrieval-augmented systems, embedding-based dense retrieval, natural language inference, LLM-as-a-judge pipelines, and standard IR metrics such as Recall@k.

Scope: The paper introduces CaseFacts, a 6,294-claim benchmark for verifying colloquial legal claims against 3,299 U.S. Supreme Court cases, and evaluates LLM prompting baselines and fine-tuned embedding retrievers on it.

What This Paper Is About

Automated fact-checking systems have mostly verified general-knowledge claims against static corpora like Wikipedia, but they ignore domains like law where truth changes over time and the supporting evidence is written in technical language. CaseFacts asks systems to map everyday phrasing ("The Supreme Court banned school prayer") to formal Supreme Court holdings, decide whether a claim is Supported, Refuted, or relies on a precedent that has been Overruled, and retrieve the specific cases that justify the verdict. The goal is a benchmark that tests retrieval across the semantic gap between lay speech and jurisprudence, reasoning against plausible-but-wrong claims, and temporal awareness that case law is not static.

Key Contributions

1. A new benchmark, CaseFacts, described as the first large-scale benchmark for verifying colloquial legal claims against SCOTUS precedents, with three distinct verdict classes: Supported, Refuted, and Overruled.

2. A synthetic data generation pipeline that resolves intra-case and inter-case inconsistencies, using semantic similarity percentile heuristics to locate legal overrulings and confirmations in a corpus too large for exhaustive LLM comparison.

3. An explicit model of precedent validity through the Overruled class, in which a claim is labeled Overruled when it relies on a holding from an older case that a later reversing case has explicitly overturned, validated against Oyez disposition metadata.

4. Comprehensive benchmarks of state-of-the-art LLMs and retrieval models, showing that unrestricted web search degrades performance relative to a closed-book baseline, and that fine-tuning embedding models on the CaseFacts training set substantially improves retrieval recall.

Main Findings

• Unrestricted web search hurt performance. A search-enabled GPT-4o scored an Evidence Score of 0.212, Verdict Accuracy of 0.574, and Verdict Score of 0.169, while the naive closed-book setting scored 0.232, 0.600, and 0.185 respectively. The authors attribute this to search surfacing cases outside the 3,299-case list or returning noisy, tangentially relevant cases.

• Evidence retrieval, not verdict prediction, is the bottleneck. Evidence scores were much lower than verdict accuracies in both baselines, meaning models pick the right verdict more often than they cite the correct controlling cases.

• Standard NLI failed on legal overrulings. A textattack/bert-base-uncased-MNLI model performed "very poorly" at the deeper legal entailment needed to identify overrulings and confirmations, and its predictions were unusable.

• Inconsistencies cluster at high semantic similarity. Contradictions clustered roughly at or above 0.70 similarity and overlaps at or above 0.80. The chosen 95th-percentile filtering windows were 0.70–0.95 for contradictions and 0.80–0.97 for overlaps.

• Fine-tuning improves retrieval across the board. Supervised fine-tuning with MultipleNegativesRankingLoss improved Recall@1, Recall@5, and Recall@10 for every encoder family and epoch count tested. Qwen3-Embed-0.6B showed the largest single-step Recall@1 gain, +24.1 points after three epochs (0.2952 to 0.5364). Baseline retrievers: BM25 reached Recall@1 0.1160, Recall@5 0.2180, Recall@10 0.2500; ColBERT reached 0.2560, 0.4620, and 0.5420.

• The Overruled class is the hardest to validate. Human annotators passed only 44 of 59 Overruled claims (74%), compared with 353 of 384 Supported (91%) and 177 of 195 Refuted (90%).

• LLM judges aligned closely with human annotations on the human-verified test set: 97.4% accuracy for factuality (487 correct, 13 incorrect), 95.5% for contradiction (238,148 correct, 11,352 incorrect), 97.4% for overlap (243,113 correct, 6,387 incorrect), and 96.5% overall (481,748 correct, 17,752 incorrect).

• Entity leakage was minimal. A roberta-large-ner-english check found about 25% of claims mentioned an entity, but an LLM judge found only 154 claims (2.5%) contained a relevant entity. Performance on that subset was close to the full test set (Naive Evidence Score 0.232 versus 0.222; Verdict Score 0.222 versus 0.185).

Methodology in Plain English

The authors start from Oyez, a public archive of U.S. Supreme Court material, and keep only cases that have all three expert-written summary fields: facts, question, and conclusion. This yields 3,299 cases as of July 2025, and these summaries serve as the ground-truth evidence for every claim.

Claim generation uses Qwen3-Next-80B-A3B-Thinking, prompted to write in simple everyday language, to avoid court-report phrasing, and to exclude party names or dates so the task is about legal principles rather than case trivia. This produced an initial pool of 10,471 unique claims. Validation uses a different model family, gemma-3-27b-it, to reduce self-preference bias in judging. All LLM calls used temperature 0.6 and top-p 0.9.

Within a single case, an LLM judge checks that each claim is entailed by that case's evidence under a closed-world assumption, which removed 1,484 claims and left 8,987. A second pass detects internal contradictions and near-duplicate overlaps, then removes one claim from each problematic pair, removing a further 1,404 claims and leaving 7,583.

Across cases, checking all pairs would exceed 25 million comparisons, so the authors compute cosine similarity between every claim pair with Qwen3-Embedding-8B and sample 1,000 pairs from each 0.1-wide similarity bin between 0.2 and 0.9, including all pairs in the 0.9–1.0 bin and excluding the 0.0–0.2 bin. LLM judges labeled the samples, and percentile statistics defined the windows where inconsistencies concentrate. Restricting judging to those windows left 199,203 contradiction candidate pairs and 15,874 overlap candidate pairs. From the contradiction pairs, 1,946 actual contradictions (0.98% of the sampled range) were resolved using ruling dates plus Oyez disposition metadata to establish the Overruled class. From the overlap pairs, 12.31% (1,955 samples) were resolved by merging the ground-truth cases of overlapping claims.

Refuted claims are negations of valid Supported claims, produced in two steps: generate a plausible but false negation grounded in the case facts, then rewrite it as a concise, general legal principle. This shortened average length from 212 characters to 108 characters so that false claims could not be spotted by length or hedged qualifiers alone.

The test set was built by two human annotators who reviewed 638 claims, validated 574 as high quality, and from those a final 500 were sampled. Annotators overlapped on 100 claims, giving a Cohen's Kappa of 0.637 (substantial agreement), and an attorney independently reviewed 50 claims and found no issues.

Evaluation uses tiered metrics. The Evidence Score is the recall between predicted and gold case sets, but it is penalized to zero if the system fails to retrieve at least half the gold supporting cases within the top five results (Recall@5 below 0.5). Verdict Accuracy is whether the predicted verdict matches the gold label. The primary ranking metric is the Verdict Score, the product of Evidence Score and Verdict Accuracy, which penalizes guesses that are right without correct evidence.

Why This Matters

Impact on research. CaseFacts reframes legal NLP evaluation away from formal-to-formal tasks such as LegalBench and CaseHOLD toward informal-claim-to-formal-ruling verification. It also treats legal truth as time-dependent, showing that a verification system must know when precedent has been overruled, not just whether a statement matches a document. Its negative result on web search is a caution for retrieval-augmented generation in high-stakes domains.

Real-world applications.

• Public-facing legal fact-checking of viral claims about Supreme Court rulings on social media and news.
• Legal research assistance that retrieves the controlling precedent behind a plain-language question.
• Journalism and civic education tools that flag when a claim rests on a precedent that has since been overturned.
• Training and evaluation of domain-specific retrieval and ranking systems for legal document collections.

Industry relevance. Legal tech, compliance, and AI-assisted research vendors can use the benchmark to test whether their retrieval and verdict layers are actually grounded in authoritative sources. The finding that fine-tuned embedding models outperform prompting baselines on retrieval suggests a practical architecture: a specialized retriever paired with an LLM verifier.

Future Directions

1. Move beyond expert summaries to full judicial opinions. The authors note that Oyez summaries abstract away long-context reasoning in raw legal texts, which may simplify retrieval relative to a full-text corpus.

2. Improve overruling detection beyond similarity thresholds. The heuristic misses cases with low semantic similarity. A better method could catch overturns that do not paraphrase closely.

3. Broaden legal coverage. CaseFacts covers only U.S. Supreme Court rulings, excluding statutory law, regulatory codes, and state-level jurisprudence. It also cannot label a ruling as overruled when Congress supersedes it by legislation, and it does not consider precedent in other countries.

4. Constrain retrieval augmentation. Since unrestricted web search degraded performance, one open question is how to build retrieval that stays within the validated case set while still adding useful external signal.

Target Audience

Researchers in NLP and automated fact-checking, information retrieval specialists working on domain-specific and temporal retrieval, and legal AI practitioners building verification or precedent-retrieval systems. The benchmark and pipeline are most useful to those who can act on the evaluation protocol, the synthetic data generation design, and the negative result about web search in high-stakes verification.

Authors’ abstract

Automated Fact-Checking has largely focused on verifying general knowledge against static corpora, overlooking high-stakes domains like law where truth is evolving and technically complex. We introduce CaseFacts, a benchmark for verifying colloquial legal claims against U.S. Supreme Court precedents. Unlike existing resources that map formal texts to formal texts, CaseFacts challenges systems to bridge the semantic gap between layperson assertions and technical jurisprudence while accounting for temporal validity. The dataset consists of 6,294 claims categorized as Supported, Refuted, or Overruled. We construct this benchmark using a multi-stage pipeline that leverages Large Language Models (LLMs) to synthesize claims from expert case summaries, employing a novel semantic similarity heuristic to efficiently identify and verify complex legal overrulings. Experiments with state-of-the-art LLMs reveal that the task remains challenging; notably, augmenting models with unrestricted web search degrades performance compared to closed-book baselines due to the retrieval of noisy, non-authoritative precedents. We release CaseFacts to spur research into legal fact verification systems.