ContractScrub: A benchmark for final review of legal contracts

AI summary

Overview

Research area: Legal AI evaluation; benchmark construction for large language models applied to transactional legal work (contract review).

Technical level: Intermediate. The core method (prompting models to emit structured JSON that is compared against gold annotations) is simple to follow; the domain conventions around contract drafting (defined terms, cross-references) require some legal context.

Scope: ContractScrub is a 3,014-task, 44-contract, 9-category benchmark hand-built by experienced lawyers to measure whether LLMs can perform the final "scrubbing" pass over contracts — finding drafting errors and inconsistencies before signature.

What This Paper Is About

Contract "scrubbing" is the final, tedious pass attorneys make over an agreement to catch residual errors: undefined capitalized terms, broken section cross-references, wrong party names, and contradictory language. The authors argue this is an economically valuable, routine, high-ecological-validity task that seems to suit LLMs (long-context reasoning, consistency checking, named entity recognition), yet no formal evaluation of LLMs on scrubbing existed. Their goal is to build the first benchmark for it and measure how frontier models actually do.

Key Contributions

1. ContractScrub, a new benchmark: 3,014 annotated tasks across 44 contracts drawn from CUAD, with gold annotations covering 9 categories — one defined-term extraction category and eight drafting-error categories.
2. A lawyer-built schema and annotation pipeline: the category schema was designed by licensed attorneys, and annotation plus error insertion was performed by 9 lawyers with commercial, corporate, and contract law experience; all had practiced at least 8 years, 8 had more than 10 years, and 6 had more than 15 years.
3. A recall-focused, multiset evaluation protocol: tuples of (category, fields) are normalized and matched as multisets pooled across all contracts, with macro-average recall over the 9 categories as the primary metric, plus a "word-only" variant that strips location fields to separate identification errors from localization errors.
4. Empirical evidence that a task can look easy and be hard: evaluations of 9 frontier and open-weight models of varying families and sizes, showing performance well below what general-purpose benchmarks would predict.

Main Findings

• Best model is far from perfect: GPT-5.5 achieves the top macro-average recall at 0.750; all F1 scores are below 0.650.
• Ranking and tradeoffs: Gemini 3.1 Pro follows at 0.744 recall, Claude Sonnet 4.6 at 0.686, Gemini 2.5 Pro at 0.632, Claude Opus 4.7 at 0.616, GPT-5.2 at 0.589, Claude Haiku 4.5 at 0.445, Qwen 3.5 (397B) at 0.438, and o4-mini at 0.409. Most models have higher recall than precision; GPT-5.5 has the highest recall but is fourth in precision.
• Scale does not buy legal ability: Qwen 3.5 (397B) reaches only 0.438 recall, comparable to the much smaller o4-mini (0.409) and Claude Haiku 4.5 (0.445), suggesting raw parameter count does not straightforwardly translate to legal document analysis.
• Large spread across error categories: the gap between the easiest category, Defined Terms, and the hardest, Undefined Capitalized Terms, is 0.484. Category means reported: Defined Terms 0.835, Unused Defined Terms 0.781, Terms Defined Multiple Times 0.689, Incorrect Capitalization in Context 0.427, Incorrect Party References 0.362, Undefined Capitalized Terms 0.351.
• Lexical signals beat contextual inference: categories with explicit lexical signals are handled well, while those requiring inference of intent — Undefined Capitalized Terms, Incorrect Capitalization in Context, Incorrect Party References — are much harder. Even GPT-5.5 reaches only 0.514 on Undefined Capitalized Terms and 0.562 on Incorrect Party References.
• Reasoning helps moderately, at a cost: turning reasoning off drops GPT-5.5 from 0.750 to 0.643 recall and Claude Opus 4.7 from 0.616 to 0.526, with the gains concentrated in categories requiring full-document term consistency rather than deeper legal interpretation. Gemma 4 (26B) without reasoning scores 0.365 recall.
• Long-distance references are a failure mode: the authors manually annotated the character distance between references and referred sections for half of the "Incorrect Section References" gold items. Recall generally decreases as distance grows, with the effect most pronounced past ten thousand characters (roughly 5–6 pages).
• Subtask scores are weakly correlated: Pearson correlations between category scores are low but positive, with a weak cluster around incorrect usage of definitions; overall the subtasks appear to measure mostly independent capabilities.
• Cheap relative to humans: the most expensive model, GPT-5.5, costs only $1.38 per contract and takes 533 seconds; Gemini 3.1 Pro costs $0.19 and returns in 76 seconds.
• Worse than related benchmarks suggest: on FaithEval, GPT-4 achieved 89.4% and Claude Sonnet 3.5 92.2% on inconsistency detection, while comparable newer models reach only 75.0% (GPT-5.5) and 68.6% (Claude Sonnet 4.6) on inconsistency detection in ContractScrub. Claude 3 achieved 98.28% at 128K tokens on long-context retrieval, and Gemini 1.5 reports near-perfect recall on needle-in-a-haystack probes — all far above ContractScrub results.
• Category counts (Table 4): Defined Terms 1,505 total (34.2 per contract); Undefined Capitalized Terms 689 (15.7); Uncapitalized Defined Terms 317 (7.2); Unused Defined Terms 202 (4.6); Incorrect Section, Article, or Paragraph References 150 (3.4); Incorrect Party References 130 (3.0); Incorrectly Capitalized Terms In Context 129 (2.9); Terms Defined Multiple Times 97 (2.2); Inconsistent Language 87 (2.0).

Methodology in Plain English

The authors selected contracts from the open-source CUAD dataset (itself drawn from EDGAR, the SEC filing repository). Lawyers first screened candidates for fundamental drafting flaws, suitable length (approximately 10–15 pages), and coherence, removing things like empty exhibits. Each surviving contract was then read and annotated for any pre-existing errors across the nine categories, including recording every defined term. In a third stage, the same lawyers deliberately inserted additional realistic drafting errors across the categories, aiming for broad and approximately balanced coverage while keeping the contract coherent and legally plausible. Every occurrence was logged separately — repeats in the same section were not de-duplicated. Two lawyers then performed targeted quality reviews of contracts, gold answers, and prompts. Notably, the benchmark does not use CUAD's own labels.

At evaluation time, each model is given an instruction prompt describing the nine categories and the required output format, plus the contract, and must return a single JSON object with one key per category. Because the authors note that scrubbing tasks have competing demands, models are prompted to identify issues from each category in separate instances. The output is parsed into category-specific tuples — for example (term, location) for defined terms and (wrong reference, correct reference, location) for incorrect references — and compared deterministically against gold annotations as multisets. Normalization is applied to both sides before scoring: terms are lowercased and whitespace-stripped; location fields are canonicalized (e.g., 1(a)(i) becomes 1ai; 1.1(h)(vii) becomes 1.1hvii), with special labels such as P/Preamble, R/Recitals, Exhibit X, Schedule X, and Signature Block also canonicalized and distinct section levels kept distinct; and paired-location categories such as Conflicting Definitions and Inconsistent Terms are matched order-independently.

Recall is the primary metric because the cost of a missed issue (false negative) is considered much higher than a false positive — it is easier to check whether a flagged issue is real than to find one you did not know about — and because CUAD contracts may contain remaining unknown pre-existing errors. Performance is reported as the macro-average across the 9 categories, with precision and F1 also reported, plus the word-only variant that removes location fields to separate identification from localization. Models tested include GPT-5.5, GPT-5.2, o4-mini, Claude Opus 4.7, Claude Sonnet 4.6, Claude Haiku 4.5, Qwen 3.5-397B, Gemini 3.1 Pro, Gemini 2.5 Pro, and Gemma-4-26B (the paper describes these as 9 frontier and open-weight models spanning families and capability tiers).

Why This Matters

The paper argues there is a measurable gap between how hard a task seems to humans and how hard it is for LLMs, and that isolated capability benchmarks (long context, NER, inconsistency detection) do not predict performance when those capabilities must be applied jointly to a real professional workflow. It also highlights a category of task that is under-represented in current benchmarks: correctness defined by conventions internal to a document rather than by external legal knowledge. Because scrubbing is described as routine but load-bearing — small errors propagate upward and can undermine negotiations, outcomes, and client relationships — the benchmark gives a concrete, ecologically valid target for measuring economically meaningful progress.

Real-world applications:

• Pre-signature quality control: adding automated scrubbing as a cheaper, faster first pass before lawyers review, so human attention goes to issues requiring legal judgment.
• More frequent scrubbing during negotiation: models are fast and cheap enough (as low as $0.19 per contract) that scrubbing could run repeatedly mid-deal rather than only at closing, reducing last-minute surprises after a deal is otherwise settled.
• Support for junior attorneys and paralegals: the paper frames automation as a way to improve first-pass review and support less experienced staff on a repetitive, fatigue-prone task.
• Issue tracking infrastructure: the structured JSON requirement reflects production needs, where downstream parsing and issue tracking depend on machine-readable output.

Industry relevance: the authors are affiliated with Thomson Reuters Foundational Research and Imperial College London, and frame the work around transactional law practice — a domain the paper notes ranks highly among professions expected to see AI-driven economic impact. The release of the dataset on Hugging Face (https://huggingface.co/tri-fair-lab/contract_scrub, licensed CC BY-NC-SA 4.0) supports direct commercial and academic use.

Future Directions

• Scaling the corpus: the benchmark covers only 44 contracts. The authors state a larger corpus would reduce the effect of idiosyncrasies in any single contract and better represent the universe of corporate contracts.
• Beyond English and the closing-stage setting: the data is English-only and approximately 10–15 pages. The authors flag limited generalizability to other legal traditions, languages, and deal types with different document lengths, such as short term sheets or multi-hundred-page complex financings.
• Removing or probing the structured-output constraint: the evaluation requires structured JSON, which the authors acknowledge may depress raw scores and introduces instruction-following as a confounder, even though they argue it is a genuine deployment requirement.
• Reaching the automation bar: the paper states full automation would likely require performance higher than the 75% recall currently attained, leaving open the question of how much targeted improvement — for example on long-distance references and intent-dependent categories — is needed before deployment alongside or in place of experts.

Target Audience

Researchers and engineers building or evaluating LLMs for legal and professional-domain applications; benchmark designers interested in ecological validity and in tasks defined by document-internal conventions; legal technology practitioners and in-house or law-firm teams considering where to insert LLM assistance into contract workflows; and evaluation scientists studying the gap between isolated capability benchmarks and joint, real-world task performance.

Authors’ abstract

Legal work, with its heavy reliance on processing large amounts of text, is often considered one of the domains most exposed to the use of LLMs. Contract ``scrubbing,'' the final review of transactional agreements for errors and inconsistencies, is a particularly suitable task for automation, because it is routine, painstaking work requiring detailed attention to long documents. Scrubbing also seems to align naturally with the general capabilities expected of frontier LLMs around long-context reasoning, consistency checking, and named entity recognition (NER). Despite the economic value and potential for automation, no formal evaluations of LLMs performing contract scrubbing have been conducted. We introduce ContractScrub, the first benchmark designed to evaluate contract scrubbing capabilities, comprising contracts hand-crafted by experienced lawyers over diverse error categories such as misuse of defined terms, incorrect references, and inconsistent language. Frontier models perform surprisingly poorly with only one model reaching 0.75 macro average recall despite strong performance on seemingly related general benchmarks, demonstrating the practical limits of current models and the importance of narrowly targeted, domain-specific benchmarks for measuring real-world impact.