When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection

AI summary

Overview

Research area: Natural Language Processing applied to law — specifically claim-level hallucination detection and verification of whether legal claims are actually supported by the authorities they cite.

Technical level: Intermediate. Readers should be familiar with large language models, natural language inference (NLI) label taxonomies (entailment / contradiction / neutral), and Retrieval-Augmented Generation (RAG). No advanced mathematics is required.

Scope: The paper introduces PARCEL (Parenthetic Al Reasoning and Claim EvaLuation), a 3,396-claim benchmark built from recent New York State Court of Appeals decisions and briefs, and uses it to evaluate seven state-of-the-art LLMs on three-way verification of legal parenthetical claims.

Publication details: arXiv:2610.10971v1 [cs.CL], 07 Oct 2026; DOI 10.1145/3836937.3836948; presented at the 21st International Conference on Artificial Intelligence and Law, June 8–12, 2026, Singapore. Authored by M. Mikail Demir and M. Abdullah Canbaz (University at Albany, SUNY). Licensed CC BY 4.0.

What This Paper Is About

Large language models are increasingly used for legal research and drafting, but they can produce confident-sounding claims that the cited authority does not actually support. Existing legal NLP benchmarks mostly test broad document understanding or general legal reasoning, not the granular, citation-driven check practitioners actually perform: does this specific source support this specific proposition? PARCEL addresses that gap by turning the question into a three-way NLI task over claims written in the style of explanatory parentheticals, including adversarially constructed claims that are stylistically plausible but textually unsupported.

Key Contributions

1. A new adversarial benchmark (PARCEL). 3,396 parenthetical-style claims labeled Supported, Refuted, or Not Found, grounded in New York State Court of Appeals decisions issued between 2024 and 2025 and in parentheticals harvested from real appellate briefs. The dataset is released at https://github.com/mmikaildemir/PARCEL.
2. A systematic, reproducible claim-generation pipeline. Seed extraction from real briefs, few-shot synthetic generation of supported claims using a five-type usage rubric, and controlled generation of refuted and not-found perturbations conditioned on the full opinion text.
3. A safety-oriented evaluation protocol. Instead of relying on accuracy alone, the paper defines and reports the False Entailment Rate (FER), decomposed into FER on Refuted claims and FER on Not Found claims, plus strategy-level breakdowns and a "hard claim" analysis over multi-model failures.
4. An error diagnosis identifying knowledge leakage. The paper documents cases where models validate claims that are legally accurate in the abstract but absent from the provided decision, showing pre-trained legal knowledge overriding strict textual grounding.

Main Findings

• High headline accuracy, uneven by class: gemini-3-flash-preview reaches the highest overall accuracy (0.97) and macro F1 (0.97), followed by kimi-k2-thinking (0.95) and glm-4.7 (0.94). All models except llama-3.3-70b-instruct exceed 90% accuracy.
• Absence detection is the bottleneck: The largest gap appears in the Not Found class. llama-3.3-70b-instruct attains Not Found F1 = 0.61 with recall = 0.45, while gemini-3-flash-preview reaches Not Found F1 = 0.97. Missing support is harder to detect than direct contradiction across models.
• Aggregate confusion matrix confirms the direction of failure: 15.4% of Not Found claims are misclassified as Supported, compared with only 4.3% of Refuted claims misclassified as Supported. Ground-truth Supported claims are predicted Supported 97.8% of the time; Refuted 94.3%; Not Found 79.2%.
• False Entailment Rate separates models that accuracy does not: gemini-3-flash-preview has the lowest overall FER (3.47%). glm-4.7 has FER on Refuted claims of 1.18% but FER on Not Found claims of 9.55%. llama-3.3-70b-instruct reaches 0.81 overall accuracy yet has FER on Not Found claims of 48.76%, meaning nearly half of Not Found claims are incorrectly validated as Supported. Although kimi-k2-thinking slightly outperforms glm-4.7 in overall accuracy, glm-4.7 attains the lower overall FER (3.97% vs. 4.36%).
• Supported recall is near-saturated: Supported-class recall ranges from 0.89 to 1.00 across models, which the authors note may also reflect an over-acceptance bias rather than genuine verification skill.
• Contradiction strategies are comparatively easy: Average accuracy across refuted strategies ranges from 0.86 for Authority Swap to 0.97 for Scope Contradiction, with Logic Reversal at 0.96 and Distractor Swap at 0.91. glm-4.7 achieves 1.00 on Scope Contradiction. Authority Swap is consistently hardest (both gemini-2.5-flash and gpt-oss-120b drop to 0.77).
• Fabricated authority is the hardest failure mode: Average accuracy across not-found strategies is 0.72 for Phantom Citation, 0.81 for Missing Detail, and 0.80 for Unaddressed Argument. llama-3.3-70b-instruct achieves only 0.24 on Phantom Citation; gemini-3-flash-preview stays between 0.93 and 0.96 across all three.
• Most errors are model-specific, not universal: 290 claims (8.54%) are misclassified by at least three of seven models; 158 (4.65%) by at least four; 75 (2.21%) by at least five; 27 (0.80%) by at least six; and only 8 claims (0.24%) defeat all seven models.
• Knowledge leakage in the hardest cases: In manual examination of the 27 claims failed by at least six models, the recurring pattern was models validating claims that are legally accurate in the abstract but unsupported by the specific decision — for example, labeling as Supported a not-found claim describing an appeal as involving "leaving the scene of a personal injury accident" when the opinion only references a Vehicle and Traffic Law § 600 (2) (a) violation without using that language.

Methodology in Plain English

The researchers chose a narrowly defined legal writing unit — the explanatory parenthetical, the short phrase attached to a citation that explains why an authority matters. They pulled 85 publicly available briefs filed before the New York State Court of Appeals via the CourtPASS system and used the google/gemini-3-flash-preview model to extract parentheticals together with their cited cases, yielding over 700 candidate examples. They discarded direct quotations and surface-level factual summaries, used an LLM-as-a-judge scoring of reasoning density, and kept 180 high-quality claims to serve as few-shot exemplars.

To build the larger corpus they retrieved Court of Appeals decisions from 2024–2025, filtered out decisions shorter than 2,000 characters, excluded decisions that triggered LLM safety filters, and processed 85 decisions. Supported claims were generated with up to ten per decision according to a five-category usage rubric — Holding rule, Reasoning rationale, Facts procedure, Treatment relational, and Descriptive label — with each type appearing at least twice, producing 849 supported claims.

Refuted claims were then created by perturbing those supported claims while conditioning on the full opinion text, using four strategies: Distractor Entity Swap, Logic Reversal, Scope Contradiction, and Authority Swap. Not Found claims used three strategies: Phantom Citation, Unaddressed Argument, and Missing Detail. The final dataset contains 3,396 claims in a controlled 1:2:1 class balance: 849 Supported, 1,698 Refuted, and 849 Not Found. Within Refuted, Logic Reversal accounts for 789 claims (46.6%), Distractor Swap 363 (21.5%), Authority Swap 256 (15.1%), and Scope Contradiction 284 (16.8%). Within Not Found, Phantom Citation accounts for 330 (38.9%), Unaddressed Argument 272 (32.0%), and Missing Detail 247 (29.1%).

Verification was posed as a zero-shot NLI problem: each model receives the full judicial decision as premise and one parenthetical claim as hypothesis, with a minimalist system prompt defining the task and label space. Seven models were evaluated — gemini-3-flash-preview, kimi-k2-thinking, glm-4.7, qwen3-235b-a22b-instruct-2507, gemini-2.5-flash, gpt-oss-120b, and llama-3.3-70b-instruct — using overall accuracy, macro F1, class-wise precision/recall/F1, FER, strategy-level accuracy, and aggregated confusion matrices.

Why This Matters

Impact on research. The paper argues that aggregate accuracy overstates trustworthiness in legal settings and that evaluation should prioritize groundedness and safety. It separates "is this claim legally plausible" from "is this claim supported by the cited authority," and shows that pre-trained legal knowledge can interfere with closed-book verification — an inverse contamination effect rather than a benchmark-contamination benefit. It also connects to prior work reporting that models reach only 77% accuracy on Bluebook citation tasks even with in-context learning of the rules (Dahl, 2025).

Real-world applications.

• Reviewing AI-assisted legal briefs for parentheticals that cite real authorities but misstate what those authorities hold.
• Auditing Retrieval-Augmented Generation pipelines in legal research tools, where unsupported inferences can appear grounded.
• Automated citation-checking and verifier cascades inside law-firm drafting workflows.
• Secondary checking or ensemble review in jurisdictions and courts assessing AI-generated filings.

Industry relevance. Legal technology vendors building RAG-based research and drafting assistants need to know whether their systems fabricate or over-accept support, not merely whether they produce fluent text. The finding that most hard claims are caught by at least some models — with only 8 of 3,396 claims (0.24%) defeating all seven — suggests ensemble or majority-voting verification is a practical near-term mitigation before architectural fixes arrive.

Future Directions

• Uncertainty-aware evaluation, so systems signal low confidence rather than confidently asserting support where evidence is absent.
• Stronger citation verification, targeting the Phantom Citation category that produced the largest performance drops (average accuracy 0.72, and 0.24 for llama-3.3-70b-instruct).
• Verifier- or ensemble-based mitigation strategies, such as majority voting or verifier cascades, exploiting the rapid decline in universally failed claims.
• Resolving the parametric-versus-contextual knowledge conflict, since long-context access alone did not stop models from validating claims that are abstractly true but textually unsupported.

Target Audience

Legal NLP researchers building evaluation benchmarks; AI and law scholars studying hallucination and grounding; practitioners and technologists building RAG-based legal research or drafting tools; and evaluation engineers who need safety-oriented metrics such as False Entailment Rate alongside accuracy. Readers primarily interested in general-purpose LLM benchmarking without a legal or verification focus will find the paper's dataset construction and error analysis most relevant.

Authors’ abstract

Large language models are increasingly used in legal research and drafting, but they can still produce claims that sound convincing without being supported by the cited source. We introduce PARCEL, a benchmark for checking whether a legal claim is supported by the underlying authority. Using recent New York State Court of Appeals decisions, we build a dataset of 3,396 parenthetical-style claims labeled as Supported, Refuted, or Not Found. We cast this task as a three-way natural language inference problem and evaluate several state-of-the-art LLMs in a zero-shot setting. Although the strongest models reach up to 0.97 accuracy, the results also show an important weakness: models still incorrectly mark unsupported claims as supported, even when the full opinion text is provided. Across models, missing support is harder to detect than direct contradiction, and fabricated but plausible citations cause the largest drop in performance. Overall, PARCEL provides a practical benchmark for testing claim-level groundedness in legal RAG systems.