Hallucination, Uncertainty, Verification, and Abstention

A brief filed in federal court cited six judicial opinions — Varghese, Shaboon, Petersen, Martinez, Durden and Miller — with quotations and citations attached to each. None of the six existed. ChatGPT had produced them. On 22 June 2023 Judge P. Kevin Castel, in the Southern District of New York, issued an Opinion and Order on Sanctions in Mata v. Avianca.

The order opens on the finding: “Peter LoDuca, Steven A. Schwartz and the law firm of Levidow, Levidow & Oberman P.C. (the "Levidow Firm") (collectively, "Respondents") abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question. Many harms flow from the submission of fake opinions.” — P. Kevin Castel, U.S.D.J., Opinion and Order on Sanctions, Mata v. Avianca, 22 June 2023.

The penalty was $5,000, jointly and severally, paid into the Registry of the Court within 14 days. On top of that came letters to the client and to each judge falsely named as the author of a fabricated opinion. That is the shape of the failure this lesson is about. Fluent output, correct form, nothing underneath. And no step in the workflow that asked whether the cited authority was real.

Different failure types require different corrections, and the vocabulary for saying so does not have to be improvised. NIST published its Generative AI Profile on 26 July 2024. It names twelve generative-AI risk categories, and defines one of them like this: “Confabulation: The production of confidently stated but erroneous or false content (known colloquially as "hallucinations" or "fabrications") by which users may be misled or deceived.” — NIST AI 600-1, Generative Artificial Intelligence Profile. The profile goes further. It notes that confabulated citations and confabulated justifying logic can themselves mislead users into trusting the output. That is exactly the mechanism that carried six invented opinions into Mata v. Avianca.

One named risk out of twelve is a useful reminder. A wrong answer has more than one cause, and the causes take different repairs.

Unsupported generation. The output states a claim absent from the available evidence. Improve retrieval or source coverage, require claim-level support, abstain when evidence is absent. Do not try to fix it with instructions about tone.

Reasoning or calculation error. The evidence is present but the transformation or inference is wrong. Use deterministic tools where possible, decompose and verify intermediate results, add targeted examples and tests. Inspect the trace, not only the citations.

Stale or conflicting evidence. Sources disagree, or no longer match the required time and jurisdiction. Track authority and effective dates, expose conflicts to the workflow, prefer authoritative current sources, escalate unresolved contradictions.

Communication failure. The core answer may be defensible, but its scope or its uncertainty is hidden. State conditions and limitations, distinguish fact from recommendation, use calibrated user messaging, and avoid performative certainty.

It is tempting to treat a citation as the check. Three commercial legal research products were sold on a promise of “hallucination-free” citations, and they were tested against that promise on 202 queries — the first preregistered evaluation of retrieval-augmented commercial legal AI. Magesh, Surani and four colleagues published the result in 2025 in the Journal of Empirical Legal Studies. Their finding: “While hallucinations are reduced relative to general-purpose chatbots (GPT-4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time.” — Varun Magesh and colleagues, Journal of Empirical Legal Studies, 2025.

Retrieval helped. Retrieval did not finish the job. Up to a third of answers from systems built on real documents still failed. So check at the level where evidence can actually support or refute a statement, not at the level of whether a source is attached.

Segment the output. Identify atomic factual, numerical, procedural, and recommendation claims.

Classify required evidence. Map each claim to a source, tool, policy, calculation, or human authority.

Retrieve or compute. Acquire the evidence using independent components and current permissions.

Compare claim and evidence. Check entailment, contradiction, scope, date, units, and missing premises — the properties a nearby citation does not guarantee.

Revise or abstain. Remove unsupported content, request clarification, or escalate the case.

Preserve the trace. Record claim, evidence, verifier result, and final action for audit and learning.

Token probabilities describe the model’s next-token distribution under a specific context and decoding setup. A verbal phrase such as “I am certain” is another generated sequence, not a calibrated factual guarantee. There is a measured version of that warning. Dahl, Magesh and two colleagues asked models specific, verifiable questions about randomly selected federal court cases, and reported: “Second, we find that legal hallucinations are alarmingly prevalent, occurring between 58% of the time with ChatGPT 4 and 88% with Llama 2, when these models are asked specific, verifiable questions about random federal court cases. Third, we illustrate that LLMs often fail to correct a user's incorrect legal assumptions in a contra-factual question setup. Fourth, we provide evidence that LLMs cannot always predict, or do not always know, when they are producing legal hallucinations.” — Matthew Dahl and colleagues, Large Legal Fictions, 2024. Between 58% and 88%, with GPT 3.5 and then PaLM 2 ranking in between. The system that would have to raise the alarm is the system that does not always know.

The deeper point is that accuracy and calibration move independently, and this was demonstrated long before generative text. A 2017 paper on calibration opens with the result: “We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated.” — Chuan Guo and colleagues, On Calibration of Modern Neural Networks, 2017. Their example is a 110-layer ResNet on CIFAR-100 against a 5-layer LeNet: 30.6% error versus 44.9%. The deeper network is clearly the more accurate of the two, and clearly the more overconfident. Their fix is not a better prompt. It is temperature scaling — a single parameter fitted on a held-out validation set — with miscalibration measured by Expected Calibration Error (ECE), a metric the paper attributes to Naeini et al. (2015).

That is what a validated signal looks like. A held-out set, a metric, and a number you can watch. Reliability can be estimated in many ways: evidence coverage, retrieval scores, verifier agreement, ensemble variation, calibration sets, tool results, historical error rates. Each of them has to be validated for the decision it supports before it is allowed to gate anything, in the way ECE and a calibration set validate a confidence score.

A check is useful only when its result changes what the system does. For high-risk systems in the European Union that is no longer only a design principle. Article 14(4) of the Artificial Intelligence Act, Regulation (EU) 2024/1689, requires that human overseers be enabled to interpret the system’s output correctly, to decide not to use the system or to disregard, override or reverse its output, and to interrupt the system through a 'stop' button or similar procedure. First on the list is this: “to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons;” — Article 14(4)(b), Regulation (EU) 2024/1689, Artificial Intelligence Act. The escalation path, the override and the stop have to exist. The reviewer has to be warned about trusting the model. Build the pipeline so that they do.

1. Define claim classes. Separate facts, calculations, policy conclusions, and suggestions.

2. Assign evidence requirements. Specify acceptable sources, tools, dates, and authority for each class.

3. Build measurable signals. Validate retrieval coverage, verifier accuracy, conflicts, and uncertainty indicators — against a held-out set and a stated metric, as ECE is validated against a calibration set.

4. Choose action thresholds. Map signals and harm tiers to answer, qualify, ask, decline, or escalate. The override and the interruption are Article 14(4) obligations, not optional affordances.

5. Test queue effects. Measure abstention rate, reviewer load, delay, and appeal outcomes. A reviewer who is subject to automation bias and buried in a queue is not oversight.

6. Learn from misses. Add incidents and disagreement cases to evaluation without exposing the final test set.

The strongest system does not maximize the number of questions answered. It maximizes useful, supported outcomes, while exposing uncertainty and routing unsupported cases safely. Every measurement in this lesson points the same way. Retrieval-augmented legal products still hallucinated between 17% and 33% of the time on 202 preregistered queries. Models do not always know when they are hallucinating. Nine of ten surveyed benchmarks pay nothing for saying so. And one afternoon's worth of unverified citations cost $5,000 and a written apology to six judges who had never issued the opinions attributed to them.

The next lesson addresses deliberate attacks and privacy failures. Verification helps with unsupported claims, but adversarial users and untrusted content require additional trust boundaries.