Generative AI as a System, Not a Magic Box

A brief filed in a New York federal court cited six judicial opinions that do not exist. It was fluent, correctly formatted, shaped exactly like competent legal work. The 'Varghese', 'Shaboon', 'Petersen', 'Martinez', 'Durden' and 'Miller' decisions had been produced by ChatGPT, along with the quotes and citations they carried. On 22 June 2023 Judge P. Kevin Castel signed the sanctions order in Mata v. Avianca, and put the whole systems lesson in his first sentence: “Peter LoDuca, Steven A. Schwartz and the law firm of Levidow, Levidow & Oberman P.C. (the “Levidow Firm”) (collectively, “Respondents”) abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question.” The penalty was $5,000, jointly and severally, payable into the Registry of the Court within 14 days. Not one layer of that failure is a model-quality problem. Five things broke, and each broke on its own.

Five layers sit behind any generated answer, and each can succeed or fail independently. The user and decision contract: the task, the consequence of error, the escalation path. Application behavior: prompt assembly, retrieval, tools, validation, state, policy, interface. The foundation model: the learned distribution, context use, decoding behavior, post-training tendencies. Evidence and infrastructure: sources, permissions, serving, telemetry, version records, evaluation artifacts. And the human organization: owners, reviewers, incident responders, subject experts, downstream consumers.

The independence is measurable, not rhetorical. Commercial legal research tools retrieve real authority before they answer, which is the evidence layer done properly. Magesh and five co-authors ran the first preregistered evaluation of them, published in the Journal of Empirical Legal Studies in 2025. Strengthening the evidence layer helped. It did not close the gap: “While hallucinations are reduced relative to general-purpose chatbots (GPT-4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time.”

The vendors had claimed 'eliminating' hallucinations, or 'hallucination-free' citations. A residual rate of 17% to 33%, measured per named product, is what a repaired evidence layer looks like when the other four layers are left unspecified.

A generative model estimates patterns in its training data and produces an artifact conditioned on the current input. That artifact can be text, code, an image, audio, video, or a structured record. It is shaped by the training distribution, the current context, the decoding policy, and whatever evidence the application supplied. None of those ingredients establishes truth, freshness, authorization, or fitness for a consequential decision.

The gap between fluent production and established truth has been measured on questions with checkable answers. Dahl and three co-authors put specific, verifiable questions about randomly chosen federal court cases to leading models, and reported the result in the Journal of Legal Analysis in 2024: “Second, we find that legal hallucinations are alarmingly prevalent, occurring between 58% of the time with ChatGPT 4 and 88% with Llama 2, when these models are asked specific, verifiable questions about random federal court cases.”

The same work found that the models often could not predict when they were hallucinating. So the failure does not announce itself in the artifact. That band, 58% to 88%, is the distance between a system that produces answers and a system that produces reliable ones.

The same model can support distinct workflows. The release bar should change with the job. Creative drafting treats variation as valuable and keeps a person as the editor: concepts, copy, storyboards, visual drafts, where quality includes subjective judgment and revision is visible and relatively cheap. Constrained transformation — extraction, rewriting, translation, summarization — puts most of the material the output must preserve into the input, so fidelity can be checked against a source and omissions and schema errors are measurable. Use it where the transformation contract is explicit. Knowledge or decision support helps answer, recommend, or act inside a consequential process: authority and freshness must be defined, unsupported claims can create harm, abstention and escalation are product features, and it should ship only with domain-specific evaluation.

Even the most permissive of the three carries duties the model cannot discharge. The U.S. Copyright Office issued its registration guidance for works containing AI-generated material on 16 March 2023, effective the same day. It will not register material a machine produced without creative human input, and it puts a duty on the applicant: “Consistent with the Office’s policies described above, applicants have a duty to disclose the inclusion of AI-generated content in a work submitted for registration and to provide a brief explanation of the human author’s contributions to the work.”

Provenance and originality are not a soft consideration filed under creative work. They are a dated, effective obligation to record which parts of the artifact a human made. That is a logging requirement in the application layer, and it is decided before the first draft is generated, not after.

Use this sequence, so that a convincing demo does not end up specifying the product. First, name the user decision: describe what changes if the output is accepted. Second, define the required evidence: sources, calculations, policies, freshness conditions. Third, bound acceptable failure: harms, abstention rules, review, fallback behavior. Fourth, choose the model role: does it draft, transform, retrieve, explain, or propose. Fifth, design the evaluation first, with representative cases and release criteria written before anyone polishes the interface.

The fifth step has had a name and a citable form since 2019, when Mitchell and eight co-authors proposed the model card: “In order to clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited, we recommend that released models be accompanied by documentation detailing their performance characteristics.” Their proposal was that released models ship with a short document recording intended use, disaggregated benchmarked evaluation, and the evaluation procedure itself.

That document is the counterpart to the operational gap in the opening case. It is what lets a later reader say which model, evaluated how, was judged fit for which use.

The next lessons move from tokens and model families through pretraining, post-training, prompting, retrieval, adaptation, evaluation, safety, and operations. Each topic is treated as a design choice with observable consequences, and each has a number attached to it somewhere in this lesson. Retrieval that cut hallucination without ending it, at 17% to 33% per named product. Evaluation whose absence cost $5,000 and a signed sanctions opinion. Operations that must produce a machine-readable mark under Article 50(2), and a disclosure of the human author's contribution under the Copyright Office guidance. A randomised study in which experienced developers estimated a 20% speed-up and were measured 19% slower.

The final capstone requires a complete system argument. A capable model is necessary for some products. It never makes the case on its own.