From Probability Distributions to Generated Artifacts

Given the beginning of a sentence, a language model assigns scores to possible next tokens. An image generator does the same for pictures: many outputs are compatible with one prompt and its conditioning inputs. Generation chooses a path through that possibility space. The chosen path can be coherent without being factual.

That is not an accusation from outside the field. It is a numbered risk in a federal standards document. NIST's Generative AI Profile, published in July 2024, lists confabulation as risk 2 of 12 and defines it as “the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived”. The profile attributes it to the design, not to misuse: “Confabulations are a natural result of the way generative models are designed: they generate outputs that approximate the statistical distribution of their training data; for example, LLMs predict the next token or word in a sentence or phrase.”

How often that happens has been counted. Dahl and colleagues put specific, verifiable questions about randomly selected federal court cases to general-purpose models. Their 2024 paper in the Journal of Legal Analysis gives the range: “we find that legal hallucinations are alarmingly prevalent, occurring between 58% of the time with ChatGPT 4 and 88% with Llama 2, when these models are asked specific, verifiable questions about random federal court cases”. The questions had checkable answers. The models could not reliably predict when they were hallucinating. The error rate arrived without a warning attached.

The mechanism producing these distributions is not new. Sohl-Dickstein and colleagues described diffusion generation in 2015 as a slow destruction of structure, then its reversal. Ho and colleagues refined it in 2020 and reported an Inception score of 9.46 on CIFAR-10. Brown and colleagues had shown the same year that a 175-billion-parameter language model could work from examples alone. Every one of those results is a claim about statistical fit. None of them is a claim about truth.

The exact factorization varies by model family. But each step depends on prior state and conditioning. And the third step — the selection policy — is not a formality that can be left at its default.

One experiment isolates it. Hold the model fixed at GPT-2 Large, 762M parameters, generate 5,000 passages, and vary nothing but the decoding policy. Greedy decoding gave perplexity 1.50 with 73.66% repetition. Beam search at b=16 gave 1.48 with 28.94% repetition. Pure sampling gave perplexity 22.73. Nucleus sampling at p=0.95 gave 13.13 with 0.36% repetition. Human text sits at perplexity 12.38 and 0.28% repetition. One unchanged distribution, and repetition running from 73.66% down to 0.36%, depending only on how a number was turned into a next state.

Holtzman and colleagues, who ran it, name the trap in their abstract: “even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops.”

Note where the lowest perplexity sits. It is 1.48, under beam search, against 12.38 for human text. The most likely continuation was the least human one.

Each has to be measured differently, and each needs its own way of reducing it. The most common collapse is to read a token probability as a confidence readout. OpenAI's own GPT-4 technical report, from March 2023, contains the number that breaks that reading. On a subset of MMLU the pre-trained model had an expected calibration error of 0.007. The same model after PPO post-training had an expected calibration error of 0.074. Worse by roughly a factor of ten, with the underlying knowledge unchanged. The figure caption states it without hedging: “The post-training hurts calibration significantly”.

What that costs a user appears in the report's own Limitations section: “GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake.” A quantity that moves by an order of magnitude because of an alignment step, while what the model knows stays put, is a property of the model's expressed confidence. It is not a measurement of missing evidence. And it says nothing at all about what the surrounding workflow will do in production.

These cases show why the kind of artifact, and what it costs when it is wrong, both matter. Two of them have measured failure rates attached, from controlled studies rather than anecdote.

Choose a generative feature and annotate the full decision path. Mark every probabilistic choice, including model sampling, retrieval ranking, and classifier thresholds. Identify the values that must satisfy exact deterministic rules. Record separately the claims that cannot be justified from available sources. Add independent checks: calculators, parsers, policy engines, human review. Then define the point at which the system should abstain rather than continue generating.

Two of those annotations are easy to skip, and the anchors above show why neither should be. Write down who chose the decoding policy and what it is set to. Changing that alone moved repetition from 73.66% to 0.36%. Then write down which fields resolve against an external database, and who runs that query. In Mata v. Avianca, nobody did.

The value of generative models comes from handling open-ended input and producing flexible artifacts. That same flexibility forces the application to declare what remains variable and what must be checked. The figures in this lesson describe one boundary from different sides: 58% and 88% hallucination on verifiable legal questions, an expected calibration error moving from 0.007 to 0.074 without any change in knowledge, 3% secure solutions against 21%, roughly 1% of transcriptions carrying invented sentences.

The next lessons examine the representations and architectures that create these distributions. Keep the boundary between statistical plausibility and product validity visible throughout the path.