Autoregressive Language Modeling and the Next-Token Objective

An autoregressive language model assigns a probability to a token sequence by multiplying conditional probabilities. Each term predicts one token from the context before it. That factorization turns many text tasks into continuation. Instructions, examples, dialogue turns, retrieved passages, and structured schemas all become prefixes that shift the next-token distribution.

The reason it works was measured before there were language models. English is far more predictable than it looks. In 1951 Shannon had human subjects guess the next letter of an English text from the letters before it. With 100 preceding letters known, his experimental bounds were an upper bound of 1.3 bits per letter and a lower bound of 0.6. He put the result plainly: “From this analysis it appears that, in ordinary literary English, the long range statistical effects (up to 100 letters) reduce the entropy to something of the order of one bit per letter, with a corresponding redundancy of roughly 75%.”

Roughly one bit of surprise per letter, and about three quarters of the text redundant given its context. That surplus of structure is what a next-token objective feeds on. Shannon's guessing game and a modern language model are the same experiment run by different subjects. The 1951 numbers are also a reminder of what is being learned: the shape of the text, not the truth of it.

Teacher forcing lets many positions contribute losses in parallel. Every position in a training sequence is a prefix for the token that follows it. So a single sequence yields as many supervised prediction problems as it has positions, and all of them share one set of parameters.

The training signal is therefore dense and local. Each position is scored only on the token actually observed next in the dataset. Nothing scores whether the passage as a whole was true, useful, or self-consistent. Cross-entropy at position after position is the whole of the supervision. The properties the loss never mentions are exactly the ones later sections have to test for.

That gap has a name. Ranzato and three co-authors coined it at ICLR 2016: “We refer to this discrepancy as exposure bias which occurs when a model is only exposed to the training data distribution, instead of its own predictions.” Under teacher forcing the observed history stays available even after a poor model guess. At inference the model conditions on the tokens its own decoding policy selected.

How much that policy matters was measured on one model. Holtzman and four co-authors scored GPT-2 Large continuations in 2020. Human text had perplexity 12.38 and repeated itself 0.28% of the time. Greedy decoding of the same model scored perplexity 1.50 with 73.66% repetition. Beam search with b=16 gave perplexity 1.48 with 28.94% repetition. Nucleus sampling at p=0.95 gave perplexity 13.13 with 0.36% repetition. The most confident decodings are the degenerate ones, and the sampling policy alone moves repetition from 0.36% to 73.66% with the parameters held fixed. Their figure caption gives the mechanism in one line: “The probability of a repeated phrase increases with each repetition, creating a positive feedback loop.”

In a product the trajectory does not stop at the token loss. Generated text influences tools, users, and later requests. Validation that can interrupt a trajectory matters as much as the decoding parameters that shape it.

Predicting text rewards a model for capturing many regularities present in its corpus, and the clearest demonstration is a task the objective never mentions. GPT-3 answered trivia questions with no question-answering head and no gradient updates. Brown and 30 co-authors reported the numbers in 2020: “GPT-3 achieves 64.3% accuracy on TriviaQA in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting”. The model has 175 billion parameters, all of them trained only to continue text. That 71.2% was state of the art relative to fine-tuned closed-book models. The task was written as a prefix, and the prefix was continued.

Build cases that separate plausible continuation from task success. Unsupported prompts already exist as a sized benchmark. TruthfulQA is 817 questions across 38 categories, built by Lin and two co-authors in 2022, and its headline is a gap: “The best model was truthful on 58% of questions, while human performance was 94%.” One other finding matters more for evaluation design than the gap does. The largest models were generally the least truthful, the opposite of the usual scaling trend. So a metric that improves with scale is not measuring this.

External checks need a ground truth the model cannot imitate. Dahl and three co-authors built one by asking specific, verifiable questions about randomly sampled federal court cases. Mean hallucination rates were 58% for ChatGPT 4, 69% for ChatGPT 3.5, 72% for PaLM 2 and 88% for Llama 2. The Journal of Legal Analysis published the study in 2024: “Using OpenAI's ChatGPT 4 and other public models, we show that LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users' incorrect legal assumptions.”

Two of those results are instructions to the evaluator. The models struggle to predict their own hallucinations, so self-reported confidence is not a check. They often accept a user's incorrect assumption, so contradiction pairs belong in the suite — change one condition, hold topic and style. Then vary prefix quality across clean evidence, distractors, and model-generated intermediate text. Trace where an early choice makes later output unrecoverable. And score claims, calculations, citations, and actions against sources independently, as the docket sampling did.

Next-token prediction offers a universal training signal wherever sequences exist. It works because text is as redundant as Shannon measured it to be in 1951. It supports flexible prompting because many tasks can be expressed as continuations — which is how 175 billion parameters trained only to predict text reached 71.2% on TriviaQA with no gradient updates.

The same design makes it easier to produce something plausible than something verified. Nothing in the loss distinguishes a real citation from a well-formed one. That is why six fabricated opinions reached a federal docket in 2023, and why hallucination rates on verifiable questions about real cases ran from 58% to 88%. Later lessons show how data, post-training, context, tools, and evaluation reshape that behavior without erasing the original objective.