Language Modeling and Sequence Probability

A sentence can be common, grammatical, and false; another can be rare because it contains a new discovery, a specialist term, or an unusual name.

Language modeling estimates patterns of sequences under data and an objective. When a product needs factual support or safe action, it has to add evidence, task constraints and evaluation on top.

A court has priced that gap. Two lawyers filed a brief in Mata v. Avianca citing six judicial opinions that do not exist. On 22 June 2023 Judge P. Kevin Castel opened his sanctions order like this: “Peter LoDuca, Steven A. Schwartz and the law firm of Levidow, Levidow & Oberman P.C. (the “Levidow Firm”) (collectively, “Respondents”) abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question.”

The six fabrications were the Varghese, Shaboon, Petersen, Martinez, Durden and Miller opinions. Of the invented Eleventh Circuit decision the court wrote “Its legal analysis is gibberish.” The penalty was $5,000, imposed jointly and severally on Peter LoDuca, Steven A. Schwartz and the Levidow Firm. They were also ordered to write to each real judge falsely named as the author of one of the fake decisions.

None of that text was improbable. It read like case law because case law is what the model had been fitted to continue. It passed a lawyer, an opposing party and a judge on that resemblance alone. Probability under a text model ranges over the form of a sentence. Nothing in it ranges over whether the case exists.

A sequence probability is decomposed into a chain of next-token predictions.

Step 3 hides a decision. Taking the single most probable token at each step looks like the neutral reading of the model. It is a measured failure mode.

Five thousand continuations were generated from GPT-2 Large (762M) on held-out WebText, capped at 200 tokens. Greedy decoding scored perplexity 1.50 with 73.66% repetition. Beam search (b=16) scored 1.48 with 28.94%. The human continuations scored perplexity 12.38 with 0.28%. Holtzman and colleagues reported that in 2020. The most likely text was the most degenerate text, and the model gave the text a person had actually written a perplexity roughly eight times higher than its own worst output. Their abstract says it in a line: “maximization is an inappropriate decoding objective for open-ended text generation”.

A second group found the same failure independently in 2022, on a different corpus and with a different metric. On the Wikitext-103 test set, GPT-2 fine-tuned with the standard MLE objective repeated 69.21% of bigrams under greedy decoding and 71.94% under beam search, against 3.92% for human text. Diversity was 0.04 and 0.03, against 0.95. Two groups, two corpora, one direction. A decoding policy belongs in the system's specification, not in a default.

The two views make different information visible during pretraining, and the size of that difference has been measured with everything else held fixed.

Swap BERT's masked objective for a left-to-right one, keeping the data, the hyperparameters and the fine-tuning procedure the same, and SQuAD 1.1 dev F1 falls from 87.9 to 77.8. MRPC dev accuracy falls from 86.5 to 77.5. That ablation is BERT's own, published in 2019, and Devlin and colleagues summarise it: “The LTR model performs worse than the MLM model on all tasks, with large drops on MRPC and SQuAD.” The corruption strategy is stated as plainly: “In all of our experiments, we mask 15% of all WordPiece tokens in each sequence at random.” A different team rebuilt both objectives at BERT-Base size, on the same data, for 1M steps. It reproduced the direction: 90.0 SQuAD 1.1 F1 for a Masked Language Model against 76.7 for a pure left-to-right Language Model. That is the BART paper, 2020.

What none of this licenses is a ranking of objectives in general. The same BART table sets six pretraining objectives side by side, all BERT-Base size, 1M steps, same data and same code base. In it the pure left-to-right Language Model produces the best ELI5 perplexity in the whole table, 21.40, alongside that worst SQuAD score of 76.7. The authors mark it themselves: “For example, a simple language model achieves the best ELI5 performance, but the worst SQUAD results.”

The comparison was run a third time for T5, by a separate group with a separate publisher. On SQuAD, prefix language modelling scored 77.99, BERT-style 80.65 and deshuffling 67.61 — the same task-dependence again. Their summary: “Overall, we find that the BERT-style objective performs best, though the prefix language modeling objective attains similar performance on the translation tasks.” The objective, not the model size, decides which task you are good at.

The same pretrained model can support several roles with different risks — and the first two differ less in quality than the phrase "fine-tuned" suggests.

On CoNLL-2003 named-entity recognition, a frozen BERT-Base reached 96.1 dev F1 against 96.4 for fine-tuning the whole model. Nothing in the encoder was updated. The concatenated top four hidden layers were fed into a randomly initialised two-layer BiLSTM, and the score is an average over five random restarts. The other frozen variants ran from 91.0 for embeddings alone up to 95.9 for a weighted sum of the last four layers. Devlin and colleagues put the gap in one sentence: “The best performing method concatenates the token representations from the top four hidden layers of the pre-trained Transformer, which is only 0.3 F1 behind fine-tuning the entire model.”

An independent team, whose whole subject is this comparison, measured the same near-parity on that task's test set. Feature extraction scored 92.2 span F1 against 92.4 for fine-tuning BERT-Base, a gap of 0.2, as the mean of five seeds. Peters and colleagues found no fixed winner — “the relative performance of fine-tuning vs. feature extraction depends on the similarity of the pretraining and target tasks”. So the first two roles below are an engineering decision about cost and operations at a stated price in F1, not a quality tier.

During teacher-forced training, the model conditions on recorded previous tokens. During free generation, it conditions on its own selected outputs, so an early mistake can alter every later context.

This difference helps explain compounding errors, but it does not identify one universal cure. Decoding, training data, objective design, verification and task structure all influence behavior.

The gap was named in 2015, by Bengio and colleagues. Training maximises “the likelihood of each token in the sequence given the current (recurrent) state and the previous token”. Inference has no true previous token. There, “the unknown previous token is then replaced by a token generated by the model itself”. The consequence is theirs: “This discrepancy between training and inference can yield errors that can accumulate quickly along the generated sequence”. Their remedy, scheduled sampling, is a curriculum. It moves training “from a fully guided scheme using the true previous token”, towards “a less guided scheme which mostly uses the generated token instead”. A mitigation with a schedule to tune is not a cure.

Separate intrinsic modeling evidence from product obligations.

Step 2 is where the evidence is worst and the assumption is strongest. Contamination is a measured condition of pretraining corpora, not a hypothetical. GPT-3 shipped with benchmark text in its training data, and its own paper says why: “Unfortunately, a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model.” The fallback was measurement rather than repair — flagging examples with a 13-gram overlap with the pretraining set and reporting clean subsets.

An outside audit then put percentages on a corpus everyone was training on. Dodge and colleagues searched C4.EN in 2021 for exact matches rather than n-gram overlap, and found GLUE test inputs present in the pretraining corpus at rates “from less than 2% to over 50%”. Generation target texts: 1.87–24.88%. LAMA T-REx and Google-RE examples reproduced verbatim: 4.6% and 5.7%. One shipped frontier model whose developers said in print they could not afford to fix it, and one independent audit with numbers on it.

Step 3 inherits the caution from the callout above. Report the tokenizer, the corpus, the length and the uncertainty, and remember that the better validation perplexity in Tay and colleagues' Table 3 lost on all three downstream scores reported. An intrinsic number is evidence about the objective. The role is what has to be tested at steps 4 and 5.

Collect ten fluent claims from a domain corpus, including common false statements and rare true statements. Compare language-model scores with independent evidence labels.

Document cases where probability tracks style or frequency rather than truth. Propose retrieval, citation, validation, or abstention controls for the intended application.

That exercise exists as a benchmark. TruthfulQA, presented in 2022, “comprises 817 questions that span 38 categories, including health, law, finance and politics”. The questions are written so that “some humans would answer falsely due to a false belief or misconception”. Its three authors ran it across GPT-3, GPT-Neo/J, GPT-2 and a T5-based model. “The best model was truthful on 58% of questions, while human performance was 94%”. The direction of the size effect is the part to keep. “The largest models were generally the least truthful. This contrasts with other NLP tasks, where performance improves with model size.”