Decoding, Search, and Sampling

The checkpoint defines token scores. The decoding policy decides how those scores become an output trajectory. Change temperature, candidate filtering, search, or stopping, and quality and risk change with no weight touched. The best policy depends on the contract. A creative ideation tool is there to vary. Structured extraction has to stay faithful to the source, stop where it is told, and give the same answer twice.

Three published measurements set the scale of what that policy controls, and the rest of this lesson is built on them. On GPT-2 Large, greedy decoding repeated itself in 73.66% of its output. Human text repeated in 0.28%. Under exact inference over the WMT15 English-German test set, a Transformer base model gave its single global best score to the empty translation for more than 50% of sentences. And 1,000 completions drawn at temperature 0 from one fixed prompt came back as 80 distinct strings. Not one of those three numbers is a property of the weights.

Six stages run for every token. The model produces logits, scoring vocabulary candidates from the current context. Constraints come next: masks, schemas, bans, or grammar rules remove invalid candidates. Score transforms follow, where temperature and penalties modify relative preferences. Candidate filtering by top-k or nucleus sampling limits the available set. Selection or search then extends the sequence by greedy choice, sampling, beam search, or another policy. Finally the stopping check runs, where stop tokens, length limits, schema completion, or application logic end generation.

Providers may implement the details differently, so the deployed settings must be tested directly. Two of these six stages are the ones most products expose as parameters. The measured failures in this lesson land mostly in the other four. One of them lands below all six, in the batch-dependent kernels that compute the logits in the first place.

Greedy or near-greedy decoding chooses the locally highest-scoring continuation. It is simple and often repeatable, it can lock into a weak early path, it may produce repetitive continuations, and it still depends on infrastructure and model version. Stochastic sampling draws from a filtered probability distribution: diverse candidates, useful for ideation and ensembles, more run-to-run variation, candidate evaluation required. Beam-style search keeps several high-scoring partial sequences: useful in constrained sequence tasks, optimizing model score under a search approximation, able to favor generic or length-biased outputs, and adding compute and memory.

No method converts the model distribution into verified truth. The third family has an extreme case documented end to end. In 2019, Stahlberg and Byrne ran exact inference — beam search combined with depth-first search — over the entire WMT15 English-German test set under a Transformer base model. Exact inference returns the genuinely highest-scoring output. That let them separate two things that are normally tangled: what the search finds, and what the model actually prefers. Beam search failed to find the global best model score in most cases, even at beam size 100. And what it was failing to find turned out to be nothing at all. Their abstract: "For more than 50% of the sentences, the model in fact assigns its global best score to the empty translation, revealing a massive failure of neural models in properly accounting for adequacy." The search approximation was covering for the model. Search harder in that system and the output moves toward the empty string.

Use output traces and distributions rather than adjusting settings blindly. The signatures are measurable, and they have been measured. The Curious Case of Neural Text Degeneration, published in 2020, put decoding strategies and human text side by side on GPT-2 Large. Its abstract states the mechanism in one line: "maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops." Its Table 1 attaches numbers to each signature below.

The matrix should measure how good the whole artifact is and what it costs to run. Define output quality first, specifying fidelity, diversity, structure, support, and allowed variation. Choose candidate policies, including a simple baseline and a small set of justified alternatives. Evaluate repeated runs, measuring variance, tail failures, and candidate diversity. Inspect stopping behavior, tracking truncation, invalid completion, and excess generation. Price the policy, including latency, tokens, retries, and downstream validation. Then lock the release settings, versioning the complete decoding configuration with the model.

Steps three and six are the ones teams skip. The cost of skipping them has been quantified. Ten authors measured it in 2025, in Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference: "For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size." The decoding policy never changes in that comparison. It is greedy throughout. Only the GPU count, the GPU type and the evaluation batch size move, and the authors trace the cause to the non-associativity of floating-point arithmetic. An accuracy number reported without the serving configuration beside it is worth up to nine points less than it looks.

Step two has its own published evidence. A 2024 survey, A Thorough Examination of Decoding Methods in the Era of LLMs, compared deterministic and stochastic decoding across tasks, models and deployment settings — on performance, on robustness to hyperparameter changes, and on decoding speed. No method won generally: "Our findings reveal that decoding method performance is notably task-dependent and influenced by factors such as alignment, model size, and quantization." Some methods only came out ahead after extensive hyperparameter tuning. That is the published reason this experiment cannot be replaced by a recommended setting from someone else's task.

Changing the selection policy can improve one output property while weakening another. The right choice follows how much the task can let the output vary, how long it can wait, and how much validation it needs. Greedy decoding is the cheapest policy, and the one that repeated at 73.66% on GPT-2 Large. Beam search is the most thorough, and the one that ranked the empty translation first for more than 50% of the WMT15 English-German test set. Temperature 0 is the most constrained, and still returned 80 distinct completions from 1,000 draws on one fixed prompt. Each of those is a reasonable policy for some contract and a defect for another. None of them is safe by default.

Prompting comes next. Prompts alter the distribution by changing context. Decoding decides how the resulting distribution is traversed.