Autoregressive Generative Architectures

An autoregressive model assigns probability to an object by choosing an order. It predicts each element from the prefix it is allowed to see. The product of those conditional probabilities is the joint likelihood.

The factorization is exact as mathematics. The order is not. It is something the designer commits to, and the commitment has a price you can read off a table.

The habit had a name by 2014: “These models use a fixed, arbitrary ordering of the data dimensions.” The three researchers who wrote that then measured what dropping it is worth. On binarized MNIST a fixed-order single-hidden-layer NADE reaches −88.33 nats. Order-agnostic Deep NADE ensembles, averaged over 128 orderings, reach −87.71 nats at one hidden layer and −85.10 at two. A journal survey reprinted the same figures in 2016, with EoNADE 2hl at 87.96 for two orderings and 85.10 for 128.

The distance between 88.33 and 85.10 is one model family arguing with itself about nothing but the order. That is the size of the design decision. It sets which dependencies are easy and which predictions must wait.

The factorization does not dictate one neural architecture.

The first two rows above meet in one paper. Pixel Recurrent Neural Networks, from 2016, is “a deep neural network that sequentially predicts the pixels in an image along the two spatial dimensions”. It “models the discrete probability of the raw pixel values and encodes the complete set of dependencies in the image”. The machinery its authors credit is drawn from both rows: “fast two-dimensional recurrent layers and an effective use of residual connections in deep recurrent networks”.

Raster order over pixels is an ordering choice, not a property of images. Every architectural decision in that paper follows from having made it.

Teacher-forced training provides the true prefix. Once deployed, the model receives its own previous samples.

The gap has a name, a date and a measured consequence. Four researchers set it out in 2015: “This discrepancy between training and inference can yield errors that can accumulate quickly along the generated sequence.” Their answer was scheduled sampling, a curriculum that mixes the model's own predictions into the training prefix. They report that it was “used successfully in our winning entry to the MSCOCO image captioning challenge, 2015”. A competition result, not a diagnostic plot.

Five years later the same mismatch was tied to a specific failure mode. Working over three datasets with multiple test domains, Wang and Sennrich concluded in 2020 that “exposure bias is partially to blame for hallucinations” under domain shift, and that Minimum Risk Training, which avoids exposure bias, mitigates them.

The last box in the row above is therefore not a hypothetical. It is where a translation system starts producing fluent output about things that were never in the input.

Some objects have a natural order. Others require one to be invented, and the invention shows up in the outputs.

Molecules are the case where the damage is countable, because a generated string either parses as a molecule or does not. On the GuacaMol distribution-learning benchmark, a SMILES LSTM scores 0.959 validity, 1.000 uniqueness and 0.912 novelty. GuacaMol's authors say what that trade looks like: “The SMILES LSTM model sometimes produces invalid molecules. However, they are diverse, as attested by the uniqueness and novelty benchmarks, and closely resemble molecules from ChEMBL.”

MOSES found the same pattern on its own suite in 2020. CharRNN scores 0.975 ± 0.026 valid, AAE 0.937 ± 0.034, LatentGAN 0.897 ± 0.002. JTN-VAE, which decodes over a junction tree of chemically valid fragments, scores 1.0 ± 0.0.

Two independent benchmarks, one conclusion. An invented character order leaks a few per cent of nonsense. An order built out of valid pieces leaks none.

Separate architectural, probabilistic, and decoding failures before tuning one number.

Step three has a documented payoff. The Curious Case of Neural Text Degeneration, from 2020, names what makes steps two and three different measurements. “Even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive.” Its authors find “that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model”. Their proposal is nucleus sampling, which draws “from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail”. Same weights, same held-out likelihood, different output behaviour.

Step five is not an internal habit either. It is how the industry benchmark is written. MLCommons put Llama 2 70B into MLPerf Inference v4.0 on 27 March 2024, with a 24,576-sample subset of the Open Orca dataset. The server scenario got two latency constraints rather than one throughput figure: time to first token ≤ 2 seconds, and time per output token ≤ 200 milliseconds. MLCommons explains where the second threshold comes from: “A TPOT of 200 ms translates to a maximum allowed generation latency that maps to ~240 words per minute (depending on the tokenizer), which is often cited as the average human reading speed.”

MLPerf Inference v5.0 then added a Llama 2 70B Interactive variant, tightened to “450 ms TTFT and 40 ms TPOT (25 tokens per second per user)”. Prefill and per-step decode get their own budgets. A system can pass one and fail the other.

A causal mask lets training compute losses for all positions in parallel, because the full target sequence is known. At generation time the next element does not exist until the previous step finishes.

WaveNet is what that floor costs when the elements are audio samples. The 2016 model is “fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones”, and it runs at “tens of thousands of samples per second”. DeepMind's own follow-up states the consequence: “However, because WaveNet relies on sequential generation of one audio sample at a time, it is poorly suited to today's massively parallel computers, and therefore hard to deploy in a real-time production setting.” Shipping it required distilling the model into a feed-forward network that generates “more than 20 times faster than real-time, a 1000x speed up relative to the original WaveNet”. Only then was it served in Google Assistant.

Caching avoids recomputing every earlier representation. Output length still creates a serial dependency. “Inference from large autoregressive models like Transformers is slow - decoding K tokens takes K serial runs of the model.” That is the opening sentence of the speculative-decoding paper from Google Research in 2023, which reports “a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs” on T5-XXL. Within months a DeepMind team measured “a 2-2.5x decoding speedup” on the 70-billion-parameter Chinchilla, “without compromising the sample quality or making modifications to the model itself”.

Two laboratories, two models, the same order of gain. The latency floor moves by a small constant factor. It does not go away.