Skip to content
AI.info

Deep architectures

Autoregressive Generative Architectures

Explain autoregressive factorization, causal architecture choices, teacher forcing, sampling, ordering assumptions, and sequential inference cost.

By the end you can

A joint distribution becomes a sequence of conditionals

An autoregressive model assigns probability to an object by choosing an order. It predicts each element from the prefix it is allowed to see. The product of those conditional probabilities is the joint likelihood.

The factorization is exact as mathematics. The order is not. It is something the designer commits to, and the commitment has a price you can read off a table.

The habit had a name by 2014: “These models use a fixed, arbitrary ordering of the data dimensions.” The three researchers who wrote that then measured what dropping it is worth. On binarized MNIST a fixed-order single-hidden-layer NADE reaches −88.33 nats. Order-agnostic Deep NADE ensembles, averaged over 128 orderings, reach −87.71 nats at one hidden layer and −85.10 at two. A journal survey reprinted the same figures in 2016, with EoNADE 2hl at 87.96 for two orderings and 85.10 for 128.

At one hidden layer, the distance between 88.33 and 87.71 is one model family arguing with itself about nothing but the order. The rest of the way to 85.10 came with a second hidden layer. Even the first part is the size of a design decision. It sets which dependencies are easy and which predictions must wait.

Autoregression trades one difficult joint prediction for many ordered conditional predictions.

Example

A chain of three conditionals, and its clock

Invented probabilities for a three-token sequence A, B, C.

  • 1. p(A) = 0.5, p(B | A) = 0.4, p(C | A, B) = 0.9. The joint is 0.5 × 0.4 × 0.9 = 0.18, a log-likelihood of ln 0.18 ≈ −1.71 nats.
  • 2. Starting from C instead is equally exact as mathematics but asks the network for different conditionals, p(C), then p(A | C), and so on. A trained model rarely learns both orders equally well, which is why averaging over many orderings, as the NADE ensembles above do, changes the score.
  • 3. Training scores all three conditionals at once, because A, B and C are known. Generation cannot: B needs A first, and C needs both.
  • 4. The clock: a 500-token answer served at the 40 ms per output token of MLPerf's interactive test takes 500 × 40 ms = 20 seconds of decoding, however parallel training was.

Comparison

Three ways to compute causal conditionals

The factorization does not dictate one neural architecture.

The first two rows below meet in one paper. Pixel Recurrent Neural Networks, from 2016, is “a deep neural network that sequentially predicts the pixels in an image along the two spatial dimensions”. It “models the discrete probability of the raw pixel values and encodes the complete set of dependencies in the image”. The machinery its authors credit is drawn from both rows: “fast two-dimensional recurrent layers and an effective use of residual connections in deep recurrent networks”.

Raster order over pixels is an ordering choice, not a property of images. Every architectural decision in that paper follows from having made it.

FigureComparison · 4 columns

Causal convolution

Masked kernels process earlier local or dilated neighborhoods.

  • Parallel training over positions
  • Fixed receptive-field design
  • Strong spatial or temporal bias
  • Generation still proceeds causally

Recurrent decoder

A state summarizes the generated prefix.

  • Compact streaming state
  • Natural variable length
  • Sequential training dependency
  • State bottleneck can forget detail

Causal Transformer

Each position attends to the visible prefix.

  • Direct content lookup
  • Parallel teacher-forced training
  • Growing key–value cache
  • Pairwise attention cost

Hybrid decoder

Combines recurrence, convolution, attention, or state-space blocks.

  • Tailored latency profile
  • Multiple memory paths
  • More integration complexity
  • Requires clear causal masking

Visual

Training and generation expose different input histories

Teacher-forced training provides the true prefix. Once deployed, the model receives its own previous samples.

The gap has a name, a date and a measured consequence. Four researchers set it out in 2015: “This discrepancy between training and inference can yield errors that can accumulate quickly along the generated sequence.” Their answer was scheduled sampling, a curriculum that mixes the model's own predictions into the training prefix. They report that it was “used successfully in our winning entry to the MSCOCO image captioning challenge, 2015”. A competition result, not a diagnostic plot.

Five years later the same mismatch was tied to a specific failure mode. Working over three datasets with multiple test domains, Wang and Sennrich concluded in 2020 that “exposure bias is partially to blame for hallucinations” under domain shift, and that Minimum Risk Training, which avoids exposure bias, mitigates them.

The last box in the row below is therefore not a hypothetical. It is where a translation system starts producing fluent output about things that were never in the input.

FigureProcess · 5 steps
  1. 1

    Training prefix

    Ground-truth earlier elements are available.

  2. 2

    Parallel loss

    All next-element losses can be computed under a causal mask.

  3. 3

    First generated element

    The model samples or selects from its predicted distribution.

  4. 4

    Self-produced prefix

    Subsequent predictions condition on generated history.

  5. 5

    Error propagation

    An early mistake changes the context for later steps.

Predict first

Molecules one character at a time

A recurrent network writes molecules as SMILES strings, one character at a time, left to right. A string either parses as a valid molecule or it does not.

Guess what share of its outputs are valid.

Settle on an answer, then read on — the next section is the reveal

Example

Ordering changes the problem the model learns

Some objects have a natural order. Others require one to be invented, and the invention shows up in the outputs.

Molecules are the case where the damage is countable, because a generated string either parses as a molecule or does not. On the GuacaMol distribution-learning benchmark, a SMILES LSTM scores 0.959 validity, 1.000 uniqueness and 0.912 novelty. GuacaMol's authors say what that trade looks like: “The SMILES LSTM model sometimes produces invalid molecules. However, they are diverse, as attested by the uniqueness and novelty benchmarks, and closely resemble molecules from ChEMBL.”

MOSES found the same pattern on its own suite in 2020. CharRNN scores 0.975 ± 0.026 valid, AAE 0.937 ± 0.034, LatentGAN 0.897 ± 0.002. JTN-VAE, which decodes over a junction tree of chemically valid fragments, scores 1.0 ± 0.0.

Two independent benchmarks, one pattern. Character-by-character SMILES generators leak a few per cent of invalid strings (0.959 on GuacaMol, 0.975 on MOSES). A decoder that assembles only chemically valid fragments leaks none.

  • Text: left-to-right order matches common reading and supports streaming, though alternative factorizations are possible.
  • Images: raster order is simple but makes nearby vertical pixels farther apart in generation time — the ordering Pixel Recurrent Neural Networks commits to before any layer is chosen.
  • Sets: imposing an arbitrary order can create spurious asymmetry unless the model or objective accounts for permutations, the cost Deep NADE buys back by averaging over 128 orderings.
  • Audio: fine temporal order is natural, but sample-level generation creates very long sequences — WaveNet models raw audio at “tens of thousands of samples per second”, one conditional per sample.
  • Molecular graphs: a generation order must preserve chemical validity, and MOSES prices the failure at 0.975 validity for a character RNN against 1.0 ± 0.0 for a fragment-tree decoder on the same benchmark.

Key idea

Strong likelihood and satisfying samples are not the same target

Maximum likelihood rewards probability assigned to observed data. Sampling quality also depends on decoding temperature, truncation, search, constraints, and the cost of rare failures.

The two targets can separate far enough that likelihood points backwards. In 2019 eight researchers trained an autoregressive PixelCNN++ on Fashion-MNIST, then scored MNIST digits with it: “Based on the likelihood from the PixelCNN++ model, we confirm that the model assigns a higher likelihood to MNIST than Fashion-MNIST, as previously reported by Nalisnick et al. (2018), and the AUROC for OOD detection is only 0.091, even worse than random (Figure 2a).”

An AUROC of 0.091 is not a weak signal. It is an inverted one. Their likelihood-ratio correction raises the same task to 0.996. They were replicating Nalisnick and colleagues, who had reported the same inversion for Fashion-MNIST against MNIST and found that flow-based models, VAEs and PixelCNNs trained on CIFAR-10 also assign higher likelihood to SVHN.

So a model can assign reasonable likelihood while producing repetitive or unsafe samples under a poor decoder. Aggressive decoding can hide uncertainty and reduce diversity. Neither number rescues the other.

Evaluate the model distribution and the decoding policy as separate components.

Steps

Audit an autoregressive generator end to end

Separate architectural, probabilistic, and decoding failures before tuning one number.

Step three has a documented payoff. The Curious Case of Neural Text Degeneration, from 2020, names what makes steps two and three different measurements. “Even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive.” Its authors find “that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model”. Their proposal is nucleus sampling, which draws “from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail”. Same weights, same held-out likelihood, different output behaviour.

Step five is not an internal habit either. It is how the industry benchmark is written. MLCommons put Llama 2 70B into MLPerf Inference v4.0 on 27 March 2024, with a 24,576-sample subset of the Open Orca dataset. The server scenario got two latency constraints rather than one throughput figure: time to first token ≤ 2 seconds, and time per output token ≤ 200 milliseconds. MLCommons explains where the second threshold comes from: “A TPOT of 200 ms translates to a maximum allowed generation latency that maps to ~240 words per minute (depending on the tokenizer), which is often cited as the average human reading speed.”

MLPerf Inference v5.0 then added a Llama 2 70B Interactive variant, tightened to “450 ms TTFT and 40 ms TPOT (25 tokens per second per user)”. Prefill and per-step decode get their own budgets. A system can pass one and fail the other.

FigureProcess · 5 steps

  1. 1. Validate causal access

    Check masks, shifted targets, caches, and any conditioning channels.

  2. 2. Measure held-out likelihood

    Use the correct tokenization and length normalization.

  3. 3. Sweep decoding policies

    Vary temperature, truncation, search, and constraints.

  4. 4. Slice long rollouts

    Inspect repetition, drift, invalid states, and early-error cascades.

  5. 5. Benchmark latency phases

    Separate prefill, per-step decode, cache growth, and concurrency.

Analogy

Writing a story without revising previous pages

An author writes one line at a time. The existing manuscript is the only context for the next line. Training lets the author study completed books with every correct previous line visible.

An author sworn never to revise cannot reach back. Generation can, through search, resampling, or constrained decoding. The histories a deployed model sees still differ from teacher-forced training histories.

Autoregressive inference turns each prediction into part of the next input.

Parallel training does not remove sequential generation

A causal mask lets training compute losses for all positions in parallel, because the full target sequence is known. At generation time the next element does not exist until the previous step finishes.

WaveNet is what that floor costs when the elements are audio samples. The 2016 model is “fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones”, and it was trained on “data with tens of thousands of samples per second of audio”, every one of which it must generate in turn. DeepMind's own follow-up states the consequence: “However, because WaveNet relies on sequential generation of one audio sample at a time, it is poorly suited to today's massively parallel computers, and therefore hard to deploy in a real-time production setting.” Shipping it required distilling the model into a feed-forward network that generates “more than 20 times faster than real-time, a 1000x speed up relative to the original WaveNet”. Only then was it served in Google Assistant.

Caching avoids recomputing every earlier representation. Output length still creates a serial dependency. “Inference from large autoregressive models like Transformers is slow - decoding K tokens takes K serial runs of the model.” That is the opening sentence of the speculative-decoding paper from Google Research in 2023, which reports “a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs” on T5-XXL. Within months a DeepMind team measured “a 2-2.5x decoding speedup” on the 70-billion-parameter Chinchilla, “without compromising the sample quality or making modifications to the model itself”.

Two laboratories, two models, the same order of gain. The latency floor moves by a small constant factor. It does not go away.

Teacher-forced parallelism and autoregressive serving have fundamentally different critical paths.

Key takeaways