Skip to content
AI.info

Deep architectures

Variational Autoencoders and Probabilistic Latent Variables

Explain the VAE generative model, approximate posterior, evidence lower bound, reparameterization, latent regularity, and common collapse or blur failure modes.

By the end you can

Visual

The probabilistic story and the inference shortcut

A VAE defines a generative model and learns an approximate route back from observations to latent variables.

The estimator that makes the last box trainable arrived twice within six months. Kingma and Welling posted Auto-Encoding Variational Bayes on 20 December 2013 and presented it at ICLR the following year. It shows “that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods”. It also shows that “posterior inference can be made especially efficient by fitting an approximate inference model (also called a recognition model) to the intractable posterior using the proposed lower bound estimator”. Rezende and two colleagues reached the same construction independently, at ICML in 2014. They developed “stochastic backpropagation – rules for gradient backpropagation through stochastic variables”. Two groups, six months, one estimator. The recognition model is the fourth box above.

FigureProcess · 5 steps
  1. 1

    Sample latent z from a prior

    The model begins from a simple distribution such as a standard normal.

  2. 2

    Generate x from p(x|z)

    The decoder specifies a likelihood over observations.

  3. 3

    Observe a real x

    Training data arrives without its latent cause.

  4. 4

    Approximate p(z|x) with q(z|x)

    The encoder predicts a tractable posterior family.

  5. 5

    Optimize the ELBO

    Reconstruction fit and posterior regularization are balanced.

The ELBO is a tractable lower bound with two jobs

The expected log-likelihood term rewards latent samples that let the decoder explain the observation. The KL term penalizes divergence between the approximate posterior and the prior. Their balance controls information flow, and the value of the bound does not tell you where the balance landed.

A toy model with a known ground truth shows how far the balance can slip. The true mutual information between observation and latent was I(x;z) = 0.5 nats. A VAE trained to the global optimum of the ELBO came back with a rate of R = 0.0002 nats. Its latent space “completely mixes” the two true clusters. A second model, trained by targeting R = 0.5, reached R = 0.4999 nats, and recovered the two clusters at roughly 70% and 30% of the probability mass. The objective does not separate the two models. That experiment is from a 2018 paper called Fixing a Broken ELBO, by Alemi and colleagues, and it says why in one sentence: “Thus the ELBO objective alone (and the marginal likelihood) cannot distinguish between models that make no use of the latent variable (autodecoders) versus models that make large use of the latent variable and learn useful representations for reconstruction (autoencoders), in the infinite model family, as noted in Huszár (2017); Phuong et al. (2018).”

This is a property of the objective, not a bug in someone's optimizer. A second group reached the same place from a different direction, at ICLR in 2017, and Kingma's name is on that paper too. Their route is a bits-back coding argument: “even if we can solve the optimization problems exactly, the latent code should still be ignored at optimum” for sufficiently powerful decoders. Too little regularization can fragment the latent space. Too much, or a decoder strong enough, and the latent variable is simply not used.

At the global optimum of the ELBO, a latent carrying a true 0.5 nats was measured retaining 0.0002 of them.

Comparison

Do not collapse the three distributions into one

Each distribution answers a different question in the model.

FigureComparison · 4 columns

Prior p(z)

What latent values are expected before seeing an observation?

  • Chosen generative starting point
  • Used for unconditional sampling
  • Often simple by design
  • May mismatch true latent structure

Likelihood p(x|z)

How does a latent value generate or score observations?

  • Implemented by the decoder
  • Defines reconstruction loss form
  • Encodes noise assumptions
  • Can produce blur under mismatch

Approximate posterior q(z|x)

Which latent values plausibly explain this observation?

  • Implemented by the encoder
  • Provides trainable latent samples
  • Approximates an intractable posterior
  • Family can be too restrictive

True posterior p(z|x)

What Bayes’ rule implies under the generative model.

  • Usually hard to compute
  • Target of variational approximation
  • Depends on prior and likelihood
  • Can be multimodal or complex

Analogy

A cartographer drawing uncertain coordinates

A cartographer assigns each city not one exact point, but a small region expressing uncertainty. The map is encouraged to fit a common coordinate system so new locations can be sampled coherently.

North and east on a map are fixed physical directions, while latent dimensions need not correspond to physical directions or human concepts. A VAE still encodes distributions and regularizes their placement.

The hope that the axes come out meaningful anyway has been tested. It has a theorem against it: “We theoretically prove that (perhaps unsurprisingly) the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases both on the considered learning approaches and the data sets.” That is Locatello and colleagues in 2019, and they back the theorem with a pile of evidence. More than 12,000 trained models, six unsupervised methods, six disentanglement metrics, seven data sets. The bill came to approximately 2.52 GPU years on NVIDIA P100s. More than 10,000 of the models were released. Their conclusion about the well-disentangled ones in that pile is that they “seemingly cannot be identified without supervision”. You cannot pick the good map out of the stack by looking at the maps.

The same limitation arrives from the identifiability side too. A 2020 paper by Khemakhem, Kingma and two co-authors states it — “This is known to be generally impossible due to unidentifiability of the model” — and proves formally that the standard models without a conditional prior “are actually non-identifiable”. It cites Locatello for the empirical version. Two routes, one wall.

A smooth latent map is encouraged by the objective; meaningful axes are impossible to guarantee without inductive biases, and 12,000 models could not be sorted without supervision.

Key idea

Posterior collapse is an architectural and optimization interaction

A powerful autoregressive decoder may model the observation without using z. The approximate posterior then approaches the prior. The KL term becomes small — and “small” understates it.

Pair a VAE with a powerful autoregressive PixelCNN decoder on CIFAR-10, impose no mitigation, and the KL term falls below 10^-8 bits per dimension within a few thousand training steps. It never recovered, for either of the two encoder structures tested. That is Razavi and colleagues, in 2019: “We also repeated these experiments with both anti-causal and non-causal structures but without imposing a committed information rate or using other mitigation strategies, and found that neither structure by itself is able to mitigate the posterior collapse issue”. The standard fixes were not stable in their hands either. With linear KL annealing, across a wide range of annealing end-steps, “the KL collapsed as soon as beta approached 1.0”. Optimizing with the free-bits loss “was challenging and sensitive to hyperparameter values”.

KL annealing, weaker decoders, free bits, richer priors, or altered objectives can help. None should be applied without measuring latent usage and generative quality together. The measurement is per-group, not aggregate. NVAE reports exactly that instrument on its own hierarchical model: “In Fig. 5b, we visualize KL per group in CIFAR-10 (for 30 groups). Note how most groups obtain a similar KL on average, and only one group is turned off”. Vahdat and Kautz got there by applying a KL-balancing mechanism only during the first ~25,000 warm-up iterations. An average over groups would have shown none of it.

A KL below 10^-8 bits per dimension after a few thousand steps is not successful regularization; it is a latent variable the decoder walked away from.

Example

Signals that reveal what the latent variable is doing

One aggregate ELBO value cannot diagnose the whole model.

The failure these signals look for was named in 2016, in a paper about generating sentences from a continuous space. Training a sentence VAE “causes the model to initially learn to ignore z and go after low hanging fruit, explaining the data with the more easily optimized decoder”. Then: “once this has happened, the decoder ignores the encoder and little to no gradient signal passes between the two, yielding an undesirable stable equilibrium with the kl cost term at zero.” That is Bowman and colleagues, at CoNLL. Their two mitigations are still the standard first moves. One is annealing the weight on the KL term upward from zero. The other is weakening the decoder with word dropout, “applied not to a feature extractor but to a decoder”.

The second bullet below has a number behind it, and a published threshold. The test rests on one sentence: “If a latent dimension encodes useful information about the data, we would expect its distribution to change depending on the observations.” So a dimension u counts as “active” when Au = Cov_x(E_{u~q(u|x)}[u]) exceeds 10^-2. On dynamically binarized MNIST, a one-stochastic-layer VAE with 50 latent units reached 86.76 nats test NLL using 19 active units. Thirty-one of the fifty dimensions were doing nothing, in a model that looked healthy from its bound alone. Deleting the inactive dimensions from every model Burda and colleagues trained “changed the test log-likelihood by less than 0.06 nats”. The same architecture trained with the k=50 importance-weighted bound reached 84.78 nats with 25 active units. The best two-stochastic-layer IWAE (k=50) reached 82.90 nats with 26+7 active units. An independent group reproduces that figure in its own comparison table, listing “DLGM 2hl + IWAE [2] -82.90” on the same benchmark. The bound never announced the dead dimensions. A separate statistic had to go and count them.

  • Per-dimension KL: identify latent dimensions that carry measurable information rather than remaining near the prior, and read them per dimension or per group — an average hides a group that is switched off.
  • Active units: the statistic Au = Cov_x(E_{u~q(u|x)}[u]) with a 10^-2 threshold made this concrete; it counted 19 of 50 units in use in the one-layer MNIST VAE, and 25 under the k=50 importance-weighted bound.
  • Prior samples: compare unconditional outputs with reconstructions to expose mismatch between encoded and sampled regions.
  • Latent traversal: vary one coordinate while controlling others, but avoid declaring a human concept without interventions — Locatello and colleagues could not identify the disentangled models among 12,000 without supervision.
  • Importance-weighted estimates: assess likelihood more tightly when comparing models — 86.76 nats against 84.78 for the same architecture — while separating likelihood from perceptual usefulness.

Steps

Specify a VAE from the observation model backward

The decoder likelihood should reflect the data representation and measurement process rather than what is convenient. What that discipline is worth can be read off a model where every one of these choices was varied and measured.

NVAE is that model. Vahdat and Kautz published it in 2020, and its main table carries two rows. Without normalizing flows it reaches 78.01 nats on 28x28 MNIST, 2.93 bits per dimension on CIFAR-10, 2.04 on 64x64 CelebA and 0.71 on 256x256 FFHQ. With flows — a richer approximate posterior family, step 3 of the recipe — it reaches 78.19 nats on MNIST, 2.91 bits per dimension on CIFAR-10, 3.92 on 32x32 ImageNet, 2.03 on 64x64 CelebA and 0.70 on 256x256 CelebA HQ. The CIFAR-10 model uses a 30-group hierarchy. Its 131M parameter count is nowhere in the NVAE paper. It turns up in an independent lab's comparison table, which lists “NVAE (Vahdat & Kautz, 2020) VAE 131M 30 ≤ 2.91”.

The ablations are where the recipe's middle steps show their price. Batch normalization plus Swish beat weight normalization plus ELU, 3.16 against 3.31 bits per dimension at L=40 groups. Depthwise-separable top-down cells beat regular ones, 3.07 against 3.11. None of these is the objective and none is the prior. They are choices inside the observation and posterior model, and each is worth a measurable fraction of a bit per dimension.

FigureProcess · 5 steps

  1. 1. Define x precisely

    State scaling, quantization, missingness, and the unit represented by one example.

  2. 2. Choose p(x|z)

    Match likelihood support and noise assumptions to the observation.

  3. 3. Choose q(z|x)

    Set posterior family, covariance structure, and encoder outputs.

  4. 4. Monitor information use

    Track KL by dimension, active units, reconstructions, and prior samples.

  5. 5. Test downstream claims

    Evaluate interpolation, controllability, uncertainty, or representation transfer directly.

Likelihood, sample quality, and representation value can disagree

A model with stronger held-out likelihood may produce samples that humans judge less sharp. A visually appealing latent traversal may conceal poor density estimation or weak coverage. The two can be pulled apart on purpose, and in 2016 they were.

Take a good model p and a bad model q and mix them as 0.01p(x) + 0.99q(x). The mixture emits samples from the bad model 99% of the time. It costs at most log 100, about 4.61 nats, in total log-likelihood — negligible beside the thousands of nats that separate models on 32x32 CIFAR-10 images. Theis and colleagues draw the conclusion: “This shows that any model can be turned into a model which produces realistic samples at little expense to its log-likelihood. Log-likelihood and visual appearance of samples are therefore largely independent.” Average log-likelihood, Parzen window estimates and visual fidelity “are largely independent of each other when the data is high-dimensional”. A later group restates the criticism — “one can achieve high likelihood, but low image quality, and conversely, high-quality images but low likelihood” — and supplies two numbers instead of one, reporting that GANs give high precision and low recall while VAEs give low precision and high recall.

The bound can also rank the wrong data set first. A Glow model trained on CIFAR-10 assigns 3.464 bits per dimension to the CIFAR-10 test split and only 2.389 to the SVHN test split. The lower number is the higher likelihood, and it belongs to data the model never saw. The same inversion appears for FashionMNIST against MNIST, 2.958 against 1.833 bits per dimension. Nalisnick and colleagues at DeepMind reported it in 2019: “We find that the density learned by flow-based models, VAEs, and PixelCNNs cannot distinguish images of common objects such as dogs, trucks, and horses (i.e. CIFAR-10) from those of house numbers (i.e. SVHN), assigning a higher likelihood to the latter when the model is trained on the former.” An independent group reproduces the phenomenon on its own models. Their RealNVP flow, trained on 64x64 ImageNet, “assigns higher likelihood to both the CelebA dataset of celebrity photos, and the SVHN dataset of images of house numbers, compared to the target ImageNet dataset”.

Report the metrics that correspond to the intended use. A VAE is a probabilistic model, a representation learner and a generator. Success in one of those roles does not guarantee success in the others.

A CIFAR-10 model that scores SVHN at 2.389 bits per dimension against its own test split's 3.464 is the reason not to treat a likelihood as a universal score.

Key takeaways