Skip to content
AI.info

Deep architectures

Normalizing Flows and Invertible Networks

Explain change of variables, invertible coupling layers, exact likelihood, sampling, architectural constraints, and the limits of density as a quality signal.

By the end you can

Visual

A flow learns a reversible path between data and a simple base

The same transformations let you evaluate a likelihood in one direction. They let you draw a sample in the other.

Nothing is discarded along the route. An observation is pushed through the inverse transform. Each invertible layer reshapes space without destroying information. The transformed point is finally scored under a simple density, and the Jacobian terms accumulated on the way correct for every local expansion or contraction. Sampling runs the same route backwards, from a base sample into data space.

FigureProcess · 5 steps
  1. 1

    Data x enters

    An observation is passed through the inverse transform.

  2. 2

    Invertible layers act

    Each layer reshapes space without destroying information.

  3. 3

    Base variable z appears

    The transformed point is scored under a simple density.

  4. 4

    Jacobian terms accumulate

    Volume expansion or contraction corrects the density.

  5. 5

    Sampling reverses the route

    A base sample is transformed back into data space.

The determinant pays for changing volume

An invertible map can stretch one region of space. It can compress another. Probability density must change inversely with that local volume transformation.

The change-of-variables formula combines the base density with the absolute determinant of the Jacobian. Flow architectures are designed so this determinant and the inverse remain tractable.

Invertibility alone is insufficient; the inverse and Jacobian determinant must also be computationally manageable.

Comparison

Common flow blocks trade parallelism for flexibility

All preserve dimensionality unless augmented or combined with special multiscale operations.

Rows one and three each have a paper behind them. Real NVP, in 2017, extended the space of tractable models “using real-valued non-volume preserving (real NVP) transformations, a set of powerful invertible and learnable transformations”. Row three is Glow, from Kingma and Dhariwal in 2018: “a simple type of generative flow using invertible 1x1 convolution”.

Row one's last two notes — partial update per layer, needs permutations or mixing — are not stylistic preferences. They follow from a theorem. The DeepMind review of normalizing flows states it directly: “The efficiency of coupling layers comes at the cost of reduced expressive power. Unlike a recurrent or masked autoregressive flow, a single coupling layer can no longer represent any autoregressive transformation, regardless of how expressive the function F is. As a result, an autoregressive flow with a single coupling layer is no longer a universal approximator.” Stack D coupling layers and you do get a universal approximator. It costs D sequential computations in both directions. Whether fewer than O(D) coupling layers would do, the review lists as an open problem. Teshima and five co-authors settled the related question in 2020: coupling flows containing affine coupling and invertible linear layers are universal diffeomorphism approximators.

Row two's “one direction may be sequential” has a factor attached to it. The factor is the dimensionality of the data. The Masked Autoregressive Flow paper put it in one sentence in 2017: “MAF is capable of calculating the density p(x) of any datapoint x in one pass through the model, however sampling from it requires performing D sequential passes (where D is the dimensionality of x).” Inverse Autoregressive Flow is the mirror image: one pass to sample, D passes to score a datapoint handed to it. Which direction is fast is decided by which variables the conditioner reads. The review states the general form: “masked autoregressive flows have computational asymmetry that impacts their application and usability. Either sampling or density evaluation will be D times slower than the other.”

Row three's “adds matrix cost” was measured by the people who introduced it. Glow's ablation compares the invertible 1x1 convolution against a fixed reversing or shuffling permutation: “The model with 1 × 1 convolution has a negligible 0.2% larger amount of parameters.” The same experiment reports about 7% more wallclock time. What that buys is Glow's own results table, which beats Real NVP on all six benchmarks. CIFAR-10 3.35 against 3.49. ImageNet 32x32 4.09 against 4.28, ImageNet 64x64 3.81 against 3.98. LSUN bedroom 2.38 against 2.72, LSUN tower 2.46 against 2.81, LSUN church outdoor 2.67 against 3.08 bits/dim. Generating one 256x256 image at batch size 1 takes about 130ms on a single NVIDIA GTX 1080 Ti. That is the shape of the trade this row asks you to make: 0.2% more parameters and about 7% more wallclock, for 3.35 bits/dim on CIFAR-10 where the permutation scores 3.49.

FigureComparison · 4 columns

Affine coupling

One partition transforms another using predicted scale and shift.

  • Simple triangular Jacobian
  • Fast inverse
  • Partial update per layer
  • Needs permutations or mixing

Autoregressive flow

Each dimension depends on an ordered prefix.

  • Expressive conditional transforms
  • Triangular Jacobian
  • One direction may be sequential
  • Ordering affects computation

Invertible 1×1 mixing

Channels are mixed through a learned invertible matrix.

  • Improves cross-channel interaction
  • Tractable log-determinant
  • Adds matrix cost
  • Requires stable parameterization

Multiscale factorization

Some variables are factored out at successive resolutions.

  • Reduces later computation
  • Supports image hierarchies
  • Can route detail unevenly
  • Architecture affects sample allocation

Example

Where exact invertible density models are valuable

A flow is attractive when you need both sampling and a normalized likelihood.

The term was coined for inference, not for image generation. That is the second bullet. Rezende and Mohamed introduced it in 2015 as “a new approach for specifying flexible, arbitrarily complex and scalable approximate posterior distributions”, built “through a normalizing flow, whereby a simple initial density is transformed into a more complex one by applying a sequence of invertible transformations until a desired level of complexity is attained”. What pushed them is the restriction this lesson's VAE predecessor runs into. Most variational inference methods “employ simple families of posterior approximations in order to allow for efficient inference”. And “this restriction has a significant impact on the quality of inferences made using variational methods”. The flow was an escape from that before it was a way to make pictures.

  • Density estimation: compare log-likelihood on held-out data, and check whether the representation and preprocessing match the application. Two models with the same architecture and the same parameter count are not reporting the same quantity if they dequantized differently.
  • Posterior approximation: use flexible invertible transforms to enrich a simple variational distribution, the use Rezende and Mohamed introduced the method for.
  • Conditional generation: condition coupling networks on labels, measurements, or context while keeping the transformed variable invertible.
  • Simulation and inverse problems: learn reversible mappings between a base source and structured observations.
  • Anomaly analysis: treat low density cautiously because likelihood can emphasize background statistics unrelated to semantic abnormality, as the next section measures.

Key idea

High likelihood does not guarantee semantic familiarity

A flow can assign unexpectedly high likelihood to out-of-distribution data when low-level statistics align with the learned density, and background, contrast, or dimensional concentration can dominate semantic novelty.

Raw likelihood does not merely fail as a detector. It points the wrong way. Train a PixelCNN++ on Fashion-MNIST and raw likelihood detects MNIST at an AUROC of 0.089. That is not a weak detector but an inverted one, ranking the out-of-distribution images as the more familiar ones. On CIFAR-10 against SVHN it scores an AUROC of 0.095. Ren and seven co-authors measured this in 2019, and they also isolated the confound. The proportion of zero (background) pixels correlates with the likelihood score at Pearson r = 0.85. In their own account MNIST scores higher because “it simply has a lot more background pixels”.

The first item of advice below is worth exactly the difference between those numbers and these: “Our likelihood ratio method corrects the background effect and significantly improves the OOD detection of MNIST images from an AUROC score of 0.089 to 0.994, based on a PixelCNN++ model trained for Fashion-MNIST.” — Jie Ren and Balaji Lakshminarayanan, Google Research blog, 2019. On CIFAR-10 against SVHN the same ratio moves 0.095 to 0.931. The images did not change between 0.089 and 0.994. The comparison did.

Use likelihood ratios, typicality tests, task-specific features, and representative anomalies; exact computation does not make the chosen density a perfect detector of meaning.

Raw likelihood detected MNIST at an AUROC of 0.089; a likelihood ratio against a background model, on the same PixelCNN++, reached 0.994.

Steps

Design and validate a flow model

Begin with the measure and representation, because preprocessing changes the likelihood being modeled.

Step one is not paperwork, and its cost is on the record. A continuous flow cannot be applied to 8-bit pixels directly. Flow-based models, the DeepMind review notes, “are not directly applicable to discrete data”, and the noise added to make them applicable “can be fixed (e.g. uniform in [0, 1] for the image example above), or learned, as for example in variational dequantization (Ho et al., 2019)”. Flow++ priced the two options on a matched parameter budget at 400 epochs. Learned variational dequantization: 3.165 bits/dim. Standard uniform dequantization: 3.292. “Switching from our variational dequantization to the more standard uniform dequantization costs the most: approximately 0.127 bits/dim.” Dropping either logistic-mixture coupling or self-attention costs about 0.03. The train-test gap widens too, from about 0.02 to about 0.06 bits/dim. One dequantization line in a model card is worth more than most of the architecture lines above it.

Step five is not a precaution. It is a response to a measured failure. In 2019 Nalisnick and four co-authors asked whether deep generative models know what they don't know. They report that “the density learned by flow-based models, VAEs, and PixelCNNs cannot distinguish images of common objects such as dogs, trucks, and horses (i.e. CIFAR-10) from those of house numbers (i.e. SVHN), assigning a higher likelihood to the latter when the model is trained on the former”. The reversal repeats for FashionMNIST against MNIST, CelebA against SVHN, and ImageNet against CIFAR-10, CIFAR-100 and SVHN. It survives restricting “the flows to constant-volume transformations”. Their conclusion is the one to carry into any audit of a density claim. The results “caution against using the density estimates from deep generative models to identify inputs similar to the training distribution until their behavior for out-of-distribution inputs is better understood”.

FigureProcess · 5 steps

  1. 1. Define data support

    Specify continuous, discrete, dequantized, bounded, or transformed variables.

  2. 2. Choose reversible blocks

    Balance expressiveness, inverse speed, and determinant cost.

  3. 3. Mix dimensions

    Use permutations, invertible matrices, or alternating partitions.

  4. 4. Validate both directions

    Test reconstruction through inverse composition and sample quality.

  5. 5. Audit density claims

    Compare likelihood, typicality, semantics, and operational anomaly labels.

Position

Exact describes the computation, not the answer

Real NVP's abstract makes four promises: an unsupervised learning algorithm with “exact log-likelihood computation, exact sampling, exact inference of latent variables, and an interpretable latent space”. Read the word that carries three of them. Exact is a claim about the computation. The architecture is built so that the likelihood, the sample and the latent variable can each be obtained without approximation. None of the three says anything about whether the density that comes out is the right density.

The confusion is easy to acquire, because the sample quality arrived soon after. Glow demonstrated “that a generative model optimized towards the plain log-likelihood objective is capable of efficient synthesis of large and subjectively realistic-looking images”. That is the result that made flows look like generators rather than inference machinery. From there it is one short step to running the argument backwards. The pictures look right, so the density must be right.

That step was closed in 2019, and the failure has four numbers on it. A Glow trained on CIFAR-10 assigns the CIFAR-10 test set 3.464 bits/dim and the SVHN test set 2.389. A Glow trained on FashionMNIST assigns FashionMNIST test 2.958 and MNIST test 1.833. In both directions the data the model never saw is a full bit per dimension cheaper to encode. Cheaper to encode is the same as more probable. Nalisnick and four co-authors put it plainly: “We again see the phenomenon: the SVHN BPD is one bit lower than that of both in-distribution data sets.”

In the abstract's words, the density “cannot distinguish images of common objects such as dogs, trucks, and horses (i.e. CIFAR-10) from those of house numbers (i.e. SVHN), assigning a higher likelihood to the latter when the model is trained on the former”. The same reversal appears for FashionMNIST against MNIST, CelebA against SVHN, and ImageNet against CIFAR-10, CIFAR-100 and SVHN. It persists “even when we restrict the flows to constant-volume transformations”, so the volume term in the change-of-variables formula is not, on its own, the explanation. Nor is it a general fact about simple images beating complex ones. The effect is not symmetric, and CIFAR-10 does not get higher likelihood under a Glow trained on SVHN. Ren and colleagues retrained a PixelCNN++ of their own and found the same thing: “the OOD images from SVHN have higher likelihood than the in-distribution images from CIFAR-10, confirming the observations of Nalisnick et al. (2018), with AUROC of 0.095”.

The exactness is real, then, and it is narrow. The number says how much density this model placed at this point, computed without approximation. It does not say whether the point is the kind of thing the model was trained on. Those are different questions, and the authors say as much. Their results “caution against using the density estimates from deep generative models to identify inputs similar to the training distribution until their behavior for out-of-distribution inputs is better understood”. An anomaly detector sold on the second reading is resting on something no paper in this lesson supports.

A likelihood computed without any approximation gives SVHN 2.389 bits/dim under a model whose own CIFAR-10 test set costs 3.464.

Analogy

A rubber sheet with an accounting ledger

Stretching a rubber sheet turns a simple grid into a complex pattern. A ledger records how much each small region expands or contracts, allowing density to be recalculated.

A sheet is one physical surface, while high-dimensional transformations are compositions of learned algebraic blocks. Local volume still has to be tracked, and reversibility preserved.

Flows gain exact density by restricting transformations to reversible, measurable operations.

Reversibility creates constraints as well as capabilities

Standard flows preserve dimensionality and cannot simply discard nuisance information. Architectures must carry every detail through the invertible path. Or they must factor it out explicitly.

The restriction has a price, and Real NVP printed it in its own results table. The flow reaches 3.49 bits/dim on CIFAR-10 against PixelRNN's 3.00. Elsewhere in the same table: 4.28 on ImageNet 32x32, 3.98 on ImageNet 64x64, 2.72 on LSUN bedroom, 2.81 on LSUN tower, 3.08 on LSUN church outdoor and 3.02 on CelebA. The authors do not hide it: “We show in Table 1 that the number of bits per dimension, while not improving over the Pixel RNN [46] baseline, is competitive with other generative methods.” Roughly half a bit per dimension is what invertibility cost against the autoregressive model. The gap did not close quickly. Three years later Flow++ still listed the best flow at 3.08 on CIFAR-10, against PixelCNN++ at 2.92 and PixelSNAIL at 2.85. A third group carried RealNVP at 3.49 and Glow at 3.35 into its own 2019 baselines unchanged.

Discrete data have to be handled separately, because ordinary continuous change-of-variables formulas do not apply directly. Flow-based models, in the DeepMind review's phrase, “are not directly applicable to discrete data”. Dequantization and discrete flows change the modeled object. They should be documented.

Real NVP's own table put the flow at 3.49 bits/dim on CIFAR-10 against PixelRNN's 3.00: about half a bit per dimension is what the tractability was bought with.

Key takeaways