Skip to content
AI.info

Mathematical foundations

Transformations of Random Variables and Change of Variables

Learn how discrete preimages, inverse branches, Jacobian determinants, and support changes determine the distribution of transformed random variables.

By the end you can

A transformed variable carries probability mass through a map

If X is random and Z = g(X), then Z is random too, because its value depends on the same uncertain outcome. The transformation changes the support, the shape, and sometimes the dimension of the distribution. For discrete variables, probabilities of inputs mapping to the same output are added. For continuous invertible transformations, density must also account for local stretching or contraction.

The central question is not merely how values change. It is how probability mass is redistributed.

One map runs through the rest of this lesson. It is equation (1) of a two-page note from June 1958, and it sends a pair of uniform draws to a pair of independent standard normal variates. The failures and the repairs that followed it run through the lesson too.

A transformed density must preserve total probability, not preserve pointwise height.

Case

Box and Muller, 1958: two pages that became a standard library call

One change of variables became a standard library routine. Take two independent draws from the rectangular density on the interval (0, 1), call them U1 and U2, and compute X1 = (−2 log_e U1)^(1/2) cos 2πU2 and X2 = (−2 log_e U1)^(1/2) sin 2πU2. The two outputs are independent standard normal variates. That is equation (1) of a note by Box and Muller, published in The Annals of Mathematical Statistics in June 1958. The whole thing is two pages long.

Later literature writes the same pair with the sine on the first coordinate. Tezuka, in 1991, gives V1 = sqrt(−2 ln U1) sin(2πU2) and V2 = sqrt(−2 ln U1) cos(2πU2), “where U1 and U2 are independent uniform random variates in [0,1) and V1 and V2 are independent normal random variates from N(0,1)”.

Tezuka opens that paper by recording what the two pages had become: “The Box-Muller method is one of the most common techniques for generating such variates.” Sampling code that looks like arithmetic is often a change-of-variables argument in disguise.

Analogy

Ink on elastic fabric spreads thinner but never disappears

Probability mass behaves like ink printed on elastic fabric. Stretching a region spreads the same ink over more area, lowering its density. Compressing it packs the ink more tightly. A Jacobian determinant measures the local area or volume change. A many-to-one fold stacks ink from several original regions onto the same output location.

A dimension-reducing map is a different situation. Total probability is still conserved, but the mass may concentrate on a lower-dimensional surface, where a density in the original sense no longer exists.

Density height changes so that total mass remains conserved.

Comparison

Linear and nonlinear transformations

The two cases preserve different amounts of distributional structure.

Which row of this table a map falls into is not only a fact about the mathematics. In generative modeling it is a design decision, taken so that the determinant discussed two sections below stays affordable to compute.

FigureComparison · 4 columns

Affine transformation

Z = AX + b.

  • Means and covariances transform algebraically
  • Gaussian variables remain Gaussian
  • Rank can reduce support dimension
  • Invertibility depends on A

Monotone scalar map

Z = g(X) with one inverse branch.

  • CDF transformation can be convenient
  • Density uses inverse slope
  • Support changes with g
  • Quantiles transform monotonically

Non-monotone map

Several inputs can share one output.

  • Requires summing inverse branches
  • Can create boundaries or singularities
  • May erase sign information
  • Example: squaring

Dimension-changing map

Output dimension differs from input dimension.

  • Ordinary determinant formula may not apply
  • Mass may live on a manifold
  • Requires integration over fibers
  • Common in summaries and embeddings

Sometimes transform the expectation, not the density

The expectation of h(X) can often be computed directly as an integral or sum over X. That avoids deriving the full distribution of Z = h(X). But E[h(X)] is generally not h(E[X]). Equality holds for affine h and in selected special cases. Jensen’s inequality explains the direction of the gap for convex or concave transformations, and it is central to log-likelihood and variational arguments.

The distinction is not merely pedagogical. It is written into an official metrology standard. JCGM 101, the 2008 supplement to the Guide to the expression of uncertainty in measurement, fixes the vocabulary at its clause 4.9: “In this Supplement, the term “law of propagation of uncertainty” applies to the use of a first-order Taylor series approximation to the model.” That is the transformed expectation — summary quantities pushed through a linearization of the model. Clause 5.4.1 lists the alternative implementation, Monte Carlo propagation of the full distributions, which transforms the distribution itself rather than a summary of it.

The standard also prices that alternative. A run of M = 10^6 trials, says the note to clause 7.2.1, “can often be expected to deliver a 95 % coverage interval for the output quantity such that this length is correct to one or two significant decimal digits”. A later note gives the binomial-based order statistics for reading the interval off the sorted output: r = 24 747 and s = 975 254 for M = 10^6, and r = 2 420 and s = 97 581 for M = 10^5. Transforming a mean or transforming a distribution is, in this standard, a choice between a Taylor expansion and a million draws.

Key idea

The Jacobian determinant is not optional bookkeeping

Ignoring the volume correction produces a function that usually does not integrate to one. In generative modeling that error means the proposed likelihood is not the likelihood of the transformed variable. The determinant must correspond to the correct direction of the map. Using ∂z/∂x versus ∂x/∂z changes whether the factor is multiplied or divided. Absolute value removes orientation while preserving volume magnitude.

The correction is also not free, which is why architectures are designed around it. Computing or differentiating det(W) for a c×c weight matrix costs O(c³), comparable to the O(h·w·c²) cost of the convolution itself. Glow, published by Kingma and Dhariwal in 2018, gives the fix in one line: “This cost of computing det(W) can be reduced from O(c³) to O(c) by parameterizing W directly in its LU decomposition”. With W = PL(U + diag(s)), log|det(W)| = sum(log|s|). The bookkeeping paid for itself. Glow reaches 3.35 bits/dim on CIFAR-10 against RealNVP's 3.49, 4.09 against 4.28 on ImageNet 32x32, 3.81 against 3.98 on ImageNet 64x64, and 2.38/2.46/2.67 against 2.72/2.81/3.08 on LSUN bedroom/tower/church outdoor.

When the map is specified by an ODE rather than by a stack of layers, the determinant is replaced by a Jacobian trace. FFJORD, a 2019 paper from five authors, says so in its abstract: “In this paper, we use Hutchinson’s trace estimator to give a scalable unbiased estimate of the log-density.” Using Tr(A) = E_p(eps)[eps^T A eps], the cost of computing the likelihood falls from O((DH + D³)L) for stacked invertible transformations to O((DH + D²)L̂) for continuous normalizing flows, and to O((DH + D)L̂) for FFJORD. D is the data dimension, H the largest hidden dimension.

Change of variables is a conservation-of-probability statement.

Case

Normalizing flows, 2015: a density assembled from Jacobian factors

Modern generative models are built on this bookkeeping. Rezende and Mohamed brought it into variational inference in 2015. Their abstract states the construction: “Our approximations are distributions constructed through a normalizing flow, whereby a simple initial density is transformed into a more complex one by applying a sequence of invertible transformations until a desired level of complexity is attained.” Every one of those transformations contributes its own Jacobian factor to the density.

The idea and the term are older, and the authors say so themselves. A 2021 survey in the Journal of Machine Learning Research, which Rezende and Mohamed co-wrote, records that “Rezende and Mohamed (2015) used the idea and language from Tabak and Turner (2013) to apply normalizing flows in the setting of variational inference.”

What made the factors affordable was choosing maps whose Jacobian is triangular. Real NVP, at ICLR 2017, leaves x_1:d unchanged and rescales and shifts the rest: y_d+1:D = x_d+1:D ⊙ exp(s(x_1:d)) + t(x_1:d). The determinant of that coupling layer is exp(sum_j s(x_1:d)_j). Section 3.3 draws the consequence: “Since computing the Jacobian determinant of the coupling layer operation does not involve computing the Jacobian of s or t, those functions can be arbitrarily complex.” Their Table 1 reports 3.49 bits/dim on CIFAR-10 test and, on validation, 4.28 on Imagenet 32×32, 3.98 on Imagenet 64×64, 2.72 LSUN bedroom, 2.81 LSUN tower, 3.08 LSUN church outdoor and 3.02 CelebA. The architecture was chosen by the change-of-variables formula, not decorated with it afterwards.

Steps

Transform a distribution safely

Follow this sequence before writing a transformed density.

The last step is the one that gets skipped, and there is a published warning about skipping it. In 1973 Neave generated 1,000,000 standard normal observations through the Box–Müller transformation, driven by a multiplicative congruential generator. All 1,000,000 were restricted to the range (−3.3 : 3.6), with marked local maxima. The change of variables was exactly right. The inputs were not what the derivation assumed: the two uniform draws were consecutive outputs of one generator, and so were not independent.

The pathology is not confined to one generator family. Tezuka's abstract states the finding: “In this paper, we show that the use of the Box-Muller method with Tausworthe sequences for generating normal deviates can produce pathological phenomena similar to the Neave effect, which was found to occur when linear congruential sequences and the Box-Muller method are combined.” The same 1991 paper shows where the tails end. On a 32-bit word-size computer the Box-Muller output is bounded above in absolute value by sqrt(−2 ln(2^−32)) ≈ 6.660. Over the entire period of the generator x_i = 16807 x_{i−1} (mod 2^31 − 1), Tezuka tabulates tail-frequency deviations as large as −11.5 standard errors. A correct Jacobian does not protect a sampler whose inputs violate the independence the derivation assumed. Only step 6 catches that.

FigureProcess · 6 steps

  1. 1. Write the map

    Declare Z = g(X), its input space, and its output support.

  2. 2. Check invertibility

    Identify every inverse branch or preimage set.

  3. 3. Choose the route

    Use probability sums, CDFs, or the density formula as appropriate.

  4. 4. Compute the Jacobian

    Track the derivative or determinant in the correct direction.

  5. 5. Verify support

    Set the density to zero outside the transformed range.

  6. 6. Normalize and simulate

    Check the integral and compare with sampled transformations.

Visual

Four transformation cases

The correct rule depends on whether the map is discrete, continuous, invertible, and dimension-preserving.

A fifth situation sits outside this table. When the map is specified by an ODE rather than by a formula or a stack of layers, the determinant gives way to a Jacobian trace, estimated as the callout above describes.

FigureHierarchy · 4 levels

  • Discrete many-to-one

    Sum the probabilities of all inputs that map to the same output.

    • Continuous monotone scalar

      Use the inverse derivative to correct local length change.

      • Multivariate bijection

        Use the absolute determinant of the inverse Jacobian.

        • Non-invertible map

          Sum or integrate over every preimage branch or level set.

Example

Squaring a symmetric random variable

Let X be standard normal and define Z = X².

  • Many-to-one map: Both x = √z and x = −√z produce the same positive z.
  • Support: Z cannot be negative, even though X ranges over the real line.
  • Density: Contributions from both inverse branches must be included.
  • Jacobian factor: Each branch includes the magnitude of dx/dz.
  • Result: Z follows a chi-square distribution with one degree of freedom.

Key takeaways