Technical Deep Dives

Diffusion Models: The Mathematics Behind AI Image Generation

The mathematics behind diffusion image generation: the forward noising chain, the noise-prediction loss, score-based SDEs, DDIM sampling, latent diffusion, classifier-free guidance and the diffusion transformer.

Gabriele Masetti · 2026-08-11

From Noise to Image: The Core Idea

Diffusion models generate images by learning to reverse a process of destruction. Take a real photograph and add a small amount of Gaussian noise to it. Repeat this hundreds or thousands of times, and the image dissolves into pure static — a sample from a standard normal distribution. That is the forward process, and it requires no learning at all: it's a fixed, mathematically defined Markov chain.

The generative model's entire job is to learn the reverse: starting from pure noise, remove it step by step until a coherent image emerges. Because each individual denoising step is small, it's tractable to model with a neural network, even though the overall transformation — noise to photorealistic image — is enormously complex.

That framing, formalized as Denoising Diffusion Probabilistic Models (DDPM) by Jonathan Ho, Ajay Jain, and Pieter Abbeel in 2020, is what underlies Stable Diffusion, DALL·E 2, Imagen, and the diffusion transformer at the core of OpenAI's Sora — a roll call in which several of the named products have since been retired while the framework outlived them. It replaced a generation of GANs as the dominant paradigm for image synthesis not because it's more elegant in principle, but because it's more stable to train and scales more predictably with data and compute.

The Forward Process: Destroying Information on Purpose

Formally, the forward process is a fixed Markov chain that adds Gaussian noise to a data sample $x_0$ over $T$ timesteps according to a variance schedule $\beta_1, \dots, \beta_T$:

q(x_t | x_{t-1}) = N(x_t; sqrt(1 - beta_t) * x_{t-1}, beta_t * I)

Because each step is Gaussian, the chain has a convenient closed form: you can jump directly from the clean image $x_0$ to any noise level $t$ without simulating every intermediate step. Defining $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}t = \prod{s=1}^t \alpha_s$, the marginal distribution is:

q(x_t | x_0) = N(x_t; sqrt(alpha_bar_t) * x_0, (1 - alpha_bar_t) * I)

In practice this means you can sample a noisy version of any training image at any timestep in a single operation: $x_t = \sqrt{\bar{\alpha}_t}, x_0 + \sqrt{1-\bar{\alpha}_t}, \epsilon$, where $\epsilon$ is standard Gaussian noise. That identity is what makes DDPM training efficient — there's no need to unroll a thousand-step chain to get a training example.

The variance schedule matters more than it might look. Ho et al.'s original paper used a linear schedule for $\beta_t$, increasing from about $10^{-4}$ to $0.02$ across 1,000 steps. Nichol and Dhariwal's follow-up work on improved DDPMs showed that a cosine schedule, which destroys information more slowly near the beginning and end of the chain, improves log-likelihoods and sample quality — the linear schedule tends to make images unrecognizable too early, wasting model capacity on timesteps that carry little useful signal.

The Reverse Process: What the Network Actually Learns

The reverse process $p_\theta(x_{t-1}|x_t)$ is what a neural network parameterizes. Ho et al.'s key insight was that rather than having the network directly predict the denoised image, it's more effective and empirically more stable to have it predict the noise $\epsilon$ that was added at each step. Given a noisy image $x_t$ and the timestep $t$, the network $\epsilon_\theta(x_t, t)$ outputs an estimate of the noise component, and the previous, slightly-less-noisy state is reconstructed by subtracting a scaled version of that estimate:

x_{t-1} = (1/sqrt(alpha_t)) * (x_t - (beta_t/sqrt(1 - alpha_bar_t)) * eps_theta(x_t, t)) + sigma_t * z

where $z$ is fresh Gaussian noise (added at every step except the last) and $\sigma_t$ controls stochasticity. The noise-prediction parameterization connects diffusion models directly to score-based generative models: predicting the noise $\epsilon$ added to a sample is mathematically equivalent to estimating the score function — the gradient of the log-density, $\nabla_x \log p(x_t)$ — up to a known scaling factor. That equivalence is not a coincidence; it's the bridge between two lines of research that converged on the same objective from different directions.

Training: A Deceptively Simple Loss

Despite the layered probabilistic derivation — DDPM is framed as optimizing a variational lower bound on the data log-likelihood, the same family of objectives used in VAEs — the practical training loss Ho et al. arrived at is remarkably simple. Rather than optimizing the full weighted variational bound, they found that a simplified objective, which just measures the mean squared error between the true noise and the predicted noise, produced better sample quality:

L_simple = E_{x0, t, eps} [ || eps - eps_theta(x_t, t) ||^2 ]

At each training step: pick a random image from the dataset, pick a random timestep $t$, sample noise $\epsilon$, form the noisy image $x_t$, and train the network to predict $\epsilon$ from $x_t$ and $t$. There's no adversarial discriminator, no min-max game, no mode collapse dynamics to fight — just regression. That is a large part of why diffusion models proved easier to train reliably than GANs at scale: the objective is a stable, well-behaved loss rather than an equilibrium between two competing networks.

On unconditional CIFAR-10, the original DDPM paper reported an Inception Score of 9.46 and a (then state-of-the-art) FID of 3.17, and produced 256×256 LSUN samples comparable in quality to ProgressiveGAN — a strong result for a model with such an unglamorous training procedure.

Score-Based Models and the Continuous-Time View

In parallel, Yang Song and Stefano Ermon developed score-based generative models, which approach the same problem from a different mathematical direction: rather than a discrete chain of Gaussian steps, learn the score function of the data distribution at multiple noise levels and generate samples via Langevin dynamics. Song, Sohl-Dickstein, Kingma, Kumar, Ermon, and Poole unified this line of work with DDPM in "Score-Based Generative Modeling through Stochastic Differential Equations" (ICLR 2021, awarded an Outstanding Paper Award).

The paper shows that both DDPM and score-matching models are discretizations of the same underlying idea: a stochastic differential equation that smoothly transforms data into noise, paired with a reverse-time SDE that depends only on the score function and can be run backward to generate samples.

The continuous-time framing matters practically because it decouples the model (a neural network estimating the score at any noise level) from the sampler (a numerical SDE or ODE solver). It opens the door to faster, more accurate samplers developed independently of how the network was trained, and it clarifies why techniques originally derived for DDPM — like changing the noise schedule or the sampling step count — transfer cleanly to score-based formulations.

Sampling Faster: DDIM

A practical problem with the original DDPM formulation is that generating an image requires walking the full Markov chain — typically 1,000 sequential network evaluations. Song, Meng, and Ermon's Denoising Diffusion Implicit Models (DDIM, 2020) addressed this by deriving a non-Markovian family of forward processes that share the same training objective as DDPM but permit a different, deterministic reverse process.

Because the reverse process no longer needs to visit every intermediate timestep, DDIM allows skipping steps — sampling with 50 or even 20 steps instead of 1,000 — while reusing weights from a model trained the standard DDPM way, with no retraining required. The original DDIM paper reported 10× to 50× faster sampling in wall-clock time relative to DDPM, and its deterministic variant additionally supports meaningful interpolation in the latent noise space, since two runs from the same initial noise reliably produce the same output.

Virtually every production image-generation system today uses some descendant of DDIM or a related fast ODE-based sampler rather than the original thousand-step ancestral sampling.

Latent Diffusion: Why Stable Diffusion Doesn't Denoise Pixels

Running a diffusion process directly on pixel space is expensive: a 512×512 image has over 260,000 pixel values, and every one of the thousand denoising steps requires a full forward pass through a large network operating at that resolution. Rombach, Blattmann, Lorenz, Esser, and Ommer's "High-Resolution Image Synthesis with Latent Diffusion Models" (CVPR 2022) — the paper underlying Stable Diffusion — solved this by first compressing images into a much smaller latent space using a pretrained autoencoder, and then running the diffusion process entirely in that compressed representation.

The autoencoder is trained to discard only perceptually irrelevant, high-frequency detail, so decoding the final denoised latent back to pixel space loses little visible quality while cutting the computational cost of both training and sampling dramatically.

Latent diffusion also introduced a clean mechanism for conditioning generation on other modalities. Text prompts, encoded by a language model, are injected into the denoising U-Net via cross-attention layers at each resolution level, letting the same architecture handle text-to-image, inpainting, and other conditional tasks through one general-purpose interface rather than bespoke architectures per task.

Guiding Generation: Classifier-Free Guidance

A diffusion model trained purely to reconstruct noise doesn't automatically produce images that closely match a text prompt — conditioning alone tends to yield outputs that are on-topic but visually weak or generic. Early work used classifier guidance, mixing in gradients from a separately trained image classifier to push samples toward a target class, but this required training and maintaining an extra classifier and only worked for labels that classifier understood.

Ho and Salimans' classifier-free guidance (2022) removed that dependency. During training, the conditioning information (e.g., the text embedding) is randomly dropped a fraction of the time, so a single network learns to act as both a conditional model $\epsilon_\theta(x_t, t, c)$ and an unconditional one $\epsilon_\theta(x_t, t)$. At sampling time, the two predictions are combined:

eps_guided = eps_theta(x_t, t) + w * (eps_theta(x_t, t, c) - eps_theta(x_t, t))

The guidance weight $w$ amplifies the difference between what the model predicts with and without the prompt, pushing samples further toward the condition than the raw conditional model would go on its own. Higher $w$ trades diversity for prompt fidelity and visual sharpness — this is exactly the "guidance scale" or "CFG scale" slider exposed in Stable Diffusion and similar interfaces. Classifier-free guidance is now close to universal in text-to-image systems because it delivers most of the benefit of classifier guidance with no extra model to train.

U-Net vs. Diffusion Transformers

The backbone network estimating noise at each step was, for years, almost always a U-Net: a convolutional encoder-decoder with skip connections between corresponding resolutions, augmented with self-attention blocks at lower resolutions and residual connections, closely following the architecture used in Ho et al.'s original DDPM and refined in later Stable Diffusion versions. The U-Net's inductive biases — locality, translation equivariance, multi-scale processing — fit images well and made it the default choice.

Peebles and Xie's "Scalable Diffusion Models with Transformers" (DiT, 2022) questioned that default. DiT replaces the U-Net entirely with a plain transformer operating on patches of the latent representation, following the recipe of vision transformers rather than convolutional networks. The paper's central empirical finding is that DiT performance scales predictably with compute — measured in Gflops via model depth, width, and token count — much like scaling laws observed in language models, and the largest variant, DiT-XL/2, set a new state-of-the-art FID of 2.27 on class-conditional ImageNet at 256×256, beating prior U-Net-based diffusion models.

That scalability, plus the fact that transformers unify naturally with other transformer-based components (text encoders, in particular), is why later systems moved toward transformer backbones. OpenAI's Sora, described in its 2024 technical report, extends this idea to video: it compresses video into a latent space, breaks that latent into spacetime patches (patches spanning both spatial extent and time), and trains a diffusion transformer to predict clean patches from noisy ones — treating video generation as the same denoising problem as image generation, just over a token sequence with an added time axis.

Sora did not survive as a product. OpenAI announced on 24 March 2026 that it would discontinue the consumer app, which closed on 26 April 2026, with the Sora 2 API scheduled for removal on 24 September 2026. OpenAI's image line moved earlier and in a different direction: ChatGPT stopped calling DALL·E in March 2025 in favour of image generation carried natively inside GPT-4o, and the dall-e-2 and dall-e-3 API snapshots were shut down on 12 May 2026. Neither retirement touches the mathematics. What changed is the product surface: the image generator inside ChatGPT is no longer a separate denoising network called as a tool, so the derivation here now maps onto Stable Diffusion and the open latent-diffusion checkpoints more directly than onto anything OpenAI ships.

Where This Lands in Practice

The lineage from DDPM to modern systems is a story of removing bottlenecks one at a time. DDPM established that iterative denoising, trained with a plain noise-prediction loss, produces high-fidelity samples without adversarial training. DDIM cut sampling cost by an order of magnitude by decoupling the sampler from the Markov chain.

Latent diffusion cut both training and inference cost further by moving the entire process into a compressed representation, which is the specific innovation that made Stable Diffusion feasible to run on consumer hardware rather than requiring the cluster-scale resources of Google's Imagen, which instead scales quality by cascading three separately trained diffusion models — a 64×64 base model followed by two super-resolution stages up to 1024×1024, conditioned on embeddings from a large frozen T5-XXL language model.

Classifier-free guidance made prompt adherence controllable without extra models. And diffusion transformers made the backbone itself scale the way large language models do, which is precisely the property that let the same denoising framework extend from static images to video: Sora, described in OpenAI's February 2024 technical report, generated clips of up to a minute by denoising spacetime patches.

None of these pieces is exotic mathematics — Gaussian noise, a regression loss, an attention mechanism. What's notable is how cleanly they compose: each advance is a modular substitution into the same forward/reverse noising framework rather than a wholesale rethinking of it, which is a large part of why the field has been able to iterate on diffusion models so quickly since 2020.