Diffusion and Score-Based Generative Architectures

Training corrupts real data at known noise levels; generation begins from noise and follows a learned reverse trajectory.

The construction is older than the models that made it famous. It was set out at ICML in 2015 by Sohl-Dickstein and three colleagues. Their abstract states it in two sentences: “The essential idea, inspired by non-equilibrium statistical physics, is to systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process. We then learn a reverse diffusion process that restores structure in data, yielding a highly flexible and tractable generative model of the data.”

Every step in the timeline below is one half of that sentence. Destruction is systematic, slow and prescribed. Restoration is the only part that is learned. Five years separate the 2015 paper from the CIFAR-10 numbers in the next section.

Training samples a data point, a timestep, and noise; the model receives the corrupted point and timestep, then predicts noise, a denoised target, velocity, or another equivalent parameterization.

Timestep conditioning is essential because the appropriate denoising operation changes with corruption level. Early reverse steps organize global structure, while later steps often refine detail.

What the network predicts is not a cosmetic choice, and there is a measured case for saying so. In 2021 Nichol and Dhariwal widened the prediction target. The reverse-process variances were learned instead of fixed. They report: “Additionally, we find that learning variances of the reverse diffusion process allows sampling with an order of magnitude fewer forward passes with a negligible difference in sample quality, which is important for the practical deployment of these models.”

An order of magnitude in forward passes, bought by changing what the same network is asked to output.

Several common targets describe related reverse-process information but are not numerically identical.

The fourth of them, score prediction, is not a theoretical aside. It was a working generative method before DDPM existed. In 2019 Song and Ermon trained a single network across many noise levels and report: “Our models produce samples comparable to GANs on MNIST, CelebA and CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.87 on CIFAR-10.”

That 8.87 is the same benchmark and the same metric as the 9.46 reported in June 2020. Score matching is the fourth card below. It had a published Inception score attached to it a year before the noise-prediction result the field remembers.

The diffusion principle does not require one specific network family. The substitution has been run as an experiment rather than argued. In 2022 Peebles and Xie described theirs in one line: “We train latent diffusion models of images, replacing the commonly-used U-Net backbone with a transformer that operates on latent patches.”

Removing the U-Net did not cost sample quality. They report a state-of-the-art FID of 2.27 on class-conditional ImageNet 256x256 — the same resolution where classifier-guided diffusion had reported 4.59. The corruption process and the reverse-estimation objective were unchanged. Only the architecture holding them moved.

Sampler comparisons should hold the trained model and conditioning setup constant wherever possible. Two published results did exactly that, and they are why the ladder below is worth walking.

DDIM changed the sampler and nothing else. Its 2020 abstract says so: “To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs.” Training untouched, it reports samples of comparable quality 10x to 50x faster in wall-clock time than DDPM. That is the swing available from step 1 alone.

Step 2 is measuring diminishing returns instead of assuming more steps help, and it has published coordinates. DPM-Solver, a fast ODE solver for diffusion sampling, was published in 2022 and measured: “Experimental results show that DPM-Solver can generate high-quality samples in only 10 to 20 function evaluations on various datasets. We achieve 4.70 FID in 10 function evaluations and 2.87 FID in 20 function evaluations on the CIFAR10 dataset”. Doubling the function evaluations moved CIFAR-10 FID from 4.70 to 2.87. The model was not trained further. Those are two points on the curve your own sweep is trying to find.

Multiple denoising steps allow correction across scales. They can yield high sample quality. They also create a longer inference path than one-pass generators.

Distillation, improved solvers, consistency objectives, and latent-space operation can reduce steps. Each changes the fidelity, diversity, or training assumptions, and has to be measured on its own.

The consistency-objective case has a paper and two figures. In 2023 Yang Song and three colleagues collapsed the iterative trajectory to a single step, and write of consistency models: “Through extensive experiments, we demonstrate that they outperform existing distillation techniques for diffusion models in one- and few-step sampling, achieving the new state-of-the-art FID of 3.55 on CIFAR-10 and 6.20 on ImageNet 64x64 for one-step generation.”

Set that 3.55 beside the 3.17 in the case section above. The earlier number came from the full iterative trajectory. The later one comes from one step. The gap between them is what a fast sampler asks a product to accept.