A Random Matrix Theory Perspective on the Consistency of Diffusion Models

AI summary

Overview

• Research area: Generative modeling theory — specifically the statistical mechanics and random matrix theory (RMT) of score-based diffusion models, bridging linear denoiser analysis with deep network behavior.
• Technical level: Advanced. The paper relies on deterministic equivalence, self-consistent (Silverstein) equations, fractional matrix powers, and two-point RMT identities. The empirical claims are accessible, but the derivations are not.
• One-sentence scope: The paper builds an RMT framework that predicts how finite training data renormalizes the noise level and generates structured disagreement between diffusion models trained on independent data splits, then validates those predictions on linear models, UNets, and DiTs.

What This Paper Is About

Diffusion models trained on completely disjoint subsets of a dataset, with different architectures, frequently produce nearly identical images when started from the same noise seed. The paper asks why this consistency happens: the authors trace it to the fact that separate splits share nearly identical Gaussian (first- and second-moment) statistics, and that a purely linear Gaussian predictor already reproduces much of the phenomenon. The goal is to build a precise theory of how the number of training samples and the randomness of the empirical covariance control both the average behavior and the split-to-split variability of the learned denoiser and of the full sampling trajectory.

Key Contributions

1. Linear origin of consistency. The authors show that shared Gaussian statistics — equivalent to a linear denoiser — already predict cross-split agreement, using the empirical mean and covariance of each split in the Wiener-filter sampling map.
2. Finite-sample random matrix theory via a renormalized noise scale. Randomness in the empirical covariance enters the expected denoiser through the self-consistent map σ² ↦ κ(σ²), which explains why limited data overshrink low-variance eigendirections and pull samples toward the dataset mean.
3. A factorized variance law. The split-to-split covariance of the denoiser is shown to factor into an anisotropy term across eigenmodes, an inhomogeneity term across inputs, and an overall scale governed by dataset size and noise level.
4. Fractional-power deterministic equivalence. Using an integral representation of fractional matrix powers (Balakrishnan's formula) plus deterministic equivalence, the authors derive deterministic equivalents for the full sampling trajectory, not just one-step denoisers.
5. Deep-network validation. Overshrinkage, anisotropy, and inhomogeneity are qualitatively confirmed in UNet and DiT diffusion models in their non-memorization regime.

Main Findings

• Consistency is visible and quantifiable. UNet-CNN and DiT models trained on two non-overlapping splits of FFHQ32 (30k images each) under the EDM framework produce visually similar outputs from the same noise seed. Paired image MSE, averaged over 512 initial noises, confirms this: generated images are more similar across splits than to their nearest neighbors in the training set, ruling out memorization.
• A linear predictor already captures much of the effect. The Wiener filter using each split's empirical mean and covariance yields nearly identical outputs across splits, and shares visual similarity with the CNN and DiT results. Pointwise, samples closer to the Gaussian solution are more consistent across splits (Pearson r = 0.244, p = 5 × 10⁻¹⁵).
• Finite data renormalize the noise scale. In expectation, the optimal empirical linear denoiser is deterministically equivalent to the population denoiser evaluated at noise scale κ(σ²) rather than σ², where κ solves a Silverstein-type self-consistent equation involving γ = d/n. The effect is strongest at low noise scales and when γ = d/n ≫ 1.
• Variance law with three interpretable factors. The denoiser variance across dataset realizations takes the form [κ(σ²)² / (n − df₂(κ(σ²)))] × ◇(v, κ(σ²), Σ) × ◇(x − μ, κ(σ²), Σ), where ◇(u, κ, Σ) = uᵀ(Σ + κI)⁻² Σ u. This separates anisotropy, inhomogeneity, and global scaling.
• Anisotropy peaks when eigenvalue matches renormalized noise. Along a population principal component with eigenvalue λ_k, the anisotropy factor reduces to χ(λ_k, κ) = λ_k/(λ_k + κ)², which is bell-shaped, uniquely maximized at λ = κ with value 1/(4κ). Directions of greatest uncertainty are those whose variance matches the renormalized noise.
• Inhomogeneity amplifies along high-variance modes. Approximating the noised input as lying on the ellipsoidal shell of 𝒩(0, Σ + σ²I), displacement along eigenvector u_k has typical radius √(σ² + λ_k), giving (σ² + λ_k) χ(λ_k, κ) — monotonically increasing in λ_k. Pointwise prediction of denoiser consistency reaches Pearson r = 0.94 across noised images at σ² = 1, n = 1000.
• Global scaling with dataset size. Marginalizing over directions and noised samples gives a closed-form overall variance that decays as n⁻¹ in the large-n limit, with renormalization effects modifying the scaling at smaller n.
• Samples overshrink toward the mean. For the full sampling map, the expected output is μ + (2/π)∫₀^∞ Σ(Σ + κ(u²)I)⁻¹ x̄ du, an integral over stronger shrink factors than the population case since κ(u²) > u². This produces systematic overshrinkage along lower-variance directions. The authors note this is partly because 𝔼[Σ̂^{1/2}]² ≠ Σ even though 𝔼[Σ̂] = Σ.
• Sampling-map variance is a double integral of the denoiser variance. The variance of the generated sample, (4/π²)∫∫ [κκ′/(n − df₂(κ,κ′))] ⬠(v; κ, κ′, Σ) ⬠(x̄; κ, κ′, Σ) du dv, factorizes into anisotropy and inhomogeneity terms sharing the same spectral structure, with κ = κ(u²), κ′ = κ(v²).
• Fine details need more data. In the sampling-map analysis, variance in the top eigenspace decays from small sample sizes, while deviation in the lower eigenspace persists and only begins decaying at larger dataset sizes — meaning fine sample detail requires a larger dataset to be consistent across training runs.
• Deep networks show a two-phase transition. Memorization phase (n ≤ 1000): models largely reproduce training samples, and samples are much closer to a nearest neighbor in their own training split than in the control split. Renormalization phase (n ≥ 3000): distances to nearest neighbors in training and control splits become statistically indistinguishable (n > 3000), and samples increasingly resemble the linear predictors.
• Overshrinkage is visible in deep models. In the renormalization phase, generated face samples resemble an average face with smoother textures and background; reduced variance is observed along low- and mid-spectrum eigenmodes, decreasing as dataset size grows and vanishing when learned and population spectra coincide at n ∼ 30000.
• DiT is more consistent than UNet. Cross-split MSE (mean ± std) shows DiT outperforming UNet at each dataset size n.
• Statistical misalignment breaks consistency. A counterfactual experiment that partitions training data to deliberately mismatch mean and/or variance by stratifying along a chosen principal component produces markedly less consistent generations across splits.

Methodology in Plain English

The authors start from a motivating observation: train two diffusion models on disjoint halves of a dataset and sample both from the same noise seed — the outputs look alike. They quantify this with paired pixel MSE over 512 noise seeds, and show that a simple linear Gaussian predictor (a Wiener filter built from each split's empirical mean and covariance) reproduces most of the agreement. That points the finger at shared second-order statistics.

To make this precise, they restrict to a linear denoiser, an affine function of the noised input that depends on the training data only through the empirical mean and covariance. In this setting the optimal denoiser and the entire probability-flow sampling trajectory have closed forms involving matrix inverses of Σ̂ + σ²I and fractional powers Σ̂^{1/2}(Σ̂ + σ_T²I)^{−1/2}.

The technical engine is deterministic equivalence from random matrix theory: in the large-dimension limit, random matrices can be replaced by deterministic surrogates. The key relation swaps the empirical covariance Σ̂ for the population covariance Σ with the noise level λ replaced by a scalar κ(λ) solving a self-consistent equation that depends only on the aspect ratio γ = d/n and the limiting spectral measure of Σ. This absorbs all the finite-sample randomness into one scalar. Extending these tools with two-point deterministic equivalences gives the variance (not just the mean) of the denoiser, and an integral representation of fractional matrix powers extends the analysis to whole sampling trajectories rather than single denoising steps.

Finally, they test whether the same structure survives in real networks: UNet and DiT denoisers trained under the EDM framework on FFHQ64, FFHQ32, AFHQ32, LSUN church and bedroom at 32 and 64 pixels, CIFAR10, and CIFAR100 (UNet on all, DiT on a subset), with two non-overlapping splits at sizes n ∈ {300, 1000, 3000, 10⁴, 3·10⁴} and 10 runs total per architecture, sampled with the same random seed using the Heun solver and trained for 50,000 steps with Adam.

Why This Matters

• Impact on research. The paper supplies a principled baseline for reproducibility in diffusion training. It links spectral properties of data to the stability of generative outputs, and reframes consistency not as a mysterious property of deep networks but as a consequence — in large part — of shared Gaussian statistics. It also gives sharp, testable predictions of where and how two independently trained models will disagree, and clarifies the boundary between memorization and generalization as a function of dataset size.
• Practical applications:
  • Auditing trained diffusion checkpoints: predicting which spatial regions or frequency bands of a generated image are most sensitive to the training set, without retraining.
  • Data curation: estimating how many samples are needed before fine image detail becomes reproducible, which informs dataset sizing decisions.
  • Reproducibility and provenance: using expected cross-split agreement as a diagnostic for whether two model runs are converging to the same solution.
  • Detecting distribution mismatch: the counterfactual experiment shows that deliberately misaligned splits break consistency, suggesting cross-split disagreement as an unsupervised signal of harmful data stratification.
• Industry relevance. Model training runs are expensive, and practitioners frequently need to know whether a change in training data or pipeline will change outputs in a meaningful way. A theory that separates "this difference is inherent finite-sample noise" from "this difference reflects a real distribution shift" has direct value for dataset governance, model versioning, and quality control.

Future Directions

• Bridging the memorization gap. The linear theory cannot memorize individual points, so the memorization phase (n ≤ 1000) is explicitly outside its scope. Extending the analysis to cover the transition and the sample size at which it occurs — which the paper notes depends on model capacity and image resolution — is an open problem.
• Quantitative deep-network agreement. The paper reports qualitative confirmation of overshrinkage, anisotropy, and inhomogeneity in UNet and DiT, and describes RMT predictions at n = 30k correlating with empirical seed-wise deviations, but the extent to which the linear closed forms hold quantitatively for deep networks remains an open question given that deep models can exploit higher-order statistics.
• Relaxing the simplifying assumptions. The results assume μ̂ = μ (isolating covariance effects) and are presented under an infinite-σ_T approximation for the sampling map, with the finite-σ_T expressions deferred to the appendix. Assessing how much these approximations matter empirically is a natural next step.
• Rank-one and aligned-observable corrections. The variance results assume the probe v and the displacement x − μ are generic; when they are strongly aligned, additional exchange terms contribute at leading order. Characterizing these regimes would sharpen the theory's predictions.

Target Audience

Theoretical machine learning researchers working on diffusion models, generative modeling, or the statistical mechanics of learning; random matrix theorists interested in applications to deep generative models; and empirically minded practitioners who train diffusion models at scale and need a principled account of when and why runs converge to the same outputs. Readers without a background in RMT will still follow the motivating observations and the qualitative deep-network validation, but the derivations require comfort with large-dimensional asymptotics, resolvents, and deterministic equivalence.

Authors’ abstract

Diffusion models trained on different, non-overlapping subsets of a dataset often produce strikingly similar outputs when given the same noise seed. We trace this consistency to a simple linear effect: the shared Gaussian statistics across splits already predict much of the generated images. To formalize this, we develop a random matrix theory (RMT) framework that quantifies how finite datasets shape the expectation and variance of the learned denoiser and sampling map in the linear setting. For expectations, sampling variability acts as a renormalization of the noise level through a self-consistent relation $σ^2 \mapsto κ(σ^2)$, explaining why limited data overshrink low-variance directions and pull samples toward the dataset mean. For fluctuations, our variance formulas reveal three key factors behind cross-split disagreement: \textit{anisotropy} across eigenmodes, \textit{inhomogeneity} across inputs, and overall scaling with dataset size. Extending deterministic-equivalence tools to fractional matrix powers further allows us to analyze entire sampling trajectories. The theory sharply predicts the behavior of linear diffusion models, and we validate its predictions on UNet and DiT architectures in their non-memorization regime, identifying where and how samples deviates across training data split. This provides a principled baseline for reproducibility in diffusion training, linking spectral properties of data to the stability of generative outputs.