Learning Latent Energy-Based Models via Interacting Particle Langevin Dynamics

AI summary

Overview

• Research area: Statistical machine learning and generative modelling, specifically the training of latent variable models with energy-based priors (latent energy-based models, LEBMs), combining stochastic differential equations, Langevin dynamics, interacting particle systems and non-asymptotic convergence analysis.
• Technical level: Advanced. The paper relies on stochastic analysis (SDEs, Wasserstein-2 distance), MCMC theory (ULA, geometric ergodicity) and non-asymptotic optimisation bounds.
• Scope: The paper proposes a diffusion-based interacting particle algorithm, EBIPLA, for maximum marginal likelihood estimation in latent energy-based models, proves non-asymptotic convergence bounds for it under log-concavity and smoothness assumptions, and validates it on synthetic data and three image datasets.

What This Paper Is About

Training an energy-based model requires samples from the model itself to compute parameter updates, and these samples are usually produced by Markov chain Monte Carlo, which mixes slowly in high dimensions. When the energy-based model is placed in a lower-dimensional latent space (the LEBM of Pang et al., 2020), the standard training procedure becomes worse: the marginal likelihood requires integrating out the latent variables, and the prior's normalising constant is intractable, giving a "doubly-intractable" problem that is typically handled with nested MCMC loops. The goal of this paper is to solve that maximum marginal likelihood estimation (MMLE) problem with a single interacting particle system whose parameters and posterior particles evolve simultaneously, and to provide the first convergence guarantees for training LEBMs.

Key Contributions

1. A continuous-time SDE framework for LEBM training. The authors construct an interacting particle system, inspired by the Interacting Particle Langevin Algorithm (IPLA) of Akyildiz et al. (2025) and adapted to the LEBM setting, in which the θ-marginal of the stationary measure provably concentrates on the maximisers of the marginal likelihood as the number of particles N grows, under mild assumptions.

2. The EBIPLA algorithm. Discretising these SDEs with an Euler-Maruyama scheme yields the Energy-Based Interacting Particle Langevin Algorithm (Algorithm 1), which replaces sequential posterior MCMC with simultaneous particle updates and approximates the intractable prior expectation with a short-run Unadjusted Langevin Algorithm (ULA) chain.

3. First convergence bounds for training LEBMs. Theorem 1 gives a non-asymptotic bound for the exact-gradient setting, and Theorem 2 extends it to the inexact-gradient setting where the prior expectation is replaced by a biased MCMC estimate. Both hold for step sizes 0 < h ≤ 2/(μ + L) under strong convexity and L-smoothness.

4. Explicit dependence on the number of data points M. Through a novel rescaling, the bounds explicitly contain M, showing that the distance to the empirical maximiser θ⋆ decreases as M increases. The authors state that this has not previously been addressed in the available IPLA literature, and it justifies using relatively few particles on large datasets.

Main Findings

• Concentration of the θ-marginal: Choosing the noise scaling η = MN makes the joint Langevin system's θ-marginal proportional to exp(MN ℓ_M(θ)), which concentrates around the maximisers of the empirical marginal log-likelihood ℓ_M(θ) as N grows.

• Theorem 1 (exact gradient): For 0 < h ≤ 2/(μ + L), the parameter iterates satisfy a bound of the form (1 − μh)^k C₀ + C₁ h^{1/2} + C₂/√(MN), with C₁ = 1.65 (L/μ) √((d_θ + MN d_x)/(MN)) and C₂ = √(d_θ/μ), where C₀ depends on the initial law of the system and θ⋆ is the unique maximiser of ℓ_M.

• Accuracy trade-off: To reach accuracy ε > 0 with fixed M, choosing N = O(ε⁻² M⁻¹ d_θ) makes the last term order O(ε), choosing h = O(ε² d_x⁻¹) makes the middle term order O(ε), and choosing k ≥ Õ(d_x ε⁻²) makes the first term order O(ε), giving an overall O(ε) bound. Because N scales as O(M⁻¹ε⁻²), small particle counts suffice on large datasets.

• Handling non-convexity: Remark 3 states that the algorithm targets the MMLE solution under Laplace-type regularity conditions (Hwang, 1980) on ℓ_M even without the strong-convexity assumption A1; A1 is only needed for the non-asymptotic rates.

• Theorem 2 (inexact gradient): Under assumptions A1, A2 and A3, the bound for iterates (15)–(17) keeps the same structure plus an additional constant term C̃₃ that does not vanish as M, N → ∞. This irreducible term comes from the bias δ(J) of the finite ULA run and is controlled by making the ULA step size γ small enough and the number of ULA steps J large enough.

• Image benchmarks (Table 1): EBIPLA achieves MSE 0.004 ± 4e-5 and FID 27.54 ± 0.42 on SVHN; MSE 0.017 ± 5e-4 and FID 75.13 ± 0.74 on CIFAR-10; and MSE 0.013 ± 9e-5 and FID 35.72 ± 0.46 on CelebA64. It attains the best MSE and FID on SVHN and the best FID on CelebA64, and the best MSE on CIFAR-10, while the best CIFAR-10 FID belongs to LEBM (70.15, with MSE 0.020).

• Baseline comparisons on SVHN / CIFAR-10 / CelebA64 (MSE, FID): VAE 0.019, 46.78 / 0.057, 106.37 / 0.021, 65.75; 2s-VAE 0.019, 42.81 / 0.056, 72.90 / 0.021, 44.40; RAE 0.014, 40.02 / 0.027, 74.16 / 0.018, 40.95; SRI (L = 5) 0.011, 35.32 / not reported / 0.015, 47.95; LEBM 0.008, 29.44 / 0.020, 70.15 / 0.013, 37.87; SM-LEBM 0.010, 34.44 / 0.026, 77.82 / 0.014, 41.21; ED-LEBM 0.006, 28.10 / 0.023, 73.58 / 0.009, 36.73.

• Efficiency on synthetic data (Figure 1): On three synthetic datasets, sample quality and energy-landscape accuracy improve with more particles or MCMC steps. EBIPLA and LEBM reach similar sample quality, but EBIPLA learns a more realistic energy landscape, and its runtime scales sub-linearly with the number of particles, staying below LEBM's runtime even at the largest particle count.

Methodology in Plain English

The model splits into two parts: a decoder that maps a low-dimensional latent variable to the data space as an isotropic Gaussian, and a prior over the latent space written as an energy-based model with an intractable normalising constant. To fit the parameters θ = (α, β) by maximising the marginal likelihood of the observed data, the authors exploit Fisher's identity, which expresses the gradient of the marginal log-likelihood as an expectation over the posterior of the latent variables. That posterior is not available in closed form, so instead of running nested MCMC chains, they let a set of N particles per data point follow Langevin dynamics that target the posterior, and use those same particles to estimate the parameter gradient. Parameters and particles are then updated together in one coupled system of SDEs; setting the parameter noise scale to 1/(MN) makes the long-run distribution of θ concentrate on the likelihood maximisers. Discretising with Euler-Maruyama gives the EBIPLA updates: an α update, a β update and particle updates, with the prior expectation inside the α update approximated by a short ULA chain initialised from a standard normal and run for J steps. In practice the implementation uses the Adam optimiser, mini-batching to keep large datasets affordable, and a noise-addition scheme for the posterior particles so that per-batch updates approximately match the full-data discretisation over one epoch. The convergence analysis separates the error into a concentration term (how well the stationary measure concentrates on θ⋆) and a convergence term (how fast the iterates approach that stationary measure), using results from Altschuler and Chewi (2024) and adapting Dalalyan and Karagulyan (2019).

Why This Matters

Impact on research. This is the first work to provide convergence bounds for training latent energy-based models, and it extends particle-based MMLE theory to the multi-data-point and inexact-gradient settings. It also gives the first explicit dependence of the bound on the number of data points M within this line of work, which reframes why particle methods can be practical at scale. Methodologically, it replaces expensive sequential MCMC loops with a single simultaneous particle system derived from an SDE, which the authors argue is a distinct theoretical foundation even though it resembles a one-step form of persistent contrastive divergence. As far as the authors know, the only other work providing a diffusion limit for EBM training is Oliva et al. (2025), which addresses only the non-latent setting.

Real-world applications (drawn from the application areas the paper cites for EBMs):

• Generation across modalities, including image generation on SVHN, CIFAR-10 and CelebA64.
• Robust classification and anomaly detection.
• Likelihood-based evaluation of generative models and simulation-based inference.
• Compositional generation, where several learned energy functions are combined.

Industry relevance. Training EBMs at scale is bottlenecked by the cost of MCMC sampling during every parameter update. A method whose runtime grows sub-linearly with particle count and that needs fewer particles as the dataset grows could make likelihood-based generative models more affordable to train on large image corpora, and the sub-linear scaling observed in Figure 1 is the kind of property practitioners look for when budgeting compute.

Future Directions

• Extending the algorithm variants: The paper notes that momentum-enriched variants of IPLA (Lim et al., 2024; Oliva and Akyildiz, 2024) and extensions to non-smooth settings following Encinar et al. (2025) are natural extensions of EBIPLA.
• Alternative geometries: Since MMLE can be viewed as a discretisation of a gradient flow in Wasserstein-2 space (Kuntz et al., 2023), generalisations based on other geometries, as explored by Sharrock et al. (2024) for Stein variational gradient descent, are suggested.
• Weighted particles: Recent sequential Monte Carlo approaches (Carbone et al., 2024; Crucinio, 2025; Cuin et al., 2025; Cuin et al., 2026) could support generalisations of this work in which particles carry weights.
• Closing the theory-practice gap: Assumption A1 (strong convexity of the joint negative log-likelihood) is a strong condition, and Theorem 2 carries an irreducible bias term δ(J) that does not vanish as M and N grow; tightening the analysis for realistic neural-network energy functions and quantifying the ULA bias more precisely remain open.

Target Audience

Researchers and graduate students in statistical machine learning, computational statistics and applied probability who work on energy-based models, latent variable models, MCMC and particle methods, or stochastic optimisation. The paper is also relevant to practitioners who train likelihood-based generative models and care about the trade-off between sample quality, runtime and particle count, and to theoretically inclined readers interested in non-asymptotic guarantees for interacting particle systems. The combination of SDE-based derivations, Wasserstein-distance arguments and image-generation benchmarks means it is best suited to readers comfortable with advanced stochastic analysis; the experimental section alone is accessible to a broader machine learning audience.

Authors’ abstract

We develop interacting particle algorithms for learning latent variable models with energy-based priors. To do so, we leverage recent developments in particle-based methods for solving maximum marginal likelihood estimation (MMLE) problems. Specifically, we provide a continuous-time framework for learning latent energy-based models, by defining stochastic differential equations (SDEs) that provably solve the MMLE problem. We obtain a practical algorithm as a discretisation of these SDEs and provide theoretical guarantees for the convergence of the proposed algorithm. Finally, we empirically validate the effectiveness of our method on synthetic and image datasets and demonstrate that using a particle based approach offers significant improvement in computational efficiency.