Latent-Autoregressive GP-VAE Language Model

AI summary

Author: Yves Ruffenach (Conservatoire National des Arts et Métiers) — arXiv:2512.09535v2 [cs.LG], 15 Dec 2025, CC BY 4.0. Note: the supplied metadata title is "Latent-Autoregressive GP-VAE Language Model"; the paper content itself is titled "Latent Autoregression via Gaussian-Process Priors in Variational Autoencoders," and the provided text contains no language-modeling results.

Overview

• Research area: Probabilistic machine learning / Bayesian non-parametrics — specifically the intersection of Gaussian processes (GPs), variational autoencoders (VAEs), and sequential generative models.
• Technical level: Advanced. The paper is written in the register of stochastic processes and approximate Bayesian inference, using measure-theoretic conditioning (disintegration on Polish spaces), Kronecker-product algebra, and Gaussian-conditioning formulas. The introductory framing of VAEs and ELBOs is accessible, but the core development is not beginner-level.
• Scope in one sentence: The paper defines and analyzes a sequential latent-variable model in which all temporal dependence lives in a Gaussian-process prior on a continuous latent trajectory, whose joint Gaussian law is factored causally into latent autoregressive conditionals, and embeds that prior in a VAE-style variational objective.

What This Paper Is About

Most sequential models put temporal dependence either in the observations (classical autoregression, where each output conditions on all previous outputs) or in learned latent transitions (recurrent or state-space models). This paper asks whether temporal dependence can instead be carried entirely by a Gaussian-process prior on a latent path, so that causality becomes a property of Gaussian conditioning rather than of a learned recurrence.

The goal is to make the causal factorization of that joint Gaussian distribution explicit, derive its conditional distributions in closed form, and show how such a latent autoregressive process can be coupled with a non-autoregressive decoder inside an evidence-lower-bound (ELBO) objective.

Key Contributions

1. Formalization of a class of purely latent autoregressive processes. The paper takes a finite grid of evaluation points on [0,1], assumes the latents arise as evaluations of a centered GP with kernel k, and defines latent autoregression through the causal factorization p(z_{1:L}) = ∏{t=1}^{L} p(z_t | z{<t}).

2. Explicit Gaussian conditionals and basic properties. Proposition 4.1 ("Causal Gaussian Factorization") gives the unique conditional laws p(z_t | z_{<t}) = N(μ_t(z_{<t}), Σ_t), with μ_t affine in z_{<t} and Σ_t symmetric positive definite, plus explicit mean and covariance formulas derived from k and the conditioning identities. The paper also discusses the covariance structure and the relation to Markov and AR(p) processes, and states a formal definition (Definition 4.2) of a latent autoregressive process in terms of Lebesgue densities.

3. A variational formulation with a correlated GP prior. The latent process is combined with a non-autoregressive observation model p_θ(x_{1:N} | z_{1:L}) that is conditionally independent across positions given the full latent sequence, an amortized Gaussian encoder q_φ(z_{1:L} | x_{1:N}), and an ELBO in which the KL term measures deviation from the latent autoregressive GP prior. A scaled variant L_β with a β > 0 weight on the KL term is also given (β = 1 recovers the standard ELBO).

4. A proof-of-concept implementation. The framework is implemented with finite-dimensional GP priors and a non-autoregressive decoder, and numerical results are reported on "a standard sequence dataset." The paper explicitly states these experiments are not the main contribution and are run in a constrained proof-of-concept regime.

Main Findings

• Causality from geometry, not weights: In this construction, causality is a probabilistic property of the latent path obtained by Gaussian conditioning, rather than the result of an explicit recurrence in the observation space. The conditional law of z_t depends on the entire history z_{<t}, not only on z_{t−1}.

• Unique causal factorization: Under the kernel-regularity and latent-dimension hypotheses (H1)–(H2), the finite-dimensional Gaussian vector z_{1:L} ∼ N(0, K_tt ⊗ I_{d_z}) admits a unique causal factorization into Gaussian conditionals, with the first term p(z_1) = N(0, k(t_1,t_1) I_{d_z}). Uniqueness is up to null sets and is derived via the disintegration theorem for probability measures on Polish spaces, citing Kallenberg.

• Explicit conditional structure: The conditional covariance depends only on t and the kernel: Σ_t = (k(t,t) − k_{(t,<t)}^⊤ K_{(<t,<t)}^{−1} k_{(t,<t)}) I_{d_z}; the conditional mean is affine: μ_t(z_{<t}) = (k_{(t,<t)}^⊤ K_{(<t,<t)}^{−1}) ⊗ I_{d_z} z_{<t}. The Kronecker structure Cov(z_{1:L}) = K_tt ⊗ I_{d_z} is used explicitly in KL and log-density computations.

• Position relative to Markov and AR models: The factorization is contrasted with the first-order Markov property p(z_{1:L}) = ∏ p(z_t | z_{t−1}) and with observation-level autoregression p(x_{1:N}) = ∏ p(x_t | x_{<t}). The paper positions its model as intermediate: it keeps full latent causality while operating in a continuous latent space where the covariance enforces smooth probabilistic coherence, and it allows fully parallel generation in observation space once z_{1:L} is known.

• Distinction from prior GP-VAE work: The paper argues that GP-VAEs induce correlated Gaussian priors z ∼ N(0,K) whose correlations are symmetric and do not encode causal direction — yielding latent continuity but not latent autoregression in the factorized sense. It states that, to the author's knowledge, no prior work combines a correlated GP prior on a latent trajectory, a fully causal factorization obtained analytically from GP conditioning, and a non-autoregressive likelihood model.

• Complexity scaling: Exact GP computations require a Cholesky factorization with O(T³) operations and O(T²) memory; the latent dimension d_z is removed from the factorization by the Kronecker structure. Using BBMM-style matrix-free inference with conjugate gradients and stochastic trace estimation, a single CG iteration costs O(T² d_z), the total is O(T² d_z n_iter), and classical CG theory gives n_iter = O(√κ(K_tt) · log(1/ε_CG)). Under a uniform spectral bound (H3) and fixed tolerance, the effective scaling in the experiments is described as close to O(T² d_z), compared with O(N² d) for a typical self-attention layer on observation tokens. Because L ≪ N and d_z ≪ d, the paper attributes the gain to a change in representation, not only to numerical approximation.

• Regularity and kernel choice: The kernel (RBF, Matérn, or spectral mixture) determines the regularity of the latent process in the classical sense of sample-path smoothness and influences the conditional means and variances appearing in the factorization.

• Experimental claims are limited: The paper reports only that the scheme "can be trained stably and exploited in practice in a constrained proof-of-concept regime" on a standard sequence dataset. The provided content does not name the dataset, give its size, list any benchmark, or report any quantitative metric; no numerical results are reported in the available text.

Methodology in Plain English

1. Assume the latents are GP evaluations. Pick ordered time points 0 < t_1 < ... < t_L ≤ 1 and a continuous positive-definite kernel k on [0,1]². The latent vectors z_t in R^{d_z} are treated as evaluations of a centered GP, so their joint law is multivariate Gaussian with covariance K_tt ⊗ I_{d_z} (d_z independent copies of a scalar GP).

2. Factor the joint law causally. Write the joint density as a product of one-step conditionals. Because the joint law is Gaussian, each conditional is Gaussian and its mean and covariance come directly from standard multivariate Gaussian conditioning — no parameters are learned for the transitions.

3. Build a VAE around it. An encoder network produces an amortized approximate posterior q_φ(z_{1:L} | x_{1:N}) over latent trajectories. A decoder produces observations from the whole latent trajectory at once: p_θ(x_{1:N} | z_{1:L}) = ∏ p_θ(x_n | z_{1:L}), so all positions can be sampled simultaneously.

4. Optimize an ELBO. Maximize the evidence lower bound, whose first term is the expected log-likelihood under the encoder and whose second term is the KL divergence from the encoder posterior to the correlated GP prior (optionally scaled by β).

5. Make it computationally feasible. Rather than cubic Cholesky costs, use matrix-free BBMM-style GP inference with conjugate gradients and stochastic trace estimation, which the paper says also enables joint optimization of kernel hyperparameters with the variational objective. A jitter term εI is added to K_tt to keep the Gram matrices numerically well conditioned, and the numerical assumptions (H3)–(H4) only justify the complexity claims.

6. Test as a proof of concept. Implement with finite-dimensional GP priors and a non-autoregressive decoder and run on a standard sequence dataset.

Why This Matters

Impact on research. The paper offers a conceptual alternative to learned state transitions in sequential generative modeling: sequential structure expressed analytically through covariance geometry rather than parameterized recurrence. It also sharpens a distinction in the GP-VAE literature between symmetric latent correlation (which existing GP-VAEs provide) and causal, direction-encoding latent factorization (which the paper claims they do not), giving a concrete theoretical target for follow-up work. A secondary contribution is the complexity argument that encoding temporal dependence once in a latent covariance may be cheaper than recomputing dependence layer by layer in observation space.

Real-world applications (as directions this framework could plausibly serve — the paper reports no application-specific results):

• Continuous-signal modeling, where the paper notes Gaussian likelihoods are one natural choice for the observation model (e.g., sensor, audio, or physiological time series).
• Discrete-symbol sequence modeling, where the paper notes categorical likelihoods are the corresponding choice.
• Settings where parallel generation matters, since the decoder is conditionally independent across positions and permits fully parallel sampling once the latent path is drawn.
• GP regression-like interpolation tasks, since kernel choice (squared-exponential, Matérn, spectral mixture) directly controls latent smoothness and can encode approximate periodicities.

Industry relevance. The most directly relevant lever is cost: the paper contrasts a GP-related cost of roughly O(L² d_z) in a small latent space against O(N² d) for a typical self-attention layer over observation tokens, and motivates BBMM-style inference for quasi-quadratic GP computation. If the stability claim holds outside the constrained proof-of-concept regime, the design could appeal to practitioners who want smooth, uncertainty-aware latent trajectories and parallel decoding rather than step-by-step autoregressive generation.

Future Directions

1. Scale beyond the proof-of-concept regime. The paper describes its experiments as constrained and illustrative; whether the latent autoregressive GP prior trains stably at larger L and d_z, and on larger datasets, is not established in the provided content.

2. Close the gap to sequence-model benchmarks. No named benchmark, dataset size, or quantitative metric is reported here, so direct comparison against autoregressive and sequential latent-variable baselines remains open in the paper text available.

3. Exploit the conditional structure further. The paper shows the conditionals are analytic and depend on the full latent history, but how much long-range dependence this actually captures in practice — relative to Markov, AR(p), or hierarchical models — is discussed conceptually rather than measured.

4. Kernel and inference choices. The paper leaves open how the specific kernel (RBF, Matérn, spectral mixture) and the CG tolerance ε_CG jointly affect both the quality of the variational approximation and the realized computational scaling, noting only that the numerical assumptions do not enter the measure-theoretic results.

Target Audience

This paper is best suited to researchers and graduate students working in probabilistic machine learning, Bayesian non-parametrics, or stochastic processes who are already comfortable with Gaussian conditioning, covariance kernels, Kronecker products, and variational inference. Readers looking for empirical sequence-modeling results or an off-the-shelf language model will not find them here; readers interested in the theory of how causal structure can be derived from GP covariance geometry, and in a variational formulation built on that theory, are the intended audience. Applied practitioners may still find the discussion of complexity scaling and matrix-free GP inference relevant, provided they treat the experimental claims as explicitly preliminary.

Authors’ abstract

We investigate a fully Latent AutoRegressive scheme based on a Gaussian Process (GP) integrated into a Variational Autoencoder (VAE). In this setting, sequential dynamics are transferred from the observation space to a continuous latent space, while linguistic generation remains parallel through a non-autoregressive decoder. We present a complete methodological formulation, including a causal GP prior, a structured amortized posterior, and a training protocol based on a…