BITS for GAPS: Bayesian Information-Theoretic Sampling for hierarchical GAussian Process Surrogates

AI summary

Overview

Research area: Gaussian process surrogate modeling, Bayesian experimental design, and hybrid (grey-box) modeling, applied to chemical engineering process design. Technical level: Advanced. Scope: This paper develops an information-theoretic sequential sampling framework for Bayesian hierarchical Gaussian process surrogates and demonstrates it on a vapor–liquid equilibrium hybrid modeling case study.

What This Paper Is About

Surrogate models built on Gaussian processes are widely used to represent the unknown parts of hybrid models, but standard acquisition strategies for choosing new data points assume the GP's hyperparameters are fixed or set to a single point estimate. This ignores the uncertainty in the surrogate model itself. The paper introduces BITS for GAPS, a framework that places priors on the GP hyperparameters so that hyperparameter uncertainty is propagated directly into the sampling criterion, and it demonstrates the approach on a distillation design problem.

Key Contributions

1. A framework — Bayesian Information-Theoretic Sampling for hierarchical GAussian Process Surrogates — that couples fully Bayesian hierarchical GP surrogates with an entropy-based acquisition strategy, so that new sampling points are chosen using uncertainty from both the latent function and its hyperparameters.
2. A closed-form approximation to the differential entropy of the hierarchical GP predictive posterior, derived by approximating the log-density with a Taylor expansion around each mixture component's predictive mean, with computational cost dominated by evaluating posterior predictive variance per component, scaling as O(Sn²).
3. A theoretical lower bound on the posterior differential entropy, together with a proposition bounding the expected absolute truncation error of the J-th order Taylor approximation by C_{J+1} σ_s^{J+1} under a uniform bound on the (J+1)-th derivative.
4. A numerical case study showing how partial physical knowledge can be translated into a hierarchical GP surrogate, with a surrogate for latent activity coefficients in a binary mixture embedded into an extended form of Raoult's law to inform distillation design.

Main Findings

• Theoretical grounding: The paper establishes a closed-form approximation and a lower bound of the posterior differential entropy for a hierarchical GP, and shows that as the number of posterior samples S increases, the Monte Carlo approximation of the predictive distribution converges almost surely to the exact posterior predictive, which implies the entropy estimate also converges almost surely.
• Structure of the predictive posterior: The hierarchical predictive posterior is shown to be an expectation over the hyperparameter posterior, which in practice is approximated as a uniformly weighted finite Gaussian mixture model with one component per sampled hyperparameter setting. The first and second moments of that mixture follow standard Gaussian mixture moment results.
• Credible regions must be empirical: Because Gaussian mixtures do not admit closed-form quantiles, the variance expression from the mixture moments cannot be used directly, so credible regions of the predictive posterior are estimated empirically by drawing and sorting samples from the univariate hierarchical posterior.
• Sampling behavior in the case study: Using BITS for GAPS increases expected information gain and predictive accuracy by targeting high-uncertainty regions of the Wilson activity model, relative to the standard practice of using fixed or point-estimated hyperparameters.
• Scope of applicability: BITS for GAPS is presented as a generalized uncertainty-aware framework for adaptive data acquisition in complex physical systems, and is stated to be agnostic to the choice of Monte Carlo Markov chain algorithm used for posterior sampling.

Methodology in Plain English

The authors start from a serial hybrid model, where a known function combines known variables with the output of an unknown black-box function. That unknown function is given a Gaussian process prior, and instead of holding the GP's kernel hyperparameters fixed, they assign prior distributions to those hyperparameters — encoding beliefs such as expected smoothness or how relevant each input dimension is. This creates a Bayesian hierarchical model.

To fit it, they use Markov chain Monte Carlo to draw samples from the hyperparameter posterior, then propagate each retained sample through the standard conditional GP predictive equations. Averaging these gives a Gaussian mixture approximation to the full predictive distribution.

For choosing the next experiment, they need the differential entropy of that mixture, but the log of a sum of Gaussians has no closed form. They therefore expand the log-density in a Taylor series around each mixture component's mean and use that approximation. They prove a bound showing the truncation error shrinks with the component standard deviation, which justifies the approach.

They then demonstrate the whole loop on a vapor–liquid equilibrium problem: a space-filling initial design, GP calibration, surrogate quality evaluation, and selection of new points by maximizing posterior differential entropy, repeated until a stopping criterion is met. The resulting surrogate for latent activity coefficients is embedded in an extended Raoult's law to support distillation design.

Why This Matters

The paper addresses a gap that the authors identify explicitly: existing Bayesian optimization and model-based design of experiments approaches typically use fixed or point-estimated GP hyperparameters, which neglects posterior uncertainty in the surrogate model itself. It also notes that information-theoretic approaches to Bayesian optimization have seen limited application in chemical engineering, and that combining information-theoretic sequential design with fully Bayesian hierarchical GP models remains scarce. Prior work on hierarchical Bayesian modeling has been applied to uncertainty modeling and model updating, but rarely to sequential experimental design. This paper bridges that gap.

Real-world applications:

• Design of distillation systems, as demonstrated through the vapor–liquid equilibrium case study.
• Chemical process modeling and optimization more broadly, where hybrid grey-box models combine first-principles equations with learned components.
• Model-based design of experiments in settings where data collection is limited by time, cost, or computational resources.
• Surrogate construction for expensive simulations in areas the paper lists as particularly constrained, including design optimization, multiscale modeling, turbulent combustion, and materials mechanics.

Industry relevance: the case study targets chemical engineering process design, and the framework is aimed at practitioners who want to embed partial physical knowledge into a surrogate while acquiring data efficiently. Because the method is agnostic to the MCMC algorithm used, it can be paired with whatever sampler an organization already has in place.

Future Directions

• Extending the framework beyond the serial hybrid model structure assumed in this work to other hybrid and grey-box modeling configurations.
• Broadening information-theoretic sequential design to constrained, multi-objective, or batch settings, which the literature review identifies as areas of active development but which the paper does not address within the BITS for GAPS framework.
• Scaling the approach: the entropy approximation cost scales as O(Sn²), leaving open questions about efficiency for larger datasets or higher-dimensional design spaces.
• Testing on additional physical systems beyond the binary-mixture vapor–liquid equilibrium case study, and with kernel choices beyond the stationary kernels discussed (squared exponential, rational quadratic, Matérn).

Target Audience

Researchers and practitioners working on Gaussian process surrogates, Bayesian experimental design, Bayesian optimization, and hybrid or grey-box modeling — especially in chemical engineering and adjacent process systems fields. Readers need comfort with Bayesian inference, Gaussian processes, and differential entropy; the derivations in the framework section assume an advanced statistical background. Readers interested mainly in the application may find the vapor–liquid equilibrium case study accessible without the full theoretical apparatus.

Authors’ abstract

We introduce Bayesian Information-Theoretic Sampling for hierarchical GAussian Process Surrogates (BITS for GAPS), a framework enabling information-theoretic experimental design of Gaussian process-based surrogate models. Unlike standard methods, which use fixed or point-estimated hyperparameters in acquisition functions, our approach propagates hyperparameter uncertainty into the sampling criterion through Bayesian hierarchical modeling. In this framework, a latent function receives a Gaussian process prior, while hyperparameters are assigned additional priors to capture the modeler's knowledge of the governing physical phenomena. Consequently, the acquisition function incorporates uncertainties from both the latent function and its hyperparameters, ensuring that sampling is guided by both data scarcity and model uncertainty. We further establish theoretical results in this context: a closed-form approximation and a lower bound of the posterior differential entropy. We demonstrate the framework's utility for hybrid modeling with a vapor-liquid equilibrium case study. Specifically, we build a surrogate model for latent activity coefficients in a binary mixture. We construct a hybrid model by embedding the surrogate into an extended form of Raoult's law. This hybrid model then informs distillation design. This case study shows how partial physical knowledge can be translated into a hierarchical Gaussian process surrogate. It also shows that using BITS for GAPS increases expected information gain and predictive accuracy by targeting high-uncertainty regions of the Wilson activity model. Overall, BITS for GAPS is a generalized uncertainty-aware framework for adaptive data acquisition in complex physical systems.