Generative Bayesian Optimization: Generative Models as Acquisition Functions

AI summary

Overview

• Research area: Bayesian optimization (BO), generative modeling, and likelihood-free/black-box optimization (stat.ML).
• Technical level: Advanced. The paper combines direct preference optimization (DPO), Kullback-Leibler divergence objectives, reward-weighted regression, and reproducing kernel Hilbert space (RKHS) convergence analysis.
• Scope: The paper proposes "GenBO," a framework that trains a generative model directly as a batch Bayesian optimization candidate sampler from noisy utility values, without an intermediate regression or classification surrogate.

What This Paper Is About

Bayesian optimization usually needs a probabilistic surrogate model (such as a Gaussian process) to define an acquisition function, and existing generative approaches to BO typically fit a surrogate first and then train a generative model on top of it, compounding approximation error and cost. This paper asks whether a generative model can instead be trained directly from simple, noisy utility values computed from observations, so that its density is proportional to the expected utility (the acquisition function value). The authors propose a general recipe for this, covering preference-based (DPO-style) losses and divergence-minimization losses, and analyze when the resulting sequence of distributions converges to the intended target.

Key Contributions

1. A general reward-model-free framework for generative BO (GenBO). The model is trained directly on noisy utility values to approximate a target proportional to the acquisition function, e.g. p*_t(x) ∝ p_0(x) a_t(x), or p*_t(x) ∝ p_0(x) exp a_t(x) for utilities that may be negative. This avoids constructing a surrogate (regression or classification) model, which the authors argue removes a source of approximation error and computational cost.
2. A preference-based (DPO-style) training objective and a robust variant. Using pairwise inputs and utility values, the generative model is trained with a Bradley-Terry preference loss, whose optimum is p*_u(x) ∝ p_0(x) exp(η⁻¹ E[u(y)|x]). Because DPO is not robust to preference noise, the authors adopt a robust preference loss (rPL) that accounts for a probability p_flip ∈ (0, 1/2) of preference directions being flipped.
3. Divergence-based learning objectives. Two forward-KL objectives are derived: a standard forward KL with importance weights, and a "balanced forward KL" derived from a Bregman divergence, which adds a penalty term so that points with u(y) = 0 are still penalized and the model does not retain high density in low-utility regions. A generalization using any proper scoring rule is also described.
4. Theoretical analysis of approximation and optimality. The paper shows the learned generative models follow a sequence of distributions that asymptotically approximate the target, including a bound for the optimal parameter that holds with probability at least 1 − δ, and an argument based on reward-weighted regression that the sequence of expected utilities is non-decreasing.

Main Findings

• No surrogate model is needed. The framework trains a single generative model whose density is proportional to the expected utility, rather than fitting a probabilistic regression or classification surrogate and then a generator on top of it.
• The framework covers a range of utilities. The listed utility functions are probability of improvement (u(y) = I[y ≥ τ]), expected improvement (u(y) = max(y − τ, 0)), soft expected improvement (u(y) = softplus(y − τ)), and mean (u(y) = y), with threshold τ_t typically the largest observation or a quantile of the empirical distribution.
• Sampling replaces acquisition-function maximization. Rather than finding the global maximizer of an acquisition function in a possibly highly non-convex landscape, the method samples candidates from the learned proposal, which the authors note concentrates in high-density, high-utility regions.
• Preference losses discard magnitude information. The authors point out that the DPO-style loss uses only the sign of pairwise utility differences, discarding the magnitude of the utilities — the motivation for also developing the divergence-based objectives.
• Balanced forward KL addresses a failure mode of zero utilities. Because PI and EI utilities can be exactly 0 below the improvement threshold, those points would receive no penalty under the standard forward KL, so the balanced variant adds a soft penalty term on such points.
• Convergence theory is stated via RKHS assumptions. Models q_θ(x) = exp g_θ(x) are assumed to lie in a reproducing kernel Hilbert space shared with the true log density g_*, the domain is assumed to be a compact metric space, and the main results are specialized to the finite discrete case (|X| < ∞). Individual losses are assumed strictly convex and twice differentiable in their first argument, and the regularizer R_n strongly convex.
• Convergence to the target is not the same as convergence to the optimum. The authors state explicitly that approximating p*_u is not sufficient for optimality, and argue that if the maximizer of the expected-utility sequence converges to the maximizer of f, the proposals concentrate at f's global optima. They note sublinear cumulative regret would require rate control left for future work.
• Importance weights were generally omitted in practice. The authors report that their experiments generally did not include the importance-sampling weights 1/q_{i−1}(x_i) for the KL-based losses, which they suggest promotes concentration of the posterior; a complete theoretical treatment of this is not provided.
• Experimental results are not reported in the provided content. The abstract states that performance is evaluated on challenging optimization problems involving large batches in high dimensions, but the supplied text is truncated and does not include the experiments section; therefore no benchmark names, dataset sizes, or numerical results can be given here.

Methodology in Plain English

The method repeats a simple loop. It starts with some initial data. At each round it fits a generative model to the available data by minimizing a loss, samples a batch of B candidate points from that model, evaluates them, adds the new noisy observations to the dataset, and repeats for up to T rounds (Algorithm 1).

What makes it different is how the model is trained. Instead of learning a function that predicts the objective (a surrogate) and then using it to define an acquisition function, the model is trained so that its probability density is high where the acquisition function — the expected utility of evaluating there — is high. The paper presents two families of losses for this. The first treats pairs of previously evaluated points and asks the model to prefer the one with the higher observed utility, following the logic of direct preference optimization; because observations are noisy, a robust version of this loss compensates for a known flip probability. The second family directly minimizes a divergence between the model and a target proportional to prior times expected utility, using importance weights because samples come from the previous proposal rather than the target.

On the theory side, the authors place both families under a common loss template combining a complexity penalty with per-point losses, assume the model's log-density lies in an RKHS, and show the optimizer of the empirical loss stays close to the true target log-density, with a high-probability bound. Since matching the target is not by itself the same as finding the global optimum, they further connect the update to reward-weighted regression, which gives a non-decreasing sequence of expected utilities under suitable conditions.

Why This Matters

• Research impact: The paper argues for an alternative to the two-stage "surrogate then generator" pipeline common in generative BO, framing generative acquisition as a single-model, likelihood-free inference problem. It also imports DPO-style reasoning from language-model fine-tuning into Bayesian optimization, and provides convergence analysis under RKHS regularity assumptions.
• Enables large batches: Because generative sampling is cheap once a model is trained, the approach is aimed at batch settings where fantasy-sampling approaches (conditioning on simulated observations) become intractable as batch size grows.
• Handles non-continuous and high-dimensional design: The authors motivate the framework for non-continuous design spaces, high-dimensional design, and combinatorial or mixed discrete-continuous search spaces where classical BO is limited.

Real-world applications identified in the paper:

• Hyper-parameter tuning for machine learning algorithms (cited as a classic BO use case).
• Material design (cited as a classic BO use case).
• Robot locomotion (cited as a classic BO use case).
• Protein design and simulated/parallel evaluation scenarios, cited as motivating domains where large batches in high-dimensional spaces are needed.

Industry relevance: Parallel and batch evaluation is common in simulation-driven and design workflows, where many evaluations can be run at once even though each is expensive. A single trainable model that directly proposes batches could reduce the compute and error-compounding associated with maintaining separate surrogate and generative models. The authors are affiliated with CSIRO's Data61 in Australia.

Future Directions

• Extending beyond the log loss to other proper scoring rules. The paper outlines a formulation using a general scoring rule S, but states that exploration of this is left for future work.
• Applying the theory to models without closed-form densities. The framework is described as readily extensible to diffusion and flow matching models (Lipman et al., 2024), which the authors leave for future work.
• Controlling cumulative regret. The authors state that while simple regret may vanish if the targets concentrate on the optimizer set, sublinear cumulative regret requires rate control that is left for future work.
• Reconciling the omission of importance weights with theory. The experiments generally dropped the importance-sampling weights for the KL-based losses, and a refined analysis of the resulting concentration behavior is an open question the authors flag rather than resolve.
• Incorporating utility magnitudes within preference-based learning. The preference loss only uses the sign of utility differences, and the authors identify this as a disadvantage relative to the divergence-based objectives.

Target Audience

Researchers and practitioners working on Bayesian optimization, black-box and combinatorial optimization, and generative modeling for design problems — particularly readers already familiar with acquisition functions, Gaussian process surrogates, and large-batch or high-dimensional search. The preference-loss component also makes it relevant to readers coming from the direct preference optimization and reinforcement-learning-from-human-feedback literature, and the RKHS convergence analysis targets theoretically inclined machine learning researchers.

Authors’ abstract

We present a general strategy for turning generative models into candidate solution samplers for batch Bayesian optimization (BO). The use of generative models for BO enables large batch scaling as generative sampling, optimization of non-continuous design spaces, and high-dimensional and combinatorial design. Inspired by the success of direct preference optimization (DPO), we show that one can train a generative model with noisy, simple utility values directly computed from observations to then form proposal distributions whose densities are proportional to the expected utility, i.e., BO's acquisition function values. Furthermore, this approach is generalizable beyond preference-based feedback to general types of reward signals and loss functions. This perspective avoids the construction of surrogate (regression or classification) models, common in previous methods that have used generative models for black-box optimization. Theoretically, we show that the generative models within the BO process follow a sequence of distributions which asymptotically approximate an optimal target under certain conditions. We also evaluate the performance through experiments on challenging optimization problems involving large batches in high dimensions.