Reproducibility Study of Large Language Model Bayesian Optimization

AI summary

Overview

Research area: Bayesian optimization (BO), hyperparameter optimization (HPO), large language models, and empirical reproducibility of machine learning methods.

Technical level: Advanced. The paper assumes familiarity with surrogate models, acquisition functions, regret curves, and calibration metrics (NRMSE, R², LPD, coverage, sharpness).

Scope: An independent reproduction of the LLAMBO prompting-based Bayesian optimization framework, re-run with the open-weight Llama 3.1 70B model in place of the original GPT-3.5 backbone, across Bayesmark and HPOBench-style hyperparameter optimization tasks.

What This Paper Is About

Bayesian optimization usually works by fitting a surrogate model (like a Gaussian Process) to past evaluations and using an acquisition strategy to pick the next candidate. LLAMBO (Daxberger et al., 2024) asks whether a large language model can do this instead, purely through text prompts that describe the dataset, the model, and the evaluation history. This paper checks whether LLAMBO's reported advantages still hold when its proprietary GPT-3.5 backbone is swapped for an open-weight model, Llama 3.1 70B.

Key Contributions

1. A full open-model replication of LLAMBO. The authors rebuilt the prompting pipeline and replaced all OpenAI API calls with local ollama invocations, running zero-shot warmstarting, candidate generation, and surrogate-style performance estimation with Llama 3.1 70B.
2. A unified, reimplemented baseline suite. GP-DKL (BoTorch), SKOpt, Optuna (TPE), and SMAC3 were reimplemented using established libraries and evaluated under one shared protocol, alongside new evaluation and plotting scripts and a standardized JSON output format for LLM responses.
3. An alternative normalization analysis. In addition to Bayesmark's global score bounds, the authors implemented per-task min-max normalization, which produces regret curves that plateau at zero once the task optimum is found, and confirmed that both schemes yield the same qualitative ordering of baselines.
4. Ablations and a capacity probe. They tested an "uninformed" LLAMBO variant with problem descriptions and hyperparameter-name embeddings removed, and attempted the pipeline with smaller backbones (Gemma 27B, Llama 3.1 8B).

Main Findings

• Contextual warmstarting helps. Contextual warmstarts (No Context, Partial Context, Full Context) produced uniformly lower regret than classical space-filling designs (Random, Sobol, Latin Hypercube), with Full Context performing best and reduced variability across runs. Classical designs showed consistently higher regret, especially early.
• The warmstarts are structured, not collapsed. On a representative task (breast, RF), Random sampling showed the lowest average absolute correlation, while contextual warmstarts induced more task-specific correlation patterns. Latin Hypercube achieved the highest generalized variance; contextual warmstarts reached diversity comparable to LHCube and substantially higher than Random and Sobol.
• LLAMBO is a weaker standalone regressor. SMAC achieved the strongest pure regression performance (lowest NRMSE, highest R²), and Gaussian Processes remained best calibrated with near-ideal coverage and stable sharpness. LLAMBO and its Monte Carlo variant were weaker at low data regimes and systematically under-estimated uncertainty, shown by higher LPD, low coverage, and overly sharp intervals, though both improved as observations accumulated.
• Textual context is essential. The ablation showed the uninformed model (no problem description, no hyperparameter-name embeddings) had consistently higher NRMSE, with the largest gap at few observations, plus substantially worse LPD and poorly calibrated, overconfident predictions. The full model retained an advantage at all data regimes.
• The LLAMBO candidate sampler is strong. Against Random sampling, TPE with independent marginals, and multivariate TPE, LLAMBO achieved the lowest average regret, reached low-regret solutions earlier and with lower variance, maintained balanced diversity, and achieved the highest log-likelihood under the surrogate density.
• Smaller backbones failed. Gemma 27B and Llama 3.1 8B frequently returned malformed outputs (invalid JSON, missing hyperparameters) and surrogate scores that did not correlate with observed performance, causing unstable loops, constraint violations, and inconsistent rankings of nearly identical candidates.
• Normalization changes the shape, not the ranking. Per-task min-max scaling uncovered more pronounced early-regret disparities than the global Bayesmark-wide bounds, but the qualitative ordering of optimizers was unchanged.

Methodology in Plain English

The authors took the original LLAMBO codebase and rewired it so that every call to a proprietary model API became a call to a locally hosted open-weight model. Because the original repository used nested bash scripts, spread prompts and configuration across multiple files, and offered no utilities for aggregating results or plotting figures, the team wrote their own evaluation and plotting scripts for Bayesmark and HPOBench and forced all model outputs into a common JSON format.

They then ran a controlled comparison. LLAMBO was tested on the five Bayesmark tabular datasets—Breast Cancer (569 samples, 30 continuous features, binary classification), Diabetes (442 instances, 10 clinical predictors, regression), Digits (1,797 samples, 64 pixel-intensity features, multiclass), Iris (150 samples, four botanical measurements), and Wine (178 instances, 13 chemical attributes, multiclass)—crossed with five model classes (Random Forest, AdaBoost, SVM, Logistic Regression, and a simple neural network), giving 25 Bayesmark tasks. The full evaluation covered 30 tasks, adding 3 private datasets not seen during LLM pretraining and 2 synthetic datasets designed to probe behavior on controlled objective landscapes.

Every optimizer, including LLAMBO, ran under identical conditions: 5 randomly sampled initial points, 25 optimization trials after initialization, and 5 independent runs per task. Four baseline optimizers were included (GP-DKL, SKOpt, Optuna/TPE, and SMAC3), and LLAMBO's components were measured with both accuracy metrics (NRMSE, R², regret) and uncertainty-quality metrics (LPD, coverage, sharpness). Bayesmark's precomputed performance statistics were used to compute regret consistently across methods.

Why This Matters

Reproducibility studies like this test whether a method's gains survive a change of implementation, hardware, and backbone model—an increasingly important question as LLM-based methods proliferate. The finding that LLAMBO's core behaviors transfer to an open-weight model is encouraging for researchers without access to proprietary APIs, while the failure of Gemma 27B and Llama 3.1 8B places a concrete capacity floor under the approach.

Real-world applications:

• Automated machine learning pipelines. Teams tuning models across many datasets could use LLM-driven warmstarting to spend fewer evaluations reaching good configurations.
• Scientific experimental design. The low-data regime that BO targets appears in drug discovery, materials science, and laboratory automation, where each evaluation may be expensive or slow.
• Robotics and control. Black-box tuning of controller parameters, where gradients are unavailable and evaluations are costly.
• Interface and product design. The paper lists interface design among the motivating domains for black-box optimization.

Industry relevance: Hyperparameter tuning is a routine cost in production ML. The paper's finding that contextual priors cut early regret and variance matters directly for compute budgets, and the requirement of a 70B-class model gives practitioners a concrete signal about the infrastructure needed. The authors also note that normalizing regret differently can change how optimizers appear to compare, which is a practical caution for anyone benchmarking AutoML tools.

Future Directions

1. Improve robustness at smaller model scales. The failures with Gemma 27B and Llama 3.1 8B raise the question of whether better prompting schemes, output validation, or fine-tuning could make the pipeline work with lower-capacity backbones.
2. Reconcile the single-task weakness. LLAMBO's surrogate under-performs GP and SMAC in isolation yet gains from cross-task priors. Understanding how to combine the two—for example, pairing an LLM prior with a classical calibrated surrogate—is an open direction.
3. Standardize normalization in BO benchmarking. Since per-task min-max scaling and Bayesmark's global bounds produce different curve shapes, the community needs agreed conventions for reporting regret.
4. Extend beyond Bayesmark and HPOBench. The private and synthetic datasets in this study were designed to probe controlled landscapes; broader testing across domains would clarify how widely the contextual warmstarting benefit generalizes.

Target Audience

Researchers and practitioners working on hyperparameter optimization, AutoML, and Bayesian optimization who want to know whether LLM-based methods are worth adopting; engineers evaluating whether an open-weight model can replace a proprietary API in an optimization pipeline; and reproducibility researchers interested in a concrete case study of re-running an LLM-driven method under a different backbone.

Authors’ abstract

In this reproducibility study, we revisit the LLAMBO framework of Daxberger et al. (2024), a prompting-based Bayesian optimization (BO) method that uses large language models as discriminative surrogates and acquisition optimizers via text-only interactions. We replicate the core Bayesmark and HPOBench experiments under the original evaluation protocol, but replace GPT-3.5 with the open-weight Llama 3.1 70B model used for all text encoding components. Our results broadly confirm the main claims of LLAMBO. Contextual warm starting via textual problem and hyperparameter descriptions substantially improves early regret behaviour and reduces variance across runs. LLAMBO's discriminative surrogate is weaker than GP or SMAC as a pure single task regressor, yet benefits from cross task semantic priors induced by the language model. Ablations that remove textual context markedly degrade predictive accuracy and calibration, while the LLAMBO candidate sampler consistently generates higher quality and more diverse proposals than TPE or random sampling. Experiments with smaller backbones (Gemma 27B, Llama 3.1 8B) yield unstable or invalid predictions, suggesting insufficient capacity for reliable surrogate behaviour. Overall, our study shows that the LLAMBO architecture is robust to changing the language model backbone and remains effective when instantiated with Llama 3.1 70B.