Bayesian Optimization in Practice

Bayesian optimization existed for decades in engineering design, but its machine-learning breakthrough was hyperparameter tuning. The reference point is the 2012 paper "Practical Bayesian Optimization of Machine Learning Algorithms" by Jasper Snoek, Hugo Larochelle, and Ryan Adams.

Their framing was simple and influential: treat the mapping from a model's hyperparameters to its validation performance as exactly the expensive black-box function this method assumes. Each evaluation is a full training run, there is no gradient from hyperparameters to validation score, and you can afford only a handful of runs. The fit is perfect.

They showed the approach matching or beating expert hand-tuning on real models, and released their implementation as a tool called Spearmint. That paper is why Bayesian optimization is now a standard item in the machine-learning toolkit rather than a niche technique.

The most widely used tuners today often do not use a Gaussian process at all. They use the Tree-structured Parzen Estimator, or TPE, introduced by James Bergstra, Remi Bardenet, Yoshua Bengio, and Balazs Kegl in their 2011 paper "Algorithms for Hyper-Parameter Optimization".

A GP models the objective directly: predict the score from the configuration. TPE inverts the question. It sorts past trials into a "good" group and a "bad" group by a performance threshold, fits a simple density to the configurations in each, and proposes points that are relatively likely under the good density and unlikely under the bad one.

This modelling-by-groups avoids the GP's cubic cost and, more importantly, handles the messy, conditional, mixed search spaces of real machine learning gracefully, where a hyperparameter may only exist if another took a certain value.

That last point is the practical reason TPE spread so widely. Real hyperparameter spaces are not clean boxes.

Suppose you are choosing an optimizer. If you pick stochastic gradient descent, a momentum value applies; if you pick Adam, it does not, but two other parameters do. The very set of dimensions changes depending on an earlier choice. A plain Gaussian process, which assumes a fixed-dimensional numeric input with a distance between points, has no natural way to say that momentum is undefined here.

TPE's tree structure, the "tree" in its name, represents exactly these dependencies, so it can search a space where the questions asked depend on previous answers. This is why the two most popular open-source frameworks, Hyperopt and Optuna, default to TPE rather than a GP.

You almost never implement any of this. The ecosystem splits by which engine a library favours.

GP-based, best for continuous, low-dimensional, expensive problems and research: BoTorch, built on PyTorch by Meta, with Ax as its higher-level interface; GPyOpt and scikit-optimize for lighter needs; Spearmint as the historical original.

TPE-based, best for general machine-learning tuning with conditional spaces: Optuna, now the common default, and Hyperopt, its predecessor.

import optuna

def objective(trial):
    lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True)
    depth = trial.suggest_int("depth", 2, 12)
    return train_and_validate(lr, depth)   # one expensive run

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=40)

The pattern is always the same: define a search space, wrap one expensive evaluation in an objective, and let the library run the surrogate loop for a fixed number of trials.

The basic loop is sequential: propose one point, evaluate it, refit, propose the next. If you have a cluster that can run eight training jobs at once, sequential evaluation wastes seven machines.

Batch, or parallel, Bayesian optimization proposes several points at once. The difficulty is that naively taking the acquisition function's top eight points returns eight near-identical points, because they all pile onto the same peak. The batch must be diverse.

Two standard fixes exist. Thompson sampling gives diversity for free, as the previous lesson noted: draw eight independent posterior samples and take each one's maximizer. Alternatively, use a fantasy update: after choosing the first point, pretend you have already observed a plausible value there, refit, and let that suppress the acquisition nearby before choosing the second. Both spread a batch across the space instead of stacking it.

The sharpest limit is dimensionality. Bayesian optimization is comfortable up to roughly ten to twenty dimensions and degrades beyond that.

The reason is the surrogate. In high dimensions, points are almost always far apart, so the space is mostly empty from any handful of observations. The Gaussian process then reports high uncertainty nearly everywhere, and the acquisition function, seeing uncertainty in every direction, has little signal to concentrate the search. You are back to something close to random.

Research attacks this by assuming the objective secretly depends on only a few directions, or by optimizing in a trust region around the best point, as in the widely used TuRBO method. But the plain method has a real ceiling, and a problem with a thousand knobs is not the place to reach for a textbook Gaussian process.

Beyond dimensionality, a few situations make the whole apparatus pointless, and recognising them saves real effort.

The honest summary is that Bayesian optimization buys sample efficiency, fewer expensive evaluations, at the price of modelling overhead and added complexity. That trade is only worth it when evaluations genuinely dominate the cost. A useful instinct: if a colleague could run the whole search by hand over a coffee break, you do not need this.

Pulling the cursus together: Bayesian optimization is a disciplined answer to one question, where should I spend my next expensive evaluation?

It fits a probabilistic surrogate, usually a Gaussian process, that predicts the objective and its uncertainty everywhere. An acquisition function, whether Expected Improvement, the no-regret GP-UCB of Srinivas, Krause, Kakade, and Seeger, or Thompson sampling, converts that belief into one next query by trading exploration against exploitation. You evaluate, refit, and repeat until the budget runs out.

For a rigorous but readable treatment of the whole framework, Peter Frazier's 2018 "A Tutorial on Bayesian Optimization" is the standard entry point, and the 2016 review by Shahriari and colleagues surveys the field. The core idea is durable: when a look costs a lot, it is worth thinking carefully about where to look.