Hyperparameter Search and Multi-Fidelity Experiments

Every trial spends compute and validation attention on one hypothesis. A search method decides which hypotheses receive more resources and when to stop weak candidates. The method cannot rescue a meaningless space or a biased metric. Search quality begins with parameter semantics, resource accounting, and a plan for confirming the winner once the search is over.

The baseline everything else is measured against came out of an experiment, not a preference. Blind sampling in thirty-two dimensions was put up against a careful human. Bergstra and Bengio reported the result in 2012 without hedging: “Compared with deep belief networks configured by a thoughtful combination of manual search and grid search, purely random search over the same 32-dimensional configuration space found statistically equal performance on four of seven data sets, and superior performance on one of seven.” Random sampling tied the human on four data sets and beat him on one. The reason they give is what makes the result travel: “only a few of the hyper-parameters really matter” on any given data set. A grid therefore spends most of its evaluations re-measuring coordinates that do not move the metric. Van Rijn and Hutter later reached the same conclusion from the other side, running functional ANOVA over OpenML meta-data for SVMs, random forests and Adaboost. They found two dominant hyperparameters per algorithm.

That baseline is beatable, and it has been beaten in public under a common protocol. Tuning was framed as Bayesian optimization in 2012 by Snoek and colleagues, who reported algorithms that “improve on previous automatic procedures and can reach or surpass human expert-level optimization”. Turner and colleagues then ran the NeurIPS 2020 black-box optimization challenge and titled their report “Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning”. In that challenge, “baselines were set using the default settings of several open source black-box optimization packages as well as random search”. Random search was one of the baselines, not the winner. Both facts point the same way. A method has to beat blind sampling — four of seven, and one of seven — at a matched budget before its complexity is earned.

Choose based on budget, parallelism, fidelity, and noise.

Random search samples independent configurations from a declared space. It parallelizes perfectly, and it wastes full budgets on hopeless trials. Its claim to be a strong simple baseline is a measured quantity rather than a slogan: statistically equal performance on four of seven data sets and superior performance on one of seven, against a thoughtful combination of manual search and grid search in the same 32-dimensional space. It is the right default when many dimensions are weakly relevant. Two dominant hyperparameters per algorithm is what the meta-learning evidence found, so that is most spaces.

Bayesian optimization models performance in order to choose promising configurations. It buys sample efficiency and pays in surrogate misspecification, and its parallelism depends on the method. The NeurIPS 2020 challenge is the public evidence that the trade can pay. It is also the template for showing it: same tasks, same budget, random search in the baseline set.

Hyperband and ASHA allocate partial budgets and stop weak trials early. They parallelize well and are the natural fit for iterative training with checkpoints. Their characteristic failure is early ranking mismatch, which the next section measures. Population-based training evolves hyperparameters while trials continue training. That buys dynamic schedules and high parallelism, and costs attribution: when the schedule itself changed mid-run, it is harder to say which decision produced the final number.

A good optimizer cannot rescue a badly defined domain. Some coordinates only mean anything jointly, and batch size is the one with a documented body count in the literature.

The largest study of it trained 168,160 models across 35 workloads and released 71,638,836 loss measurements. It was published in 2019, and its abstract says what the field's disagreement was actually made of: “Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes.” The papers were not disagreeing about batch size. They were disagreeing about how well each batch size had been tuned, and reporting that as a property of batch size.

In 2017 the same coupling had already been exploited in the useful direction. A linear learning-rate scaling rule plus a warmup schedule let ResNet-50 train at minibatch size 8192 on 256 GPUs in one hour, with no accuracy loss. Sweeping the coordinate alone reproduces the confound. Sweeping it with the rule that ties it to the learning rate does not.

Fidelity must preserve enough ranking information to be useful. Cheapness is available along at least five axes: training duration, dataset fraction, model size, input fidelity such as image resolution or sequence length, and evaluation fidelity such as fewer seeds or a cheaper metric. Each one is a bet that the cheap ordering survives to full budget.

Hyperband is the canonical version of the idea. It allocates “a predefined resource like iterations, data samples, or features” to “randomly sampled configurations”, and the 2018 paper reports “over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems”. The saving comes from stopping bad trials, not from picking better ones. So the whole speedup rests on the partial ordering being informative.

That assumption has been measured, and it can fail outright. NAS-Bench-101 did the exhaustive version of the measurement in 2019. It trained 423k unique architectures at four budgets — 4, 12, 36 and 108 epochs — three times each: about 5 million trained models, a full record of what partial training predicts. On that record the speedup vanished. “The multi-fidelity optimization algorithms HB and BOHB do not yield the speedups frequently observed compared to RS or Bayesian optimization. We attribute this to the relatively low rank-correlation between the performance obtained with different budgets (see Figure 7 in Supplement S2).” NAS-Bench-201 then put a coefficient on the same effect. Validation accuracy at epoch 12 of a 200-epoch cosine schedule correlates 0.7396 with final CIFAR-10 test accuracy. The full 200-epoch run correlates 0.9937.

Model size is the axis where the assumption can be engineered to hold rather than hoped for. Under the Maximal Update Parametrization many optimal hyperparameters stay fixed as model width grows, so a small proxy can be tuned and the result transferred. Yang, Hu and colleagues showed this at NeurIPS 2021: “by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost.” Cerebras adopted µP independently, for a family running from 111M to 13B parameters, and reported that it improved “accuracy and hyperparameter predictability at scale”. The lesson is not that small proxies work. It is that this one works because a parametrization was designed to make the ranking stable. The 0.7396 above is what the same bet looks like when nothing was designed.

Keep three things apart: exploring, promoting, and finally confirming. Six steps, each with a documented failure attached to it.

1. Specify the objective — metric, constraints, direction and tie-breakers — before any trial runs. 2. Define the space with meaningful scales, bounds, conditionals and invalid-combination rules, including joint constraints where coordinates interact; the batch-size sweep that ignores the learning rate is the omission a 168,160-model study held responsible for years of contradictory publications. 3. Choose fidelity and then verify it. The correlation between the cheap signal and the full-budget objective is a number you measure: a validation accuracy at epoch 12 correlating 0.7396 with the final result is a failed check, and a 40M-parameter proxy whose optimum transfers zero-shot to 6.7B under the Maximal Update Parametrization is a passed one. 4. Allocate the budget — trial count, parallelism, stop policy, maximum resource — and record all four, because the budget can decide which model wins. 5. Confirm finalists by retraining the selected configurations at full data, full budget and several seeds. 6. Protect final evidence by evaluating frozen winners on a less-used test or temporal holdout.

Steps 5 and 6 are the expensive ones, and they are the ones that get skipped. They amount to what Cawley and Talbot call the internal protocol. The thirteen-benchmark gap in the next section is the price list for leaving them out.

Publish the search space, sampler, early-stop rule, failed trials, total compute, seed policy, and distribution of finalist results. The best single score conceals search variance. Sometimes it also conceals the fact that the identity of the winner was set by the budget.

That was demonstrated in 2019, with 50 random-search trials per classifier on five-way SST: “under a budget of fewer than 10 hyperparameter search trials, logistic regression achieves a higher expected validation accuracy than the CNN.” The CNN only overtakes logistic regression as the budget grows. So a paper reporting one number reports a fact about its compute as much as about its architecture, and the same authors' estimator implies published results whose unreported budgets range from hours to weeks. How to report this is contested rather than settled. A second group re-examined that expected-validation-performance estimator at ACL 2020, showed it is biased, and derived an unbiased replacement.

When possible, compare the automated method with random search under the same budget — and expect the margin to shrink. Li and Talwalkar ran that comparison for neural architecture search in 2019. At roughly the compute of DARTS, random search with early stopping reached PTB test perplexity 56.4 against ENAS's published 56.3. The image benchmark came out the same way: “for CIFAR-10, random search with early-stopping achieves a test error of 2.85%, whereas the published result for ENAS is 2.89%.” A separate benchmark, of eight NAS methods on five datasets at ICLR 2020, found most of them failed to beat a randomly-sampled-architecture baseline. Complex search should earn its overhead and operational burden, and two independent groups have shown what happens when it is asked to.