Bayesian optimization addresses problems where evaluating a configuration is expensive and the relationship between inputs and outcomes is not directly available. In hyperparameter optimization, the inputs might be learning rate, regularization strength, and tree depth, while the outcome is a validation metric produced by a training run. Unlike model weights, which the training algorithm learns, hyperparameters configure how training operates.
The process begins with initial evaluations and builds a probabilistic surrogate model of the objective. Gaussian processes are a common surrogate, but implementations can use other approaches. An acquisition function uses the surrogate's predictions and uncertainty to select the next configuration. This balances exploitation, which favors settings predicted to perform well, with exploration, which investigates settings about which the optimizer has limited information. Each completed evaluation supplies evidence for subsequent decisions.
In the cloud, a tuning controller launches training jobs, collects their objective metrics, and schedules additional trials within configured resource limits. Engineers define parameter types, ranges, scaling, the objective metric, and whether to minimize or maximize it. Logarithmic scaling is often appropriate for positive parameters such as learning rate that can vary across orders of magnitude. Narrow, justified ranges generally use a limited trial budget more effectively than unnecessarily broad ranges.
For example, an engineer tuning a gradient-boosted fraud classifier might maximize validation area under the precision-recall curve while varying learning rate, maximum tree depth, and regularization. Bayesian optimization uses completed trials to guide subsequent configurations rather than evaluating every combination. The data split, feature processing, and metric calculation must remain comparable across trials. Selecting hyperparameters repeatedly against the test set leaks information into model selection; reserve that set for final evaluation.
Amazon SageMaker AI automatic model tuning provides Bayesian search alongside other tuning strategies. An implementation requires a training job definition, tunable hyperparameter ranges, an objective metric, and limits on total and concurrent training jobs. Custom training code must emit a metric that the service can extract, commonly through metric definitions with regular expressions. Set the optimization direction correctly: validation loss is normally minimized, whereas validation accuracy is normally maximized.
The main benefit is sample efficiency, not guaranteed lower elapsed time. Sequential feedback helps the optimizer learn, while high concurrency can launch multiple trials before their results are available. Parallel execution can shorten wall-clock time but may reduce the information available for each selection. Random search is simple and highly parallelizable; grid search suits small, deliberately discrete spaces. Bayesian optimization is often attractive when individual evaluations are expensive and the search has a manageable number of influential dimensions.
Noisy metrics, failed trials, irrelevant parameters, and overly large search spaces can impair results. Record dataset versions, code, seeds, configurations, and artifacts, and repeat promising trials when randomness materially affects ranking. Bayesian optimization does not guarantee a global optimum, does not inherently minimize infrastructure cost, and is not the same as training a Bayesian neural network. For foundation-model fine-tuning, it can search supported settings such as learning rate and batch size, but only where the training interface exposes those settings and repeated fine-tuning fits the budget.
Amazon SageMaker